Dear Geist,
Why would a publisher sign up an unknown novelist's first book and say no thanks to an experienced writer's second book? That publisher bought my first novel two years ago. They did a great job, I did everything they said to promote the book, I earned out my (small) advance, the book got on a couple of awards shortlists. Meanwhile I was writing my second novel. A friend of mine was also writing a novel. He didn't have any contacts in the publishing business so I praised him to my agent, who took on my friend and his novel and then sold it to my publisher. All of this would have been swell except that when I finally finished my second novel, my publisher turned it down, mainly because my first novel hadn't sold as well as they'd hoped! WTF?
Dear Pete,
Ideally, publishers and writers work together over time, each getting better at their trade and enjoying the fruits of each other's labour. But in the current climate, publishers are less and less willing to take chances, and writers dash off to larger and/or more prestigious companies as soon as they or their agents can negotiate it. So every book has to earn its way. Clearly the publishers expected your first book to do better; our guess is they didn't see any significant difference in the potential audience for the second book, and therefore chose not to risk it. But they saw something in your friend's book that they guessed would appeal to a sizable market that they could reach.
None of this is your fault, or the publisher's fault. No one in the business can always get an accurate read on the public taste, the mood, the competition, and so on. But overall they're better at prognosticating than the rest of us—it's their job.
Our unsolicited advice for you is to continue writing the material you are excited to write, rather than what you think might sell. Markets are fickle and, as you've seen, even the experts sometimes guess wrong.