Firstly, Pete, for me a story usually starts with a situation--something that happened in my own life or that of someone I know. Some writers cull newspapers for inspiration. I sit with my bit of an idea for a while, letting it resonate. Then I turn to this huge file of notes I have--things that have occurred over the years, bits of dialog I've overheard, things I decided not to use in another book for whatever reason, etc.--and cull through to see if any of it might fit to move that initial idea along.
Sometimes characters are part of that initial idea, sometimes not. I try not to force this early phase of things; based on your comments, I sense this might be where you're getting in trouble. And because I'm a visual person, once I have an idea for a character, I have to find a photo (LLBean catalogs, Time Magazine--they've both worked for me). That makes all kinds of things "come." You never know what will push things along. Sometimes coming up with a title helps. I'm currently editing Island Mystery #2, for instance, but also working on plot elements for #3 and have been stymied. Then my husband came up with the book's title and things began to fall into place. No idea why it works this way, but it does--at least for me.
In the beginning I, too, had trouble being "mean" to my characters. An early reader of a draft of one of my first books complained that the people I was hurting/killing off weren't any that he cared about. Made me realize the only way to really engage the reader is to lure him emotionally, make him care about the very people you're gonna kill off. Trust me, as a writer you become inured to it after a while.
Your questions about subplots and layers and believability had me digging deep. The mythology sub-plot in HH, for instance, only occurred to me when I was about a quarter way through an early draft and after I'd had a series of dreams much like the episodes in the book where Jamie's dreaming about the figure running through the woods. Again this part of the process has to do with patience. You have to embrace the fact that things will come in spurts, sometimes requiring you to rewrite everything to date. It takes me anywhere from two to eight years to finish a draft of a book. Probably not what you want to hear, but it is what it is.
I think a lot of your writerly discomfort stems from the somewhat misguided belief that the writer is in control of the process. I find things go more smoothly if I simply trust my characters and get out of their way. Writing fiction is a very intuitive thing. Just wait and listen, and it will happen.