Philip K. Dick is quoted as saying (paraphrase) “In a short story, the characters exist to serve the idea. In a novel, the plot/story exists to serve the characters.” Ok, so yes it’s a gross oversimplification and we can all think of exceptions (with all due respect to the Good Doctor, just about anything Isaac Asimov ever wrote, at any length). And it also seems to imply that short story characters, for lack of time, interest, and emphasis, are always going to pale against characters from a novel, which is nonsense on the face of it (see Fritz Leiber or Kelly Link or Andy Duncan. ’nuff said.).
And yet there is a grain of truth there. A short story by its nature is tight focus rather than wide-angle lens, and in this field especially is often very idea-driven, as the fantastic element which usually contains the core of the idea usually must be essential to the plot for the piece to qualify as sf/f in the first place. Even so, the implication (or my inference) is that all this is a conscious process. The writer gets this neat/nifty idea and finds characters and a situation to express that idea. I know it works exactly that way for a lot of people.
Doesn’t work that way for me, or at least not most of the time. Usually I start with an image, a character or two, and little more than a hint of what the situation is. Then I turn the character(s) loose and try to follow. Then the situation begins to clarify (or not, in which case the story is stillborn). If the story progresses, then, and only then, do I begin to understand what the story is about (plot/theme), which is what people are usually referring to when they refer to the “idea.” By that definition, half the time I don’t even know what the idea really is until the story is nearly done. I also know that I’m a long way from the only writer who works this way.
Even so, in an odd sort of way Philip K. Dick was right, even for this opposite-seeming approach—the characters/plot serve to reveal the idea, and the entire process hinges on something that often remains invisible until almost the very end. So in a technical sense you could still say that the characters serve the idea, in the sense that the characters exist to reveal and clarify the idea. That is, one could say that, if one believed that the idea thus revealed is automatically paramount in any story short of novel length, and that any given story is really about what it appears to be about. I mean, yes, that’s often true. At least as often, it is not.
For examples, one could contrast and compare Ted Chiang with, say, Jeffrey Ford. Chiang’s brilliant Tower of Babylon is a good example, a what-if look at cosmology based on the Babylonian concept of concentric spheres. Here it really is all about the idea, skillfully revealed through the actions of the characters. Now I contrast that with Jeffrey Ford’s equally brilliant The Weight of Words, where he seems to be in Chiang territory of the “big idea,” but I’d argue that here the central conceit, that words have physical weights and calculable effects, serve mostly to reveal the characters, and the moral and philosophical dilemmas they face as a direct result of the reality of the central idea. Read those stories one after the other and then see if you think Chiang and Ford are doing the same thing.
I tend to think of any hard and fast rule as something like a bucket. Fine if what you’re trying to contain is water, but what if that water is actually steam or mist half the time? Still chemically the same thing, but no longer bound by the rules that worked pretty well just a minute ago. It’s not so much that “rules are made to be broken,” but rather that, in both art and craft, sometimes rules are made to be understood and acknowledged so you know when to ignore them. Not to be difficult or “cutting edge” (whatever that means), but because sometimes they just get in the way.