This is a fascinating thread.
It's been a while since I've mapped out a fantasy campaign, but I've been doing a lot of stuff with other games, mostly Warhammer 40k Dark Heresy RPG.
That game has the advantage (but also limitation) of an exhaustively predefined campaign world. This can constrain things a bit, but makes a lot of the "common sense" of the characters' world fairly intuitive. How I approach campaign-building, though, is through defining the structure of motivations of various people/factions in the campaign world.
Recently, for example, I've created a world (a Place, with physical confines that serve both as context and also provide various options for agency) where there are Imperial and provincial interests, the Law and various criminals, the Ecclesiarchy of the Cult of the Emperor, the Imperial Navy, a secret society of sorcerers, and the PCs themselves. Right now, they have other factions and persons with whom they interact, these from the history of the various story lines that have occurred in the campaign (over the course of about three years). I try as much as possible to do things that make sense within that configuration of actors, motivations, and the forms of agency available to them. Here's an example of some of my campaign info for our current storyline.
First, on the issue of Place: I like to provide enough details that my players have a sense of where they are, down to describing the geography, a general sense of the weather, and a general sense of the relationships between the physical locations and the people and things in them.
A sample of Place:
PLACES
Ursula II (a.k.a. Valis)
Ursula II is the only habitable planet in the Ursula System, and is called "Valis" by its inhabitants. Valis is a medium sized planet in a system with three small to large gas giants, and their accompanying moons. There is also a large and rich asteroid belt inside of the orbits of the gas giants, but relatively distant from Valis. Most likely, these are the remains of at least three planets. It is an imperial world, and is the seat of a largish, and tense, congregation of Imperial interests.
These interests include the Imperial Navy, the Ministorum, the Adeptus Arbites, and, of course, the Emperor's Most Holy Inquisition. There is also an entrenched nobility, mostly in the two major hive cities, and tensions tend to run high among them. There is, in addition, a small but influential criminal underground, led by two crime lords whose interests run in different directions (for now).
Among other things, the Ursula system contains an important refueling station for the Imperial Fleet, a lot of asteroid mining, some manufacturing, and a limited amount of agriculture. The planet Valis (aka Ursula II) is host to a large Adeptus Arbites training base. For this reason, the planet is considered to be among the most orderly planets in the Empire. This is true among the commoners, who are relentlessly surveilled by the Adepus Arbites trainees, and, as a result, the criminal underground of Valis is secretive and incredibly paranoid. Its members tend to either stay out of the spotlight entirely, or to take on "protective coloring" as businessmen and philanthropists.
The nobility of Valis, on the other hand, fight and feud among themselves like unto the fabled Stunties with their Book of Grudges. Many of the families have lineages which have "always been at war" with other families. Their dwellings usually are castle-like hive towers, and each family seeks to outdo the others in its own particular ways. Assassinations are common, and dueling is both common and accepted. The nobility, male and female alike, are expected to carry dueling pistols and/or swords, and to be able to use them.
So, we introduce the place and the people. There are a lot of other details (e.g., the two hive cities, the various seats of power of the factions, etc.), but this is the background for the more granular details.
As far as the NPC factions and their motivations, I represent them a bit like this. Here are my notes about two of the eight noble houses and how they relate to each other. It also includes info about who in those factions are the focal NPCs, the ones who actually will interact with the PCs:
House de Jagger
House de Jagger is one of the oldest and most honored of the Great Houses of Valis. They are reputed to be exemplary citizens of the Empire, and are pious followers of the Imperial Creed, donating generously to various Ministorum charities, which explains their friendly relations with Sapphon, the Crime Lord. They are, however, pretty thorny about their reputation, and expect a high degree of respect. They are excellent duelists.
Base of Operations: Spire de Jagger, Greystone
Motivation(s):
1. Primary: Reputation of the house.
2. Secondary: Service to the Empire.
3. Tertiary: Personal glory
Potential Allies:
• Senjak’s party
• Ministorum Witchfinders
• Adeptus Arbites
• Sapphon
• House Dakkarta
• House Gallo
Enemies and Potential Enemies:
• House Mythrux
• Willian Bellhouse
• House Narutha
• House Paxxu
This Faction Consists Of:
• Groot de Jagger, Patriarch. Straight-shooter, associated with the Adeptus Arbites. His daughter, Cornelia, was one of the young nobles killed by the daemonhost. He wants to get to the bottom of this, and lusts for vengeance.
House Paxxu
At war with House Cortenses over control of shipping and manufacturing interests. They are merchant magnates of the highest order: venal, greedy, and just a little bit evil.
Base of Operations: Spire Paxxu, Greystone
Motivation(s):
1. Primary: Power
2. Secondary: Destroy House Cortenses, once and for all
3. Tertiary: Destroy Sapphon, and take over his enterprises.
Potential Allies:
• Adeptus Arbites (because of their hatred of criminals)
• Administratum (because of their ability to produce revenues and exploit resources)
Enemies and Potential Enemies:
• House Cortenses (traditional enemy)
• Sapphon (because of their harsh treatment of workers)
This Faction Consists Of:
• Paxxu Melus, Patriarch—He is not aware of his young daughter’s involvement with sorcery, and would probably kill her, himself, if he knew. Much of his day to day work involve running the business interests, dealing with the crime families that run the system’s shipping concerns, and so forth.
• Paxxu Gloriana, Youngest Daughter of Patriarch--She is a spoiled little tart, with no morals to speak of, but she presents herself well, and is far, far more intelligent and cunning that her shallow socialite demeanor would lead one to believe. In "normal" society, she is considered the greatest beauty on Valis, and uses her looks, wits, and ruthless nature to great effect, cutting a swath through her vastly overmatched peers. She is, however, a bit of a prude, and usually uses the implicit promise of sex, rather than its actuality, to achieve her ends.
Secretly, she is a sorceress, and a member of The Order of the Infinite Way. She is affliated with Miklas Freeny, though she has embraced darker magics, and sees him as naïve about the true nature of power. She is a former apprentice of Miklas Freeny, but he's grown wary of her increasing power. She trusts him, to an extent, and will support him against his rivals, though mainly to keep him around until she can get him to part with some of his great store of arcane knowledge. She was jealous of Davilia Mythrux’s growing influence, though she wouldn’t whore herself out to him like, “that slut.” She resents the growing distance between herself and Freeny, and is incredibly glad to be rid of Davilia Mythrux.
Finally, I try to have a sense of what possibilities are in play. This means that there is a plot afoot, and that the PCs have a stake in dealing with it. In this case, they work for a relatively puritan faction of the Inquisition, and they have recently discovered the existence of a cabal of heretics consisting of practitioners of old-style sorcery (not the Psychic powers, not simply evil Chaos magic). Here's how I set that up:
What's really going on?
The planet's nobility contains elements of a very, very old order of sorcerers, The Order of the Infinite Way. Their symbol looks like an ankh, but with a Moebius Strip in place of the regular, single loop. They are sorcerers, some powerful and some not, and they tend to draw members from the nobility, especially among the Great Houses of Mythrux, Dekkarta, and Paxxu.
A member of the Order of the Infinite Way, Miklas Freeny, brought in a new member, his young mistress Davilia Mythrux, who more or less began to teach the magic she was learning to her peers, in order to show off and gain status among them. The young nobles tried to create a daemonhost out of a mendicant found in one of the hive city's seedier districts, not taking it as anything but an amusing diversion. Many of them had no clue what could (or would) happen next. Things did not, shall we say, go as planned. The mistress was killed along with her circle of dilettantes, and her lover/teacher's grimoire was recovered at the crime scene. It is the malefic codex found in the room where the creature was made to exist. Feeny will want to recover it at all costs.
Miklas Freeny. is also, in fact, the leader of The Order of the Infinite Way, and has many resources at his disposal, though they will have to be deployed subtly, so that The Order is not discovered. He is motivated to act now, not only because of the obvious danger of discovery by the Inquisition, but also by his anger at himself for falling prey to the oldest of magics: Feminine wiles. He also, now, and somewhat irrationally, has hardened his stance toward House Mythrux, largely due to the fact that one of its members (Davilia, his mistress) precipitated all of his current problems. He fears that Alize Mythrux might just use this incident, should it come fully to light, to replace him as leader of The Order. He is not incorrect in this, by the way. She does want to do that. This campaign to replace his leadership, probably, would precede (but only by the slimmest of margins) her campaign to end his life.
So, all in all, I think my approach is a literary one. I provide settings, characters and factions with real motivations, a sense of a plot (or several to choose from), and a the ongoing, historical inertia of the campaign as it has unfolded, complete with friends and enemies acquired along the way, and so forth.
I think it would be hard to do all of this in a module, but I think that most modules (good ones) do tend to provide, as one of you suggested, a "playground" (or theme park, perhaps), in which certain things that are possible, probable, definite, unlikely, etc. It also is sort of hard to do in that the GM (and players to some extent) have to remember a LOT of information. I like that, though, because I'd rather have a good sense of all of these things, as I don't necessarily try to drive the action if I don't need to. While the various plots and motivations and so forth do lead fairly predictably along various tracks, my "railroad" has many tracks to choose from. To extend this metaphor, the PCs also have knowledge that some of the things happening on each of those tracks are way more important than others, both in terms of their "mission" and in terms of their broader, long-term campaign goals.