We cannot predict how the development of something large and complex is going to play out. We are not sure about what the right product architecture is, nor where the snags are hidden. In most cases we don't even know what we'll actually want by the time the product is done, when the compromises that reality will inevitably impose on us are bought off. Nor whether anybody will actually want what we envision today.
If you have a history of developing similar things you have a feeling of how the events are going to unfold. You know which architectural elements are most likely to work, and which to avoid. But, unless what you are building is merely a clone of something you've done in the past, any certainty is delusion —except the sure knowledge that there'll be surprises.
We are apt to forget that we don't know the road ahead, and to see our plans as reliable roadmaps. This leads to rigidity and sub-optimal solutions.
We are bad at dealing with indefinition. Faced with an uncertain future, we find in our imagination what feels like solid comfortable ground. Unable to see a path, we invent it: we build detailed schedules, set checkpoints and deliverables, and make assumptions about how the final product is going to be that feel like a certain, final design.
And this would be all right, if it was not that we tend to believe the fiction that we built and inhabit, and act accordingly. This leads to decisions taken before their time, rigidity, and trying to hammer reality into conforming with our scheme of things, which reality resents.
We need schedules: if we want to introduce on June 2012 and we know that it typically takes six months to ramp-up production, we'd better be ready with a proven design by January, and it might be wise to schedule a couple of iterations of prototypes before that. But we should avoid putting too much into our schedules, and be weary of the assumptions that go in our plans, because otherwise we'll lose sight of the uncertainty inherent to them.
The Aymara people is an ethnic group that has lived in the Andes since long before the Incas, who conquered them in the XV century. Aymaras have a metaphor for past and future that is not found in any other human group: for an Aymara, future is behind, while the past is ahead. You see what's ahead, as you know your past; and you don't see what's behind, as you don't know your future.
It is life walked backwards. You see the past as is appears in front of you, but you never know what you will be stepping into when your next feet moves back. We imagine the future by extrapolating the past: if all you see, as you walk backwards, are nice round grassy hills, your best bet is that the nice hills will keep coming. There's no way you can imagine the cliff, or the wall, or the sandy patch a few meters behind you.
This might be a sensible point of view. Seing a complex project as a long trip, walking backwards in poorly charted territory, can help us stay grounded in the present, and keep our senses attuned to the subtle changes in landscape that give early warning of bigger changes to come.
We are bound not to foresee the black swan lurking after the next corner, but we should not stretch the metaphor too far. The things we do today have an effect on how the future's going to look like: if we play our cards right we can alter part of the landscape in our favor.
And we can try to be ready when surprises arise.
If you don't wed yourself to a particular fiction of how things should go, you'll be more able to spot a change in the program requirements, or a better architecture. And the better the tools you have built, the easiest it will be to change course.
There is a programming technique, popular among Lisp programmers, called building domain specific languages.
When faced with a complex program, you don't dive head first into it: instead, you extend the programming language by growing a set of constructs designed to deal with the kind of problem you are working on. The kind of tools that make exploration easy, and give you the high ground from which the problem is more easily grasped.
And you build tools on top of other tools, of increasing power —but usually decreasing generality— until you end up solving the problem by combining a few very powerful tools in the right way. Working like this makes you very resilient to changes in the landscape. You can cope with large adjustments of the scope of the program without having to go back to stage zero: you just recombine your tools, saying a different thing with the powerful language that you've built for this kind of problem.
It's easier to do this when building software than when designing hardware, but the basic technique is always worth keeping in mind. Build the tools, from the ground up, that will make it easy to assemble your product. And try to make them so that many different combinations can be made with them.
A decision is an anchor. Avoid them until they cannot be delayed any longer, because they will make you rigid. The more committed you are to a particular course of action the more difficult it is to adapt to changes in the environment. A decision always closes potential paths that might have proven to be the best: keep them open as much as you can.
(Disclaimer: I do get a cut from your Amazon purchase.)