So to your first question "Why do my requirements keep changing?" The first point I want to make is that I haven't seen any studies that look at why requirements go bad, but there are plenty around that indicate they are a leading source of project problems. So the discussion will only be built around my personal experiences and observations of projects going on around me.
My view on this question is that there are four basic answers.
- Clients jump into their solution work before they properly understand their problem
- In multi-stakeholder environments (i.e. large organizations) there is rarely one clear voice singing the requirements tune,
- The people you put into project management and requirements management roles don't have the right skills and knowledge
- The external market conditions are changing fast and it's just hard to get the product lined up with the market
You'll see some of these issues link back to the issues that the old Chaos reports used to call out about project management but are now just more specific to the requirements management work stream. The only one that seems to me to be legitimately out of the client company's control is the last one. A variation on this is in political organizations and government departments where often PR and politics can overwhelm sensible analysis and planning.
Bas and Craig have a weekly conversation, back and forth on their respective blogs, Project Shrink and Better Projects. With blog titles like that, you don't have to guess what the topic will be.