19 April 2011

The Danger in User Surveys

One of the dangers, as pointed out by the BABOK is that surveys are not good at collecting actual behavioral information about our users. It seems to me that a large number of executives at Wal-Mart should have done a better job at understanding exactly when and why to use a survey.

If you're not familiar with Wal-Mart or their recent attempts to de-clutter their stores, its well worth looking into. Back when this was announced a couple years ago, I remember thinking to myself, why would they want to decrease the number of products they carry in each of their stores? There are many things I dislike about Wal-Mart, poor product quality, never enough cashiers, the other shoppers in the store with me, but the one thing I never complained about was how many products they offered.

Sure, it wasn't always the easiest store to find what you needed, but with the help of a sales associate or two, you almost always could find it. I can think of only a few times I ever shopped a Wal-Mart and was completely unable to find exactly what I was looking for or a close enough substitute that I left satisfied. Given that the number of options available was a good thing for me, I was astounded to find that management had decided to shrink their offerings.

Don't get me wrong; I was all in favor of wider isles and better organization, but not at the cost of fewer choices. I don't live at Wal-Mart, I just shop there. I can deal with clutter if I have to because at any time, I can turn around and leave the store.

This is a good lesson for those of us who work in projects to remember, that just because we think we know an answer and especially when we think we receive validation that our answer is correct, it doesn't mean anyone was really asking that question. Customers said they would like less clutter in the stores, but did anyone ask in the survey if the customer preferred clutter or choices? Obviously, given the outcome, customers choose clutter and options.

Have you ever used a survey in a project? Was the answer you received as faulty as the one Wal-Mart found?


  1. They might have SAID it was about reducing clutter in the stores, but I'm going to bet the real driver for reducing the number of SKU's was reducing the number of suppliers and getting lower costs out of them. Every product in the inventory has an overhead cost, even if you have the world's most sophisticated supply chain management tools, as WalMart does. Fewer products, lower carrying costs, possibly quicker turnover, and if you do it correctly, higher profits.

  2. Dave, you make a great point. I did assume, partially because I had seen no reason in the media to believe otherwise, that Wal-Mart was telling the truth and not making up an excuse. If they were lying, and to the good folks at Wal-Mart I'll give you the benefit of the doubt as 30 years from now when you're the only employer on the planet and I don't want this post to come back and haunt me, that actually points to an even larger flaw in their organization. Now they're not only putting forth an invalid assumption, they're rigging their own surveys to position themselves to prove the point they want to make. Not a good place to be in, either.

  3. Well, the post is actually the freshest on this worthy topic. I fit in with your conclusions and will eagerly look forward to your incoming updates.