When using group based estimates how many people do you need for a representative sample?
There are a couple of things to think about when getting into this.
Firstly you need to understand the underlying principles behind group based estimates. I go to Mike Cohn's work for this - for example Agile Estimating and Planning. Secondly your team need some basic schooling in what estimating is about, and the team need data to continually learn from what has gone before. Estimates without evidence are guesses.
So, once you have your systems and approach worked out, the question: How many people from the team need to participate? I used the Sample Size Calculator at Survey System to get a statistically relevant sample size the above numbers. The site has also got some definitions of confidence level and confidence interval there for you in case (like me) you need to check the definitions.
The numbers are presented in the diagram above are for a 20% interval (i.e. results will be within 20% tolerance) with a 95% confidence level. By knowing this you know whether you can get away with estimates with less than everyone on the team and what it does to your confidence in the results.
This isn't the end and to start with the 95% confidence level with a 20% interval will only reflect a band of what people think the work will take.
In reality you'll have awful estimates to start with. Just like anything you do at work, it's a skill you need to develop. It's not hopeless though. Observe, learn and improve.