I think you should seriously consider doing this. It's the best way to measure the true impact of YC on the companies it selects. Otherwise, it's hard to know to what extent you're just picking the most successful teams vs. genuinely making the teams more likely to succeed.
Addition: To avoid having to completely reject people, you could take randomly select half to enter immediately, and let the other half enter in one year. Then you could measure the difference between the two groups after a year.
It's not important know, and certainly not important enough to justify discarding half of our profits. It's also very difficult to get anything statistically significant out of the data when there are so few data points (there's probably only about one DropBox per batch).
Btw, if it were truly just a matter of us picking the right companies, that would be even more impressive given that we reject over 97% of applicants :)
That's obviously not the case though, since some of our most successful companies (such as AirBnb) wouldn't exist without YC.
Good point that the question is much less valuable from the perspective of YC partners. From that perspective, all that matters is that the companies perform well in the end. It matters more from the perspective of the applicants. Though even then, because the costs of joining YC are low, it's probably not a big issue for many people.
Agree the success rates would be hard to compare. You'd need to have some other proxies for eventual success besides profits.
Excellent point that if you can show you significantly helped the most successful couple of companies (which account for 80%+ of the value), and there were no other large successes among the people you rejected, then the job is done. You've shown YC increases the expected value of the companies you select.
Addition: To avoid having to completely reject people, you could take randomly select half to enter immediately, and let the other half enter in one year. Then you could measure the difference between the two groups after a year.