[Computational Complexity] Accuracy of Predicted Probabilities
I stumbled upon the so called College Admissions Services which will give, for a fee, your percent chance of being admitted to undergraduate colleges in the US. I can't vouch for or against this service but I did catch an interesting claim of being 98% accurate. What does 98% accurate mean when you give probabilities? There are some reasonable answers to this question but not the one used by this site.
They do give the formula they use, roughly the fraction of people who didn't get refunds. Someone is eligible for a refund if the prediction was at least 51% and they didn't get in or the prediction was less than 50% and they were accepted.
What's wrong with this picture? Suppose everyone who was eligible for a refund got one. Consider people who they predict have a 60% chance of acceptance. This means 40% of them should not be accepted. But if they are all accepted they would have considered this a perfectly accurate prediction though it clearly is not. Conversely if 60% of them were accepted, this is what you expect but they would consider that only a 60% accuracy rate. And if they predict 50% the formula counts this as an accurate prediction even if all or none of them were accepted.
Either we have the very unlikely scenario that the rounding to zero or one of the prediction is a very good predictor or more likely that not many people claim the refunds they are entitled to. When you make a claim to accuracy that doesn't match the service you provide you end up giving no claim to accuracy at all.
Posted by Lance to Computational Complexity at 2/20/2006 04:17:00 PM