## Re: [hackers-il] I want to believe

Expand Messages
• ... First, the square root of the variance (the standard deviation) gives a more meaningful measure... Second, I don t see what the high variance has anything
Message 1 of 2 , Feb 20, 2001
On Tue, Feb 20, 2001, Chen Shapira wrote about "[hackers-il] I want to believe":
> Well, this particular test, had 18 questions, a right answers gives you 6
> points, a wrong one substracts 2.
> Lets assume I came to the test with 75% knowledge.
> So each question I get right with 0.75 probability.
> The expectancy of my score is quite reasonable - 72.
> The variance on the other hand is 6048 (18 independent questions each with
> variance of 12).
> Unless I had an error of calculations, this is an extremely high variance,
> meaning that a test score has low chance of reflecting my actual knowledge.

First, the square root of the variance (the standard deviation) gives a
more meaningful measure...

Second, I don't see what the high variance has anything to do with the
correlation between the score and your knowledge! I mean, if you answered
questions by random, then you're right - you'd expect a crazy distribution
of random scores, ranging from -36 to 108 with a high variance. But that's
not what happens: assume you have a intrinsic success fraction A (e.g., 0.75 in
your previous example), and a small chance of making errors on questions
you ought to know or succeeding in questions you ought not to know.
This small chance means that A is no longer a constant, but rather a random
variable with a narrow variance (not that it doesn't have anything to do
with the variance you calculated).

So, if A is gaussian with a mean 0.75 and standard deviation, say, 0.05,
then you can calculate your score to be roughly
S = 18*6*A - 18*2*(1-A)
or
S = 144*A - 36

If you had a success rate A=0 (a constant distribution), you'd get -36,
if you had a success rate A=1 (a constant distribution), you'd get 108
(your teacher invented these weird limits)

But in our example, where A is a gaussian with mean 0.75 and stdandard
deviation 0.05, you'd get that the expected value of S is 144*0.75-36,
or 72 (as you calculated), and a standard deviation of 144*0.05, or 7.2
points.

It's only slightly higher than the 5 point standard deviation you'd "expect"
when you chose 0.05 as a standard deviation for A, and it is higher only
because your range (-36..108) is higher than the range 0..100.

So I don't see why you think this test can't reflect your knowledge.

--
Nadav Har'El | Tuesday, Feb 20 2001, 28 Shevat 5761
nyh@... |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |I love deadlines. I love the whooshing
http://nadav.harel.org.il |sound they make as they go by.
Your message has been successfully submitted and would be delivered to recipients shortly.