Subject: RE: [hackers-il] statistical rant

From: "Arik Baratz" <[email-masked]>
Date: Tue, 27 Aug 2002 18:54:34 +0200

> -----Original Message-----
> From: Oleg Goldshmidt [mailto:[email-masked]]
> Sent: Tuesday, August 27, 2002 11:52 AM
> To: [email-masked]
> Subject: Re: [hackers-il] statistical rant

[snip]

> I believe they mean correlation, not covariance. Correlation is a
> factor in covariance, but they are not the same. In this particular
> case you got the sign right, but only because your observables are
> non-negative.

Please enlighten us.

-- Arik
**********************************************************************
This email and attachments have been scanned for privileged 
or sensitive information by PortAuthority. 
Vidius, Inc. We value information.

---

From: Oleg Goldshmidt <[email-masked]>
Date: 27 Aug 2002 22:22:17 +0300

> Please enlighten us.

I am not sure this list is a place where such enlightenment is
appropriate. You can get the details from any statistics/probability
textbook, but in simple words and without math it is roughly as
follows:

Covariance is a measure of how much 2 random variables vary, together
or separately. Mathematically, it is the expected value of the product
of their deviations from the mean.

(Linear) correlation is a statistical measure of whether 2 random
variables tend to vary in the same direction or in the opposite
directions. If, statistically, y increases when x increases x and y
are said to be positively correlated, if y decreases when x increases
they are negatively correlated. There is no information about *how
much* they vary. Mathematically, correlation is covariance normalized
by the product of standard deviations. The normalization takes the
expected amplitude of the deviation out of consideration.

This lack of amplitude information is what prompted me to make that
statement regarding correlations. After a bit of reflection, I have to
take it back, for the following reason.

Newspaper statements regarding various scientific (or not) studies
usually are not even about correlations, even when they look like
"people who read Haaretz have higher IQs". Such statements, even those
that seem to have some quantitative information in them, e.g. "the
study compared two groups of subjects, those who read at least 12
articles in Haaretz every week and those who read less than 2 articles
a week; the first group have average IQ higher by 20 points than the
second; hence people with higher IQ read Haaretz" are about neither
correlation nor covariance - they are about comparing
distributions. Nor are statements like that scientific - there is
normally no information about statistical, sample, or other errors.

Of course if you start reading Haaretz to appear smarter it will mean
that you have missed the point entirely.

About errors a bit more, as it is pertinent to the thread in general:
your typical opinion poll, e.g. during an election season, looks like
this: the Labor candidate leads the Likud candidate by 54% to
46%. Seems a significant difference, doesn't it? Very few such polls,
as reported by media, give you the error bars on these numbers. When
they do, the typical error of a poll of several hundred / a thousand
people is (by the pollsters estimate, under certain strong assumptions
- errors normally distributed etc - and thus probably underestimated
quite a bit) are of the order of 3-4%. This means that - believe it or
not - 54% == 46%, and there is *no* lead. In technical terms, the gap
is "not statistically significant". To understand that qualitatively,
think in terms of a rather high probability that the sample of
respondents has just a few more Labor supporters *purely by chance*. A
different sample of a similar size may have a few more Likud
supporters, *purely by chance*, reverting the result.

At TAU, things like that are (or used to be) taught in the first year
physics lab - a mandatory course for most exact and life sciences
students and engineers, but not for others (grin: I used to teach it,
I hope some of my students did learn a thing or two). I would not be
surprised if it were not taught at all in the probability/statistics
courses in social sciences / economics - these courses do not
emphasize measurements or data acquisition and analysis. I am not sure
mathematicians bother about these "real world" issues much, either. I
hope they do.

I didn't mean to disparage Haaretz, either.

Finally, and to bring it a bit closer to computer science, if you are
in the mood to learn more about comparing distributions, I suggest
looking up "Numerical Recipes" by Press et al. and - what else -
Knuth's TAOCP vol. 2 (it is on every hacker's shelf, isn't it?). Be
ready for math.

-- 
Oleg Goldshmidt | [email-masked] 
=================================================================
"... Of theoretical physics and programming, programming embodied 
the greater intellectual challenge." [E.W.Dijkstra, 1930 - 2002.]

---

From: "Nadav Har'El" <[email-masked]>
Date: Wed, 28 Aug 2002 01:29:11 +0300

> mathematicians bother about these "real world" issues much, either. I

We sure do :) I, for example, took as many as 4 semesters of probability
theory courses in the Technion, and this stuff is covered in the 2nd course,
I think. Later courses covered even crazier probability concepts, like
stochastic processes, martingales, brownian motion and stochastic intergrals
and other useless stuff like that :)

Of course, it doesn't mean I remember anything of it ;) Unfortunately,
unlike you, I never got to teach (and thus, remember) this stuff. This is
the downside to working full-time while doing your masters degree :(

-- 
Nadav Har'El                        |     Wednesday, Aug 28 2002, 20 Elul 5762
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |If Barbie is so popular, why do you have
http://nadav.harel.org.il           |to buy her friends?

---

From: Oleg Goldshmidt <[email-masked]>
Date: 28 Aug 2002 20:09:11 +0300

"Nadav Har'El" <[email-masked]> writes:

> Later courses covered even crazier probability concepts, like
> stochastic processes, martingales, brownian motion and stochastic
> intergrals and other useless stuff like that :)

Oh, and I spent years of my life making a (pretty good) living off
this kind of stuff... ;-)