Subject: RE: [hackers-il] statistical rant

From: "Arik Baratz" <[email-masked]>
Date: Tue, 27 Aug 2002 11:23:25 +0200

> -----Original Message-----
> From: Chen Shapira [mailto:[email-masked]]
> Sent: Tuesday, August 27, 2002 5:48 AM
> To: [email-masked]'
> Subject: [hackers-il] statistical rant

> For example, countries with high divorce rates also have 
> higher junk-food
> consumption. This doesn't mean  that divorce cause people to 
> eat junk-food,
> or that eating junk-food leads to divorce. The most probable 
> explanation is
> that both are a symptom of modernisation, so they tend to 
> appear together.

I am not very good in statistics, but IIRC, there is a measurement called Covariance, defined over a pair of two random variables, which is defined

Cov(X,Y) = E((X-mX)(Y-mY)) 

where mX, mY are the means (avarage) for X,Y respectively E(X) is the expectency. A covariance of 0 indicates no relationship between two random variables, a positive covariance means there is a direct relation and a negative covariance... well, you got the point.

What the newspaper claims is that Covariance(divorce-rate-in-country,junk-food-consumed-in-country) > 0. There is a positive relation between the two random variables. This is not a useless piece of data - you could go to Tokyo, and assume that since there are a lot of junk food places there is a good chance that the divorce rate is high.

However, the newspaper goes farther than that, and assumes that there are no other variables involved, and hence there is some connection between the actual eating of junk food and actual divorce process. It might go on to assume that divorcees go into depression and hence consume junk food, or that divorcees have less money and resort to junk food (which is so cheap in the states).

In other words, the newspaper neglects to say that there may be another random variable, X, where

Cov(X,divorce-rate-in-country)>0 

and 

Cov(X,junk-food-consumed-in-country) > 0.

There is some kind of transitive relation between covariances, but I will not risk putting down the formula because I will probably get it wrong. Ofir, care to help here?

In short - statistics cannot determine a connection. It can suggest a connection, which can be established only if there is no X such as above - which is very hard to prove. A better way is to find the exact chemical in the junk food that causes people to want to divorce, conduct a double-blind reproducible experiments, etc.

Besides, why do you believe anything you read in the newspaper? HAARETZ wrote in its science column a week ago about how verifying that a number is prime will break all known encryption forms we know today, only to publish a counter-article a week later claiming the previous one was bull and actually the cryptoanalysis is a search problem and not a verification problem, and still exponential for the length of the input.

If you want I can scan both articles - it's a good example on how the newspaper can totally get it wrong.

> So, this week Haaretz cites a study from the economist, allow me to
> summarize:
> Higher people get payed more. Each centimeter in height adds 
> about 1.7% to
> the income in England, 1.8% in the states. Another study adds 
> that short
> people also have shorter lives. IIRC the figure was 2.3 years for each
> centimeter.

Cov(people-size,people-income) > 0
Cov(people-size,people-life-time) > 0

-- Arik
**********************************************************************
This email and attachments have been scanned for privileged 
or sensitive information by PortAuthority. 
Vidius, Inc. We value information.

---

From: Oleg Goldshmidt <[email-masked]>
Date: 27 Aug 2002 11:51:39 +0300

Arik Baratz <[email-masked]> writes:

> What the newspaper claims is that
> Covariance(divorce-rate-in-country,junk-food-consumed-in-country) >
> 0. 

I believe they mean correlation, not covariance. Correlation is a
factor in covariance, but they are not the same. In this particular
case you got the sign right, but only because your observables are
non-negative.

> In short - statistics cannot determine a connection. 

This is true: correlation does not mean there is a causal connection.

> Besides, why do you believe anything you read in the newspaper? 
> HAARETZ wrote in its science column a week ago about how verifying
> that a number is prime will break all known encryption forms we know
> today

Funny enough: people notice when a newspaper prints some BS about a
field they know a thing or two about, but tend to believe reports
about fields they know nothing about.

-- 
Oleg Goldshmidt | [email-masked] 
=================================================================
"... Of theoretical physics and programming, programming embodied 
the greater intellectual challenge." [E.W.Dijkstra, 1930 - 2002.]