Subject: statistical rant

From: Chen Shapira <[email-masked]>
Date: Tue, 27 Aug 2002 04:48:11 +0200

Hi,

Lots of statistical junk can be found in the newspaper. From carelessly
conducted polls to conclusions not justified by data.
One of the most popular errors is the idea that if two things occure
together, one of them must cause the other.

For example, countries with high divorce rates also have higher junk-food
consumption. This doesn't mean  that divorce cause people to eat junk-food,
or that eating junk-food leads to divorce. The most probable explanation is
that both are a symptom of modernisation, so they tend to appear together.

So, this week Haaretz cites a study from the economist, allow me to
summarize:
Higher people get payed more. Each centimeter in height adds about 1.7% to
the income in England, 1.8% in the states. Another study adds that short
people also have shorter lives. IIRC the figure was 2.3 years for each
centimeter.

Reading this, one may decide that Randy Newman was right in his song:
http://www.guitaretab.com/n/newman-randy/13264.htm

Or you may decide that someone is cheating. 
Its a well known fact that height has much to do with nuitrition. If you
don't eat healthy food as a child, you'll be shorter. 
So, one may  suspect that people who belong to lower social-economic
classes, those who have less access to nutritious food, and less education
about its importance will be shorter. Of course people from these classes
will also earn less, and since they have less access to medical treatment,
they may also live less. Malnutrition also affects the lifespan.

So, its not necessarily an employment bias against short people as the
article in Haaretz tried to claim
(http://www.haaretz.co.il/hasite/pages/ShArtPE.jhtml?itemNo=201319&contrassI
D=2&subContrassID=11&sbSubContrassID=0 free registration required). It may
be a case of irresponsible conclusions.

Thanks,
Chen.

---

From: Oleg Goldshmidt <[email-masked]>
Date: 27 Aug 2002 08:56:36 +0300

> that both are a symptom of modernisation, so they tend to appear together.

Or maybe that people who are careless about their eating habits tend
to be careless about choosing their mates or careless about their
mates once the choice is made. Or the other way around. Or maybe they
only notice each other's burger wraps on the floor...

Among other well-known correlations (correlations do not imply
cause/effect relationships): Dow Jones Industrial Index and the
population of polar bears, birth rates and number of stork nests
in some regions (Bavaria, IIRC, I may be wrong here), and my favorite,
the level of NY stock market (probably INDU again) and the most
fashionable skirt length. The latter has an "explanation": in good
times the society tends to be permissive, in bad times - conservative.

The most important thing is to check the data. A statistical analysis
can be verified given the data, it's only maths, after all. The way
the data were gathered, the biases that could have crawled in, are
much harder to verify or reproduce. E.g. from time to time our media report
that, say, researchers in the University of Aberdeen have proved
scientifically that people who drink 1-2 beers every night have a
lower risk to die of a cardiac arrest. Even if that's right, it's
about Scots in Scotland, in the Scottish climate, with Scottish food,
etc. Not necessarily relevant to us at all...

> the income in England, 1.8% in the states. 

Anglosaxons... ;-)

> Another study adds that short people also have shorter lives. IIRC
> the figure was 2.3 years for each centimeter.

Well, here is an explanation: higher people are generally healthier,
stronger, more competitive. They work harder and get paid more...

Your explanation is quite ingenious, Chen. You can sell it to the
media because they'll make a whole social phenomenon of an income gap
perpetuating itself through nutrition. They'll love it. There will be
just as much truth in it as there is mathematics (to quote Karl Marx).

;-)

-- 
Oleg Goldshmidt | [email-masked] 
=================================================================
"... Of theoretical physics and programming, programming embodied 
the greater intellectual challenge." [E.W.Dijkstra, 1930 - 2002.]

---

From: Adi Stav <[email-masked]>
Date: Tue, 27 Aug 2002 14:34:17 +0300

> together, one of them must cause the other.

My favourite is the often-quoted datum that couples who lived together
before their wedding have a higher chance of divorcing one another later
on than couples that did not.  It's completely obvious, once you think 
about it, that people who live together are probably more sexually 
liberal to start with than those who did not, and that more liberal 
people are also less likely to be intimidated by the possibility of 
divorce if their marriages don't work later on.  Yet people quote this 
as an absolute truth and even plan their lives according to it...

---

From: "Nadav Har'El" <[email-masked]>
Date: Tue, 27 Aug 2002 12:23:40 +0300

On Tue, Aug 27, 2002, Oleg Goldshmidt wrote about "Re: [hackers-il] statistical rant":
>...
>...

The late Stephen Jay Gould has an important book "The Mismeasure of Man"
(also available in Hebrew translation, should be available for 30 shekels
in Zomet Sfarim). It discusses how screwed-up statistics, false correlation
calculations, biased data gathering, and so on was used to perpetuate
ideas about the superiority of various races over others, the superiority
of males over females, the inferiority and tendency for criminality of
various immigrants, and so on.

I think it's a very important read, especially for people who still believe
in the validity of things like IQ tests, psychometric tests, and so on (which
Gould mostly thinks are the 20th century replacements of skull volume
measurements, skull angle measurements, etc.).

-- 
Nadav Har'El                        |       Tuesday, Aug 27 2002, 19 Elul 5762
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |I am thinking about a new signature. Stay
http://nadav.harel.org.il           |tuned.

---

From: Tzafrir Cohen <[email-masked]>
Date: Tue, 27 Aug 2002 12:59:53 +0300 (IDT)

On Tue, 27 Aug 2002, Nadav Har'El wrote:

> in the validityof things like IQ tests, psychometric tests, and so on (which

I admit I haven't read that book.

But there is a common misconception regarding psychometric and similar
exams: the exams are supposed to be 'statistically correct', they are not
suposed to be 100% correct. They should give a reasonable prediction (for
a reasonable price) of who has a good chance of making it through the
university. The university does not want to waste too many resources on
students that will not graduate, or will graduate, but not well enough.

The inherent problem of any such prediction test (if it is shortenough)
that no matter how perfectly it was originally designed, there will always
be ways to "prepare" for it without improving your 'university-studying'
skills. And there will be a positive coelation between the level of income
and the chance of using such ways.

-- 
Tzafrir Cohen
mailto:[email-masked]
http://www.technion.ac.il/~tzafrir

---

From: Oleg Goldshmidt <[email-masked]>
Date: Tue, 27 Aug 2002 09:55:19 +0300

> as an absolute truth and even plan their lives according to it...

The way you describe it, it *is* true... :)

---

From: Oleg Goldshmidt <[email-masked]>
Date: 27 Aug 2002 12:55:52 +0300

> measurements, skull angle measurements, etc.).

Another interesting read is a book (the author's name slips my mind at
the moment) called "Science, Non-Science, and Nonsense" that adds
things like lie detectors and drug screening to the list, among
others. The statistics involved is explained in layman terms, which
will probably still be too difficult for the average PHB.

Actually, the real purpose of psychometric tests (not the university
ones, the ones some companies send prospective employees to),
graphology evaluations, etc is covering the hiring manager's behind:
someone external and unbiased said something about the candidate. This
sounds reasonable only until you think whether you want to work at an
organization that makes important decisions this way...

---

From: "Nadav Har'El" <[email-masked]>
Date: Wed, 28 Aug 2002 01:49:29 +0300

> various immigrants, and so on.

Today on "Universita Meshuderet" (in Galey Zahal) about Criminology, the
teacher discussed whether criminality is inherited or not. Before he gave
his conculsion (that some traits that may assist criminality, like hot-
headedness, risk-loving, etc., may be inherited but the overall influence
of inheritence on the individual's becoming a criminal is very small), he
discussed a few flawed research attempts that invariably tried to "prove"
that criminality is caused by biological defects in the brain, and perhaps
even inherited (in the genes) and some of them even suggested that criminals
should therefore be prevented from procreating.

One of the example he gave on statistically-wrong conclusions: A study in
Denmark studied the correlation between mothers smoking in the third
trimester of their pregnancy, and the child's turning up to be criminals
or drug addicts. They found a correlation between those things, and suggested
that smoking in the third trimester causes biological damange to the fetus's
brain that will later cause it to become and addict or criminal.
The teacher in that Universita Meshuderet said that this is a false
conclusion (is "non sequitor" the Latin term? Excuse me for a slight case
of amnesia at this time of night...). Another explanation might simply be
that smoking while pregnant is clearly a sign that the mother doesn't care
very much about the child (because everyone knows that it is bad for the
child!), and the same mother will continue to care less about him as he
grows up, and non-cared-for children might have a higher chance of becoming
criminals, turning to drugs, and so on.

-- 
Nadav Har'El                        |     Wednesday, Aug 28 2002, 20 Elul 5762
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Why do doctors call what they do
http://nadav.harel.org.il           |practice? Think about it.