Subject: RE: [hackers-il] RE: My Random Ideas Page

From: Chen Shapira <[email-masked]>
Date: Wed, 5 Jun 2002 13:42:31 +0200

> -----Original Message-----
> From: Shlomi Fish [mailto:[email-masked]]
> Sent: Wednesday, June 05, 2002 12:33 PM
> To: [email-masked]'
> Cc: Hackers-IL
> Subject: [hackers-il] RE: My Random Ideas Page
> 
> 
> 
> Note to Hacker-ILers: I'm CCing this message here because it 
> bears great
> relevance to computers.
> 
> On Wed, 5 Jun 2002, Chen Shapira wrote:
> 
> >
> >

> I'm not sure how much what Brooks says about team management 
> and software
> engineering cannot be directly implied to open-source. In 
> many open-source
> projects the developers are more concerned of playing with 
> the elements,
> and the final product is a beneficial side-effect of it. In many
> commercial or commercially-sponsored projects, it's the other 
> way around.
 
I'm not sure either, which is why I'd love to test it.
I agree that seperating coders from "others" isn't easy, and defining and
measuring progress isn't trivial either, but I still find the idea
fascinating, and come back to it once in a while.

One of the potential biases is that the large and successfull projects
aren't in source-forge at all, they must be hunted down seperately, and
there is no clear way to get the number of developers. 

> I find the field of statistics relatively interesting: all this stuff
> about collecting, analyzing and verifying data, and all those tests,
> buzzwords and processes. As an electrical engineer we have to deal
> extensively with Probability Theory, which is a sister subject of
> statistics, that I found to be fascinating.

So do I. I keep wondering why I didn't think of it as a potential area of
study, before my Probability TA strongly suggested it. 
I think that statistics may have bad reputation, or just not enough
reputation.

> "There are three kind of lies: lies, damn lies and statistics"

One can also say that "If you torture numbers long enough, they'll admit
anything".

In many ways its very very true. Anything from sampling techniques to
analisys can be skewed in any direction. 
On the other hand, the more you know, the more you can question.
Suprisingly, newspapers often give enough information about the polls they
publish to let you analise the data yourself. You are still affected by
sampling biases of sorts, but those can also be detected sometimes.

Recommended book on biases and sampling: "Emet Va-Seker" (Truth and poll), I
don't have the author name in front of me now.

---

From: Shlomi Fish <[email-masked]>
Date: Wed, 5 Jun 2002 14:16:47 +0300 (IDT)

> there is no clear way to get the number of developers.

It reminds me of an article someone wrote on Advogato, in which he
interviewed some prominent developers, and tried to analyze what can make
a project successful. The problem with the article is that it only
included very brief points. My "Rules of Open-Source Programming" was also
an attempt to define some good guidelines for managin projects well.

I generally think of Open-Source as a philosophical innovation that became
possible (and common) because of the digital revolution. It may still be
premature to try to successfully analyze it.

> don't have the author name in front of me now.

It is known that even "Mishaley Am" are not an objective way of
determining what the people want as sometimes there isn't a very
good choice or the questions are phrased badly. There is a similar issue
with making polls. I know for once, that web polls or mailing list polls
hardly ever seem to work and most people will not take the time to answer
them.

Slashdot polls are cute, but they are hardly ever serious, and are not
supposed to reflect the opinion of the masses or to be a good statistical
source. But I still find them amusing.

Regards,

	Shlomi Fish

>
>
>
> To unsubscribe from this group, send an email to:
> [email-masked]
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>

----------------------------------------------------------------------
Shlomi Fish        [email-masked]
Home Page:         http://t2.technion.ac.il/~shlomif/
Home E-mail:       [email-masked]

"Let's suppose you have a table with 2^n cups..."
"Wait a second - is n a natural number?"

---

From: "Nadav Har'El" <[email-masked]>
Date: Wed, 5 Jun 2002 14:32:05 +0300

> premature to try to successfully analyze it.

I believe that "open-source" is much less a new idea than people realize.
It has existed for several hundred years, but had a slightly different
focus and went after a different name: "Academia".

After all, what is the academia, if not people who devote their life to
furthering humankind's knowledge and technology, while making all their
results open to their peers and the general public. Like open-source,
academic researchers must publish their results including all data needed
to replicate them in a convenient way. These academia guys get paid
to do whatever they wish, with the only criterion to continue their
employment is that they continue to produce more results that are useful
to the public (as judged by a decentralized peer-review system).
Every open-source programmer would love to be employed under such a contract.

Academia didn't need the *digital* revolution to appear, but it did require
both the printing press to be invented in the end of the 15th century (before
that, passing knowledge between academia groups, a.k.a. universities, was
too expensive) and it needed the Catholic Church to release its thousand-
year-long stronghold on the European people's minds.

> source. But I still find them amusing.

But who is Cowboy Neal? :)

-- 
Nadav Har'El                        |     Wednesday, Jun 5 2002, 25 Sivan 5762
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |"[I'm] so full of action, my name should
http://nadav.harel.org.il           |be a verb" -- Big Daddy Kane ("Raw", 1987)

---

From: Oleg Goldshmidt <[email-masked]>
Date: 05 Jun 2002 15:47:50 +0300

"Nadav Har'El" <[email-masked]> writes:

> to replicate them in a convenient way. 

And I suggest Feynmans "Cargo Cult Science" address (an adaptation is
included in "Surely You're Joking") as auxiliary reading on the topic
of reproducibility and integrity in science.

By the way, about half of "What Do *You* Care...", the second book of
Feynman's stories, is his fascinating account of the investigation of
Challenger disaster. IMHO, besides being a very interesting reading,
it has a lot to contribute to the art and science of how engineers
work on complex technological projects (and how managers screw up).

> These academia guys get paid to do whatever they wish, with the only
> criterion to continue their employment is that they continue to
> produce more results that are useful to the public (as judged by a
> decentralized peer-review system).  Every open-source programmer
> would love to be employed under such a contract.

This is much more complicated though the underlying idea of the
academia (don't forget the educational aspect) is expressed by the
above fairly well. I strongly suspect that the openness and the free
information flow in the academia are related to the long "time to
market" of the useful ideas generated by scientists.

One example of the long time to market of fundamental science from the
first half of the 20th century is Einstein. In the beginning of the
century, he developed several theories of fundamental importance,
including the special relativity theory (my drift is in the direction
of E=mc^2 and further to the Bomb), the theory of induced radiation
(leading to lasers and masers), and the theory of photoelectric effect
(for which he got his Nobel prize, that in those years recognized
practical importance of the achievement). The practical applications
of all these arrived in the forties and the fifties.

If you think that time to market is much shorter in our days, I doubt
that this would be confirmed by analysis. One of the late 20th century
biggest scientific splashes with obvious practical significance (and
arguably less fundamental than Einstein's works) is high-T_c
superconductivity. Let me remind you that by now it is 20 years old,
and it's not in the market yet. An interesting point is that Berdnotz
and Muller made the discovery in an IBM lab, and it was still as open
source as anything that comes out of a public university.

The long time to market has two immediate effects: 1) there is no
money to be gained in a couple of years - hence no business plan or
hopes to get rich quickly enough; 2) there is no other reward but
recognition of one's peers (forget about the general public - how many
people in the street can name a single scientist who got two Nobel
prizes?). Thus - publish, and make the skeptic colleagues believe you,
thus let them reproduce your results.

Now, what happens if there is some hope of a quick application and
money. This happens, for instance, in applied science. Appplied
science is in fact much bigger than fundamental, and is concentrated
in industrial labs rather than universities. There are first-rate
scientists in industry, no worse than in the academia, they just
assign a bigger weight to their creations being useful for someone
soon enough.

Industrial labs are not as open-source as universities. There is less
need - if something is done wrong then the product won't be as good as
it could, and competition will take care of the rest, but our basic
understanding of the Universe will not suffer as a result. And there
are rewards (and I don't mean just material ones, as anybody who saw a
client exclaiming "This is just what I need!" will confirm) beyong the
respect of one's peers. In effect, besides peers, the customers review
the result.

Actually, this separation of "academic", fundamental science, and
applied one, whether spontaneous or by design, makes a lot of sense.
While companies (except the likes of IBM) are not very interested in
furthering our understanding of the Universe, the publicly funded
academic institutions cannot support all the applied research, or all
the open-source programmers (though there are cases where universities
spawned startups). Some allplied research, and some open-source
programmers do work in the academia, of course.

Even inside the academia some (fields) are more equal than
others. Quite a few times I asked various scientists for their data
(and even source code) that they obtained with a lot of effort, and
invariably got the real McCoy by return email, no questions
asked. That's in physics. My impression is that the situation is quite
different in biology, for instance (this is based on reading and
hearsay - friends and relatives who work in biology). There are quite
a few branches of biology where there is always a chance that one's
research will lead to a breakthrough drug or treatment method, there
is money, patents, whatever. As a result, people often (not always)
are reluctant to share their results, let alone data, with potential
competitors. The free flow of information is impeded, and the whole
field is much more industrialized.

I know someone who was invited to lecture at two neighbouring biology
labs in a top-notch US institution. The two labs were sufficiently
close in research direction to be interested in this person's
research. However, neither knew that this person visited the other (on
successive days), because the personnel don't talk to each other. It
is very difficult for me to imagine that a solid state physics
department does not get invitations to nuclear physics seminars in any
respectable institution.

To boot, the labs in question are typical successful labs with 30-40
postdoctoral fellows each and budgets of hundreds of millions of
dollars. A scale quite unheard of in most other academic fields.

I hope my drift is clear (have you read it all? Wow!). One more
thought: publicly funded scientific projects (think NASA, ESA) are typically
required opening the data to the community after giving the principal
investigators (those are the guys whose proposals were accepted, who
designed and built the instruments, generally did all the work) a head
start of typically a few months, maybe a year. After that anyone can
access the data and do whatever research they want.

This is a great principle. I wonder if it can be generalized to
private businesses. Some relevant principles do exist. If you did your
homework on the legal meaning of confidentiality and non-competition
clauses that many of us have in their employment contracts, they have
(and if they don't, the courts will uphold) a time limit. Generally,
the law recognizes that after some time (12-18 months is typical)
whatever confidential information about technology or business that
you had becomes obsolete. I wonder if a hypothetical law that requires
open-sourcing older versions of software (e.g. those that are no
longer actively supported) is so farfetched.

Tired? Sorry... Got a meeting canceled today, decided to read my
mail... ;-)

-- 
Oleg Goldshmidt | [email-masked] 
"A sense of the fundamental decencies is parceled out unequally at birth."

---

From: Shlomi Fish <[email-masked]>
Date: Sat, 22 Jun 2002 08:29:50 +0300 (IDT)

> there is no clear way to get the number of developers.

If you are interested in an economical/psychological analysis of
open-source projects, I refer you to Eric Raymond's "The Cathedral and the
Bazaar" and its sequels. Raymond refers to the Mythical Man-Month a lot
and explains how he believes open-source solves some of the issues that
Brooks raised.

I found CatB to be very interesting now that I've read it. I remember that
I found it boring the first time I tried to read it, but it's actually
very enlightening.

Regards,

	Shlomi Fish