Subject: Reading Open Source Code

From: Shlomi Fish <[email-masked]>
Date: Fri, 26 Dec 2003 20:36:52 +0200 (IST)

Hi all!

How much do you read the source code of open source programs you are
interested in? I know some people recommend doing so. Paul Graham
recommended doing so in his "Hackers and Painters" article:

http://www.paulgraham.com/hp.html

He says that at his time there wasn't too much code available to learn
from, while today there's plenty of it out there, so hackers can benefit.

ESR, also recommends doing so in TAOUP.

I, personally, did not get too much to reading the code of programs, even
those that interest me a great deal. I did read the source code of some
programs, but it was mostly to achieve a certain end. For instance:

1. I analyzed the code of PySol and GNOME AisleRiot to write programs that
can generate their boards for input into Freecell Solver.

2. Having realized that GIMP did not have support for editing gradients
from scripts, I modified the code (several times for several different
versions), to do so, and naturally read it in the process.

3. I once tried to understand why a pam module was giving me trouble, and
read its code. (only to realize it was fixed in the CVS version, which was
released as a package for RedHat, but not as an official upgraded version)

4. Many times I delved deep into the KDE code trying to analyze why a
certain thing did not work as it should and what I should do to fix it.
(it's errors can be a bit cryptic).

5. I recently modified the code of some CPAN modules and read it in the
process.

6. Once Mark Jason Dominus gave a task to implement a "patch" program on
his Perl Quiz of the Week. To find out something, I read the source code
of GNU patch, and am not ashamed to admit that I couldn't make heads nor
tails of it.

7. I read the source code of Subversion while patching it, but naturally,
not always understood what to do correctly.

(maybe I exaggeratd in these examples).

The point is that I don't usually read code for fun, only when I wish to
achieve a certain end. I'm not saying it's OK that I don't, just that I
usually prefer writing code (and coming up with new techniques) than
spending time reading and understanding other people's code in the hope of
finding new techniques.

I do know of some people who regularly read the code of other people's
programs. One distinguished hacker I know, even commented on my code (but
only said she found it hard to understand). I could testify that I
found reading the code of various programs I did not write very difficult
at times as well.

One thought I have is that clueful hackers can infer the implementation of
many programs from their behaviour. Thus, reading the code is not always
time-worthy. However, there are many programs which I don't yet fully know
how they work (albeit may have a vague idea), and did not read the code so
far.

Regards,

	Shlomi Fish

----------------------------------------------------------------------
Shlomi Fish        [email-masked]
Home Page:         http://t2.technion.ac.il/~shlomif/

Writing a BitKeeper replacement is probably easier at this point than getting
its license changed.

	Matt Mackall on OFTC.net #offtopic.

---

From: Muli Ben-Yehuda <[email-masked]>
Date: Fri, 26 Dec 2003 20:44:48 +0200

On Fri, Dec 26, 2003 at 08:36:52PM +0200, Shlomi Fish wrote:

> interested in? 

Quite a lot, thank you. And how are you today?

> spending time reading and understanding other people's code in the hope of

"he who does not understand history, is bound to reinvent a square
wheel". 

> how they work (albeit may have a vague idea), and did not read the code so

Code reading should be a required class in computer science. 

Cheers, 
Muli 
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

"the nucleus of linux oscillates my world" - gccbot@#offtopic

---

From: Shlomi Fish <[email-masked]>
Date: Fri, 26 Dec 2003 21:02:41 +0200 (IST)

> Quite a lot, thank you. And how are you today?

Fine, thanks. Thanks for the verbosity of your answer.

> "he who does not understand history, is bound to reinvent a square

"He who does not understand history, is bound to reinvent an inferior
wheel, and gradually improve it, until it's better than its predecessors".

Seriously, that may or may not be the case. I'm all for code reuse and
starting from a working implementation, refactoring it or adapting it to
your needs, etc. etc. In this case, you do have to study the code or at
least its interface.

Now, if you want to create something similar to what already exists, you
are right that it is a good idea to ask around, or to inspect the code of
the similar implementations. But reading the code of programs that has
nothing to do with what you need to achieve, may or may not help you in
the future.

As an example of inventing a better wheel, I can (while risking your
corrolary of Godwin's Law) give Freecell Solver. I did not look for the
existence of previous efforts, and as a result believed (mistanely, BTW)
that I need to use meta-moves to solve Freecell with. Meta moves are a
series of more than one move used to achieve a certain goal in mind. (done
as one by the solver).

Now, meta-moves were not implemented in any previous solver, so it made my
solver quite unique. Furthermore, they later enabled configuring FCS to
solve a board in a multi-tude of different heuristics. Finally, a
certain human player who watched the solutions play said they were quite
innovative and interesting.

Had I simply looked for previous effort, I would have understood that a
less interesting approach would have sufficed and so the benefit of doing
it would have been lost.

> Code reading should be a required class in computer science.

Do you mean that people should be trained to read the code that other
people wrote (even if it's relatively uninteresting)? Or that people
should be given tasks to read and understand the source code of prominent,
high-quality, possibly very unique open-source packages? (or something
else entirely?)

Regards,

	Shlomi Fish

---

From: Muli Ben-Yehuda <[email-masked]>
Date: Fri, 26 Dec 2003 21:54:44 +0200

> Fine, thanks. Thanks for the verbosity of your answer.

Don't mention it. I'm on a touch-typing-diet. 

> people wrote (even if it's relatively uninteresting)? 

Yes. 

> Or that people

Yes. 

What I meant is that code is written for humans, not machines. Forcing
students to suffer through some bad good, while also hopefully getting
them to enjoy good code, should make that point clear. I am so sick
and tired of "professional" programmers who write code that if
translated to Hebrew would be the equivalent of "Dana Kama. Dana
Holechet. Yofi Dana!", if not complete gibberish. Code is poetry, and
should be treated as such. It should also be correct, unlike poetry,
but that's a different aspect I do not wish to discuss right now. I
hope that was verbose enough ;-)  

Cheers, 
Muli

---

From: Shlomi Fish <[email-masked]>
Date: Fri, 26 Dec 2003 22:32:36 +0200 (IST)

> Don't mention it. I'm on a touch-typing-diet.

Well, have an easy diet, which will hopefully last as long as it is
needed. (and no longer than that)

> What I meant is that code is written for humans, not machines.

Maybe you meant that it is written primarily for humans and secondly for
machines? (paraphrasing on Abelson & Sussman from SICP)

> Forcing
> students to suffer through some bad code, while also hopefully getting
> them to enjoy good code, should make that point clear.

A better idea may be to tell them to implement a new feature, re-organize
the code, or whatever based on an already working implementation. That way
they will have to read the code, but will also make them appreciate how
well or how bad it is written as.

> I am so sick
> Holechet. Yofi Dana!", if not complete gibberish.

Actually, I'm not ashamed to admit that some of the Perl and shell scripts
I wrote are not better than "Dana Kama, etc.". But you're in luck. Some of
the code I encountered would be translated to Hebrew as "Asiti eth
hatheatron hakeilo nehmad kazeh, veraiti sham shalosh anashim osim ehad
lasheni shikha kazoth".

> Code is poetry, and
> should be treated as such.

Well, not all code. Production code, yes. But throwaway code, or such
scripts that just accomplish one thing well, can be left as pretty ugly.

> It should also be correct, unlike poetry,

Yes, it was. Thanks.

Regards,

	Shlomi Fish

----------------------------------------------------------------------
Shlomi Fish        [email-masked]
Home Page:         http://t2.technion.ac.il/~shlomif/

You are banished! You are banished! You are banished!

Hey? I'm just kidding!

---

From: "Tzahi Fadida" <[email-masked]>
Date: Fri, 26 Dec 2003 23:52:54 +0200

> -----Original Message-----
> From: Shlomi Fish [mailto:[email-masked]]
> Sent: Friday, December 26, 2003 10:33 PM
> To: [email-masked]
> Subject: Re: [hackers-il] Re: Reading Open Source Code
>
>

I think the issue is indeed writting for other people. since we are a long
time away in computer years from assembly(which was quite fun at at
the time), nowdays you should think of what the next person that will
maintain or extend on your code will need.
I think that this issue is not given its rightful place at computer sciences.
I would refer you to "concepts of programming languages", SEBETA.
Aside from that, i think many open source projects would benefit from
a good design. the thing is, its not fun for a lot of people to design first
based on requirements, using a methdology. so they use XP "kinds" of
organic evolution to their software cycles.
However, this projects suffer from problems that comes from mistakes just
like that. Mistakes, that are apparent when the project needs to grow
beyond the 12 people stage.
I didn't suggest this before because i don't have the time, and the usual
response would be to do it yourself but i think that someone should
round up a group of people that would design projects based on
requirements for teams that do open source projects.

anyway, back to the main issue. I think that reading an open source
would be much easier if it was well designed using a proven methodology
and "enough" documented so that you can at least pinpoint the
whereabout of a glitch and fix it in half the time it took you.

> scripts that just accomplish one thing well, can be left as pretty ugly.

they can be maybe left ugly but things like "use strict" in perl are a must
if you want to take the code more seriously.

> To unsubscribe from this group, send an email to:
> [email-masked]
>
>
>
>
> Yahoo! Groups Links
>
> To visit your group on the web, go to:
>  http://groups.yahoo.com/group/hackers-il/
>
> To unsubscribe from this group, send an email to:
>  [email-masked]
>
> Your use of Yahoo! Groups is subject to:
>  http://docs.yahoo.com/info/terms/
>
>
>

---

From: Shlomi Fish <[email-masked]>
Date: Sat, 27 Dec 2003 01:39:55 +0200 (IST)

> maintain or extend on your code will need.

Correctly.

> I think that this issue is not given its rightful place at computer sciences.

Right again. Reading code and starting from a working codebase only to
extend it is rarely done in programming schools, which is a shame, because
it's a very needed skill.

> I would refer you to "concepts of programming languages", SEBETA.

What is it? A book? An article? Can you give a link? What is "SEBETA"?

> a good design.

Naturally. But so would a great deal of in-house or even commercial
software, some of which was heavily designed and written with a lot of
"red tape". If your project suffers from bad design, you should refactor
it, so it will have a good one.

Otherwise, (I may have misinterpreted you) it is always a good idea to
design and plan a change before you start hacking on it. Unless of course
we are talking about mindless logical transformations, where you just
change stuff in the code.

> the thing is, its not fun for a lot of people to design first

I'm not entirely familiar with Extreme Programming (= XP). I only heard
about it and attended the first half of (a very good) lecture about it.
What I do know is that in the open source world, the recommended way of
organizing and designing a software project is the "Bazaar" way as
described by ESR in "The Cathedral and the Bazaar" (which is an excellent
read even if you disagree with it). The Bazaar model seems very similar to
XP, but naturally accepts the constraints of an Internet-wide development
and that open source software is such that needs to be deployed on a large
number of variant systems.

I don't think either XP or the Bazaar model tell you not to design.
Rather, they tell you to do incremental design. I.e: if you want to add a
feature, then mentally plan how to integrate it into the core code without
breaking its modularity. If you feel the code is unmodular, refactor it
according to a planned design. In XP, there is a mandatory one design
meeting per day.

> beyond the 12 people stage.

Luckily, most projects don't need to grow beyond the 12 people stage. Most
projects have a core team of 1 or 2 developrs, and a halo of a bit more.
I'm personally involved in Subversion, where the number of contributors is
very large, and yet it's in pretty good shape. Still, the number of people
doing most of the work is relatively small.

I think open source has proved that its projects can scale to a large
number of developers, without grinding to a halt, or its code
deteriorating into a very low quality mess. The reasons why this is the
case were discussed in the "Go! Linux" panel, and also in many other
places and occasions.

> requirements for teams that do open source projects.

I don't quite understand this last sentence, but I'll try. I think you
mean that they would design tools to aid in quality control, software
management, etc. of open source projects. Like SourceForge/GForge, only on
steroids. Well, that may be a good idea, but of course will involve a lot
of work.

If you want these tools to be programming language-aware, then you'll need
to handle all the different languages out there. Good luck to whoever
choose to go this way.

> whereabout of a glitch and fix it in half the time it took you.

"proven methodology" - do such beasts exist? I think you've spent too much
time being an industrial engineer. Have you ever read "The Mythical
Man-Month"?

Also, define "adequately documented"? Do you want a Literate Programming
Style book documenting the program? Do you want a detailed architecture
document? Is a user's guide enough? Do you agree with what XP says that
even comments should usually be unnecessary and the code should be
self-explanatory?

I'm not saying a code should not be documented at all. However,
documenting a working code, and keeping the documentation in sync with the
code is very difficult. The best way to understand a code is to ask a
programmer who worked on it extensively.

I did encounter some times where code I did not write was very easy to
understand. One such case was a Perl module I needed for a CGI script I
maintain. The code proved to be not particularly extensible, but I was
able to tweak it to do what I needed quite easily. It was heavily
commented, but I felt that the comments kind of prevented the good flow
of reading it and were redundant. (so eat your hearts out - Python
bigots!)

> if you want to take the code more seriously.

Correct. I use "use strict" and "-w" on every serious code I write. I
don't usually on "perl -e " command line one liners, though, but they
fall into the second category.

Hey? I'm just kidding!

---

From: "Tzahi Fadida" <[email-masked]>
Date: Sat, 27 Dec 2003 02:54:57 +0200

> Sent: Saturday, December 27, 2003 1:40 AM
> Subject: RE: [hackers-il] Re: Reading Open Source Code
>
>

a book
http://www.amazon.com/exec/obidos/tg/detail/-/0201752956/qid=1072484373/sr=1-1/r
ef=sr_1_1/104-7844860-9557568?v=glance&s=books

> change stuff in the code.

yeah,
before, during and make sure that the maintainers also maintain the
design.

> according to a planned design. In XP, there is a mandatory one design

XP is against the traditional sens of design. I was attending a seminar
where one of the doctoral works was presented. it was on behavioural
stuff so that part isn't important here, however, they talked about
how they do it in companies and what are the limitations and offers to
make it better etc...
Now, here comes the interesting part - While i liked it in general for
small to medium term projects the inhouse staff of Information Systems
almost kicked them down the stairs. I myself feel that of course, for
big and "important" projects where you need a good design to refer
to where there is a problem this kind of methodology is out of place.
However, the throwaway with overall design and start programming
and document a bit inside the programm and in the end is what
most software houses do already and it works for the most simple
and non life threatening or most reliable projects.
The excuse is of course that the design is supposed to be recycled
several time during the software life time and no one does that because
of budgetery concerns. To the scientific sense this of course stinks but
the commercial world do what they want and what is considered better
for them.
I must stress here though that there are many militairy and proven "safe"
systems that must have a good design. one example would be the black
boxes in airplanes or in the robots they send to mars. This can be done
using mathematical models that can be checked by a computer. Of course
it would be impossible for large scale projects to be built like this.
So, the next best thing is to take something like GAIA for agents,
or UML for small projects, etc... to each appropriate field of software,
and make sure it works.

THE MAIN PROBLEM i see here is that in CS and any other faculty in the
technion, i never heard of the IEEE/ACM Code of Ethics(c IEEE/ACM
1999). this code is like the doctors - not to do harm and negligence.
see a Book:
Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1

> case were discussed in the "Go! Linux" panel, and also in many other

I'll have to check that. I guess good will and "this is my baby and i know
anything about it"  is a good part of that.

> to handle all the different languages out there. Good luck to whoever

Though a really good idea, and i don't think its really possible using
regular methodologies like UML for reasons that would take a course time
to explain. Mainly due to the many models. Orthogonality plays a crucial
roles in this matters.
Anyway the IE IS area has a new methodology called OPM, which
I must state here first that its patented and thus i have personal feelings
about it, though it maybe continues our last discussion of universities
internal funding sources. this methodology is the new wave suppose
to supercede UML using one model for all, and supposed to be able
to one model - > every programming language.
And I for one, believe it is possible.
Ok, comming back to the issue, i was talking about OUTSOURCING
open source style. i.e. people with design skills at different areas
with different methodologies would design projects with the base
requirements and maybe a meeting place between designers and
project developers that will commision designs and refactoring from
them.
The main problem i see that this is supposed to solve, is the experienced
designer issues, and the knowledge time - to - develop issues.

> time being an industrial engineer. Have you ever read "The Mythical

yes there are, though rare. mathematical models which are "safe" can
be proved, however these are for small scale. Mostly a few lines of code.
No, i was talking about how much time they exists and if they have a
record track of successful usage.

> even comments should usually be unnecessary and the code should be

don't count of self-explanatory, this rarely happen when a multitude of
people write different bits of code.
i can't define here what is adequate precisely because every project
is different. I can only say that it should be adequate in the sense
that further down the line the code could be maintained using this
documentation.
This is the XP way of thinking - "adequate enough".
I am still ambivalent about this so i can't be accurate.

> programmer who worked on it extensively.

yes. again you are quoting XP, and i'll requote traditional methodologies.

> of reading it and were redundant. (so eat your hearts out - Python

yes, if the code is self explenatory around the 200 lines or so, then
there is no need to comment every line. java's javadoc is very good
to my thinking.

> fall into the second category.

right

---

From: "Nadav Har'El" <[email-masked]>
Date: Sat, 27 Dec 2003 11:41:08 +0200

> recommended doing so in his "Hackers and Painters" article:

At this stage, I don't normally say to myself "Wow, this Mozilla program is
nice and I use it everyday, why don't I just read its entire code". But there
are good reasons why I have read source code in the past, and continue to do
so once in a while:

1. To learn.

   I started learning Unix by reading source code - or actually, by *typing*
   source code, because no network was available to me and the only source
   code I had access to was printed in books (most notably, Kernighan & Pike's
   "The Unix Programming Environment" and Kernighan & Ritchie's "The C
   Programming Language".
   After reading (and typing) all the programs in those books, I sometimes
   took source code of other Unix programs (which was available at the time)
   to learn from it techniques I didn't yet know, such as how to write programs
   for the graphical terminal we had back then (The AT&T 5620, a.k.a. the
   jerq). Today I still read available code (usually free software, of course)
   when I want to learn new techniques - in addition to reading the relevant
   manuals, of course.

2. To change.

   The most important reason to read someone else's code is if you want to
   change it. My first experience of this sort was 18 years ago (wow, time
   flies...) when I thought that BASIC was too lacking a language, and I
   thought I knew better and wanted to invent a new feature in BASIC. So I
   found the source code (in C, running on UNIX) of a BASIC interpreter, and
   with some help from my father (I was a lousy programmer at the time) I
   added a new feature to that BASIC interpreter. That interpreter was not
   free software, so I had nothing to do with my "patch", and it was lost
   in the mists of time (or in my infamous "rm" accident).
   More recent experiences of this sort involved me sending patches (and
   them being accepted) for problems that bothered me in Gzip, Apache, and
   OpenSSL. Fixing these problems naturally required reading these projects'
   source code - usually not the entire source code but only certain (but
   sometimes substantial) parts.

3. To do better.

   Think you can write a faster web-server from Apache? Think you can write
   a version of OpenOffice which takes less memory? You better look into the
   code and check if your hunches are write, or whether the tricks you were
   thinking of implementing in your new super-duper project are already in
   the familiar code, and they are not the silver bullets you hoped for.

-- 
Nadav Har'El                        |      Saturday, Dec 27 2003, 2 Tevet 5764
[email-masked]             |-----------------------------------------
Phone: +972-53-790466, ICQ 13349191 |My password is my dog's name. His name is
http://nadav.harel.org.il           |a#j!4@h, but I change it every month.

---

From: Orna Agmon <[email-masked]>
Date: Sat, 27 Dec 2003 12:03:36 +0200 (IST)

>    the familiar code, and they are not the silver bullets you hoped for.

I believe most people follow this rational, if they actually do reas
source code. And the question which bothers me now is: We have an
often-assumed assumption, which we take for granted:

If it is open-source, it must be safe, it cannot have Trojans or otherwise
be malicious (to the one running it), because somewhere out there someone
who understands the code other than the original programmer would have
noticed and screamed about it to the world. (I do not count mistakes here-
errors happen).

If everybody behaves like this, then this assumption is totally out of
order.

Orna.

---

From: Muli Ben-Yehuda <[email-masked]>
Date: Sat, 27 Dec 2003 12:22:20 +0200

> If everybody behaves like this, then this assumption is totally out of

It really depends on the project, the number of active developers,
their attitude towards reviewing each other's source code and the
ammount of community involvement. Compare and contrast: 

- someone tried to inject a backdoor to the Linux kernel. It was
immediately discovered. 

- there was a database (informix?) that was released as open source a
few years ago. Only after a couple of years of it being "open" did
someone discover that the original programmer left a huge backdoor. 

Therefore, just assuming that if something is open it contains no
surprises is dangerous. 

Cheers, 
Muli

---

From: guy keren <[email-masked]>
Date: Sat, 27 Dec 2003 15:31:59 +0200 (IST)

> 3. To do better.

add to that "4. to find features". i often find myself reading the source
code in order to find undocumented features. and with open-source - there
is often rather sparse documentation, and hence a lot of undocumented
features.

this also includes understanding how some feature works, in order to find
why my program does not work well or properly.

this could include reading the kernel's source in order to find what
exactly some poorly documented ioctl() do, or reading the source of a
networking library in order to find why it sometimes aborts the main loop.

i got into the habbit of reading source to the leevl that i sometimes do
that without first checking if the FAQ contains the question i want an
answer for.

and who said there is no teaching of "reading source code" in the academi?
what is the "intorodction to operating systems" course, of the technion's
CS department, if not a course in reading source code, finding bugs and
adding features? ofcourse, you got to want to learn that well, in order to
learn that well. and you need to have good homework partners to do that
"standard questions" for you, in order to have the time to look for the
more subtle bugs.

-- 
guy

"For world domination - press 1,
 or dial 0, and please hold, for the creator." -- nob o. dy

---

From: Shlomi Fish <[email-masked]>
Date: Sat, 27 Dec 2003 14:53:42 +0200 (IST)

> networking library in order to find why it sometimes aborts the main loop.

Heh, me too. Use the source, Luke!

> adding features?

You mean the course that used to teach XINU and now User-mode-Linux? I
suppose that does qualify. However, in the EE department, there's a
different corresponding course, in which we just have to write various
user-land programs that use the system facilities. (a shell, a malloc/free
implementation, etc.) It's not a bad course, but we don't get to do any
kernel hacking when starting from a working code base.

I still think it's not enough. The purpose of the OS course is to teach
about OS design, and its side-effect is reading and extending code. But we
could probably use a dedicated course for just that.

Regards,

	Shlomi Fish

Hey? I'm just kidding!

---

From: Shlomi Fish <[email-masked]>
Date: Sat, 27 Dec 2003 20:49:22 +0200 (IST)

> ef=sr_1_1/104-7844860-9557568?v=glance&s=books

OK, I may check it out eventually. Is it available in Technion libraries?

> before, during and make sure that the maintainers also maintain the

Or make it even more modular to form an even better design. Software is
not static and what is an adequate design for a software that does A,B and
C, may not necessarily be good for a software that needs to do A,B,C and
D.

> to where there is a problem this kind of methodology is out of place.

Perhaps. Albeit a great deal of large scale open source projects have
emerged recently and have been developed in a Bazaar way. Usually, you
divide such projects into sub-components and define well-defined
interfaces for making use of each component. Then the internal design of
each component is left as the "private zone" of its core developers, and
the other developers of the project need not be concerned about it.

> and non life threatening or most reliable projects.

True.

> the commercial world do what they want and what is considered better

Right.

> Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1

I saw the IEEE code of ethics once in writing. Can't remember it all.
However, I'm not sure it says anything specific about which
software engineering methodologies to use.

> anything about it"  is a good part of that.

Some of the points that were raised (by the Go! Linux Panelers and
others):

1. By splitting the code into components it is possible to divide the work
and make sure it scales.
2. A great deal of code is not automatically accepted into a project and
its correctness, style and quality is criticized by many reviewers before
it is accepted.
3. In many software shops, programmers are terriorial about their code.
Those shops in which programmers encourage others to take a look at their
code, look for potential bugs and suggest or implement improvements,
eventually see a boast in their productivity. Open Source is like that
only on steroids.
4. (my own) In many open source projects there is less "red tape" than in
those projects that are run by software houses. Thus, a programmer can
achieve more as he is not restricted by factors induced by his workplace.
5. (Homesteading the Noosphere). It was shown that incentive pay (bonuses,
etc.), actually reduce the output of programmers, and workers of other
professions in which there is a genuine mental effort involved in the
work. In fact, what can encourage them is letting them work on things
they desire to work on. Thus, open source hackers who receive little
monetary incentive and work on what they like embody the ideal of this
approach.

> project developers that will commision designs and refactoring from

This could be a problem for one good reason: as ESR identifies in:

http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar01s10.html

You have to start a project by writing a working version of the code. It
may not have to do everything the project needs to do in its final
version, but it has to prove that the source code has a future in it. Most
projects that start by forming a mailing list and writing a future arch
doc or whitepaper wither away into oblivion, without having too much to
show for.

I can testify from experience that this was the case for my projects.
Thus, I always make sure that if I wish to release a project into the
wild, I first sit down and write a version that achieves a small subset of
the planned featureset.

However, your idea may hold for re-designing projects, refactoring them or
improving them. I know that Subversion started from a vapourware design
that a certain RedHat employee designed, and let others implement. The
latter is well-known for this tendency.

> record track of successful usage.

Well, the Bazaar model so far proved itself spectacularily, at least
according to the eyes of observers. I don't know about XP, though there
are reports it had been successful for certain projects in its
target niche. It reminds me of the book "Facts and Fallacies of Software
Engineering" which aims to analyze which trends increase productivity and
which don't based on research and statistics:

http://www.amazon.com/exec/obidos/tg/detail/-/0321117425/joelonsoftware/102-6670515-1781755

> people write different bits of code.

Probably true. That's why many software projects have style guidelines and
review every contribution before its inclusion.

> I am still ambivalent about this so i can't be accurate.

OK. Eventually, by experiencing enough with the code, you can eventually
understand even the most obfuscated one. Even a code with tons of
comments, meaningful variable names, a consistent style convention, an
organized architecture document, API documentation, etc. may not
necessarily be easy to understand. And like I said I many times
encountered code that had only a small subset of these things, and was
very clear and easy to modify.

I cannot testify I've been a saint in this regard. Between version 2.4.x
and 2.6.x of Freecell Solver, I added a great deal of optimizations
some of which I feel may have decreased readability a great deal. But they
did made the code 1.5 times faster, which is still significant.

> yes. again you are quoting XP, and i'll requote traditional methodologies.

I am not quoting XP as I'm not entirely familiar with it. I'm speaking
from experience and intuition. Maybe my opinions may seem XPish to you,
but maybe that's what my experience has taught me.

> there is no need to comment every line. java's javadoc is very good

I've seen some pretty huge codebases that were self-explanatory. I found
the GIMP (or at least its gradient editing functionality which is it what
I've focused on for the time being) very legible and easy to modify. And
that was without reading any description of its internals.

So, to conclude: some code will be perfectly legible for experienced
programmers even with little documentation. Some code will be very hard to
understand even with mountains of documentation. (which many programmers
will feel like they don't have time to read). Documentation is not an
excuse to write hard-to-understand code, and cannot compensate for its
lack of readability. However, it could help sometimes.

What goes without saying is that a software needs to have proper user/API
documentation, and that its behaviour should correspond to the user model.
This is sadly something that a great deal of open source software does not
yet implement to an adequate level.

Regards,

	Shlomi Fish

Hey? I'm just kidding!

---

From: Tzafrir Cohen <[email-masked]>
Date: Sat, 27 Dec 2003 21:27:00 +0200

> > noticed and screamed about it to the world. (I do not count mistakes here-

Actually: it means that if this a project with its source code avilable
then sneaking in backdoors is more difficult.

But then again, I can make trojaned binary of my program avilable on the
web site without changing the source code.

> - someone tried to inject a backdoor to the Linux kernel. It was

Actually: it was never officially submitted, and in the place it was
added it wouldn't have had much effect. So it was not cought before 
the "many eyeballs" of kernel programmers would have had a chance to
spot it.

> someone discover that the original programmer left a huge backdoor. 

The database is firebird, and the back-door was spotted about half a
year after the release of the code.

> Therefore, just assuming that if something is open it contains no

Fully agree.

-- 
Tzafrir Cohen                       +---------------------------+
http://www.technion.ac.il/~tzafrir/ |vim is a mutt's best friend|
mailto:[email-masked]       +---------------------------+

---

From: "Tzahi Fadida" <[email-masked]>
Date: Sun, 28 Dec 2003 11:02:10 +0200

> Sent: Saturday, December 27, 2003 8:49 PM

maybe, i myself bought it. maybe you can find something similar. among
others its about readability writability and how you can define criterias to
judge programming languages when you come to evaluate if you need
to use it.

> C, may not necessarily be good for a software that needs to do A,B,C and

well, scalability is always good. the reason UML, which is the most widely
used, doesn't seem to be loved by many is because they fail to
understand it does not scale. the modularity issue is just an illusion,
and the "package" attempt at one of the models is pathetic.
A good scaling must come with more attention to details.
again, i don't say to use it with open source because of patents and
licensing issue, but see OPCAT at the IE IS department.
I wish we had something more free, or maybe there is and i don't know
about it. if not, please enlighten me.

> the other developers of the project need not be concerned about it.

yeah, the blackbox idea. however, this is an ad hoc methods. i.e.
if for example someone decided to use one driver, and then someone
decided to use a different one, and now you have to install 2 drivers.
etc... you need to have an overall look. you can design using for example,
the class model in UML and as you probably know, its not getting to
specifics there, and so 90% is left for the "private zone" to design.

> software engineering methodologies to use.

of course not, its a code of ethics, but it certainly defines what is
acceptable.
it even starts: "those who contribute by direct participation or by
teaching, to the analysis, specification, design, development,certification,
maintainence...."
http://www.acm.org/serving/se/code.htm

> monetary incentive and work on what they like embody the ideal of this

the last,5, is a known phenomena in psychology, or at lest that's what
we learned in one of the cognitive courses i took.
I am a bit skeptic about the conclusion, because it does not solve anything
but avoid the phenomena. on the same respect, if you lower a person's
salary or cancel a bonus, you will get the effect of more productivity.
should that mean that we should start dissing the work of open source
developers? :)

> doc or whitepaper wither away into oblivion, without having too much to

well, this is a murphy's law kind of thing. usually inexperienced or
experienced people from different reasons don't have the time or
streangth to continue a project and what the quoted reason means is
that the left overs can be continued by someone else, but if there aren't
any then its a problem.
I think its negligable, the same can be argued for someone who leaves
a message about an idea without implementing it because he doesn't have
the time. (like i did)

> wild, I first sit down and write a version that achieves a small subset of

i know the feeling, but its a psychological thing and not a scientific reason.
i even skipped a few myself, but not before i told them in some mailinglist
so others, if they have the streangth, can implement.
an example, is the idea that not using an open source documents
and software in govrenment facilities is against the freedom of information
law before the committe about this thing convened for the first time. i
corresponded with rabia and others and even got the law copy.
but of course i didn't try to get on the amuta wagon when a few of us
raised the issue of amuta several times in linux-il. i don't have the time, but
it doesn't mean someone else can't do it.

> latter is well-known for this tendency.

well, my idea is about who ever wants a design for whatever reason can
commision one from the community. so yeah.

> http://www.amazon.com/exec/obidos/tg/detail/-/0321117425/joelonsoftwar

i personally doubt the empirical research of such things because they
tend to be biased to a certain methodology. usually because, for example,
XP was concived by companies (I think IBM or HP, can't remember) for
companies. Companies usually like to cut corners, and so i would refer
those companies to the ACM code of ethics.

> review every contribution before its inclusion.

right.

> did made the code 1.5 times faster, which is still significant.

right.

> but maybe that's what my experience has taught me.

XP was developed from experience rather than science, and so it seems
to integrate perfectly into one's practicies. however, it should be carefully
checked when used.

> yet implement to an adequate level.

right.

---

From: Shlomi Fish <[email-masked]>
Date: Sun, 28 Dec 2003 15:54:25 +0200 (IST)

> about it. if not, please enlighten me.

I felt UML was too nitty-gritty as well. Too many diagrams to represent
the micro-management of the code instead of the really important stuff.
Thanks for enlightening me in this regard!

But you can do good design without any official diagrams and with only
a-posteriori diagrams that are given to illustrate various points (and
need not necessary be anything official like UML).

> specifics there, and so 90% is left for the "private zone" to design.

Well, this is one problem that is prominent in the open source world any
way you look at it. There's duplication of work anywhere. What's good for
Linux needs to be duplicated in the BSDs. What's good for GNOME needs to
be implemented in KDE as well. We have a gazillion Wikis, window
managers, editors, soundwave editors, bug tracking systems, CMSes, etc.
etc. Of course, this problem is not restricted to the open source world.
For instance Microsoft have released a large number of database packages:

ODBC, RDO, DAO, ADO, OLEDB, ADO.NET

All by the same company under the same roof. So I guess you can't really
avoid it, albeit can do your best to minimize it. Perhaps the only true
way to do so is to break backwards compatiblity if necessary.

> http://www.acm.org/serving/se/code.htm

Right. But I'm not sure XP will be frowned upon it in much regard. After
all it has test-driven development, pair programming (and so catching a
lot of bugs, and good peer review), refactoring and other elements that
increase the quality of the code. I'm not sure if it's entirely suitable
for developing software to guide missiles, but it does stress quality.

> should that mean that we should start dissing the work of open source

Maybe you should read "Homesteading the Noosphere" to get more
understanding. Lowering a salary of a programmer who gets paid to do
programming, will likely be a demotivator as well, because he'll feel that
he's no good. An increase in salary, can in fact motivate if presented
correctly, or if done as part of a desireable promotion.

I agree that when ESR is taking this phenomenon to extreme, it may not
necessarily be a correct thing to do scientifically. But it's might as
well have a grain of truth to it.

Right.

> so others, if they have the streangth, can implement.

Well, you are right that it is mostly psychological and not scientific.
Some projects that started with a mailing list and a planningware
announcement are flourishing now. Many others which still have a working
codebase are now neglected.

In any case, be aware that as nice ideas are. They don't require a lot of
effort. An individual needs much more effort in developing a good product
out of it, mass producing it and mass distributing it. (or whatever is the
equivalent for software).

> it doesn't mean someone else can't do it.

Well, if you're not going to do it, either someone else will and get most
of the credit, or it won't get done at all. We all share the same
existence.

> commision one from the community. so yeah.

Well, I suppose you can post a message to mailing lists or web forums
asking for a good design for a program that does A,B and C. Many people
will be happy to suggest and discuss such good designs.

If you want to set up a site, then a rudimentary one using some web-log
system (Slash, Post-Nuke, Scoop, etc.) should be very easy to do and
you'll only need hosting. It may actually prove to be quite popular.

> those companies to the ACM code of ethics.

Well, this book is supposed to be based on a great deal of researches
conducted. I did not read it, so I cannot testify for how well it is
based.

> to integrate perfectly into one's practicies. however, it should be carefully

"experience rather than science". Isn't science supposed to be
experimental? If someone has a theory, he needs to prove it actually works
in the real world. This is very difficult to do in software engineering
and team management where the success of the team is dependant on a great
deal of many factors. But it is still necessary to show it is.

I don't really care if a theory was deduced from experiments and then
explained theoretically or vice versa. I know that in physics, both thing
s were witnessed. And XP was eventually explained why it is working.
Albeit, with such sciences that involve a lot of human interaction, it is
hard to satisfy all researchers.

Regards,

	Shlomi Fish

Hey? I'm just kidding!