Subject: Rules for Open Source development (the serious version)

From: Chen Shapira <[email-masked]>
Date: Sun, 20 Jan 2002 17:04:26 +0200

Hi,

Joel Spolesky from http://www.joelonsoftware.com has suggested a test for
checking how well managed a software company is.
You can find the test here:
http://www.joelonsoftware.com/articles/fog0000000043.html.

It features 12 yes/no questions, and the more "yes" you have, the better off
you are. It seems like a reasonable test, although it does have many flaws
(lack of severity is one problem. I'd rather work at a company with cubicles
than work at a company with no schedule).

Anyway, most of those things don't really apply to open-source projects, for
many diffrent reasons.
For one thing, release dates aren't nearly as important, and code is
developed in a slower pace, so nightly builds are usually not necessary.
QA and usability tests are another sour issue. You can't really hire them,
so you'll probably not have professional testers. Using the best tools and
giving developers quite working enviroment, doesn't really apply either.

Moreover, there are many things the project manamgement will need, that
corperate managment can safely ignore. Making developers feel good is an
obvios one. Something about good communication channels is also important.

BTW. Does anyone know of a software project that has professional QA team,
with the usual practices (detailed test plans, regressions, forcing the
developers to stop adding features and fix a showstopper)?
Do QA teams in open-source projects have enough political power over
developers ("Head of QA said we can't release until we have these bugs
fixed")?

Thanks,
Chen.

---

From: Tzafrir Cohen <[email-masked]>
Date: Sun, 20 Jan 2002 17:40:08 +0200 (IST)

On Sun, 20 Jan 2002, Chen Shapira wrote:

> For one thing, release dates aren't nearly as important,

Some projects treat release dates and release plans more seriously than
others (compare KDE to mozilla).

Note that linux distributions are sometimes include pre-releases of
certain packages. So the fact that you didn't release your package does
not prevent others from using it.

> and code is

Some projects have them, though. Two I can recall are mozilla and abiword.

You can also treat debian's "unstable" and "testing" constantly-updating
branches (as well as redhat rawhide and mandrake cooker) as sort of
"nightly builds".

> QA and usability tests are another sour issue. You can't really hire them,

One exception to that is the usability team that Sun has working on gnome.

> developers ("Head of QA said we can't release untilwe have these bugs
> fixed")?

Debian, is probably the largest and most distributed open-source project.
They do have a seperate qa team, and a policy of "no showstopper bugs",
but I'm not sure. Maybe check:
http://www.debian.org/doc/debian-policy/

Mozilla is also a project with a QA team. Mozilla is originally a more
centralized project.

-- 
Tzafrir Cohen                        /"\
mailto:[email-masked]        \ /  ASCII Ribbon Campaign
Taub 229, 972-4-829-3942,             X   Against  HTML  Mail
http://www.technion.ac.il/~tzafrir   / \

---

From: Adi Stav <[email-masked]>
Date: Sun, 20 Jan 2002 21:26:42 +0200

> not prevent others from using it.

I think there's a difference between a release plan and a development
plan. The first deals with inclusion, integration or stabilization of 
already developed features while the second deals also with 
developing the features to start with. 

> branches (as well as redhat rawhide and mandrake cooker) as sort of

Also BSDs etc. I agree, a lot of projects have these. Also Linux has
builds, although not precisely daily, but we rarely have less than 
two or three prepatches a week.

> One exception to that is the usability team that Sun has working on gnome.

Sun's usability team work is fairly separated from GNOME itself.

Also consider the big testing lab (forgot its name) that passes Linux
kernels thorugh standard tests.

A /lot/ of individuals do dedicated testing for a lot of projects. 
E.g., Telsa Gwynne for GNOME, or our very own Hetz Ben Hamo for just 
about everything under the sun.

I think an important point is that all these testers, including 
Linux's testing lab and Sun's usability lab, are just another user 
of the software. They do not necessarily have special status as far 
as the project as a whole is concerned.

> http://www.debian.org/doc/debian-policy/

I find that Debian's QA team's name is misleading because it has 
somewhat different roles from what QA traditionally does, and 
testing is not one of the important ones. Debian doesn't really
need a dedicated QA team because of all of its developers dogfooding
it constantly.

"No showstopper bugs" is a good example though... But in Debian it
goes on a per-package basis. It's not that the distribution as a 
whole fail, just that a specific package will not get out of the
"unstable" distribution if it has "release critical" bugs. That's
usually a pretty good leverage on the developer, but it relies on
Debian being composed of a large number of relatively independent 
packages.

> Mozilla is also a project with a QA team. Mozilla is originally a more

Mozilla and to a larger extent OpenOffice are very different and I
am not sure they apply. Perhaps they can be considered regular 
commercial products that just happen to have open-source licenses.

---

From: mulix <[email-masked]>
Date: Sun, 20 Jan 2002 21:10:09 +0200 (IST)

> developers ("Head of QA said we can't release until we have these bugs

gcc has a rigorous test suite, a release manager for each release and
very strict cvs checkin procedures. see http://gcc.gnu.org/develop.html
for example.
-- 
mulix

http://vipe.technion.ac.il/~mulix/
http://syscalltrack.sf.net/

---

From: Shlomi Fish <[email-masked]>
Date: Tue, 22 Jan 2002 16:31:54 +0200 (IST)

1. Do you use source control?

Very important. Perhaps even more than commercial software because the
developers are not concentrated in one place.

If the code is relatively small and there's only one person hacking on it,
it can be ignored. Otherwise, CVS or whatever is a must.

2. Can you make  build in one step?

Also applicable. If the following commands:

config-foo-app.sh
make
make install

install everything on your system - you're on. Otherwise, users may find
it tedious to keep up with the various releases of the programs.

3. Do you make daily builds?

Since the effort is not concentrated "daily" is not necessary. Obviously,
developers should from time to time checkout the code from the CVS and
compile it, to see if it works OK. So the concept still stands but with a
slight modification.

4. Do you have a bug database?

Very important. (need I say more)

5. Do you fix bugs before writing new code?

Also very important. It seems that Linus Torvalds have decided to go
against this paradigm in the 2.5.x tree. Then, Dave Jones decided to
release a set of patches that fixes the bug-fixes that were found in the
2.4.x tree.

6. Do you have an up-to-date schedule?

KDE has a schedule plan for its releases, although it too cannot tell in
advance when is the next second digit release going to be. Many commercial
projects with tight release schedules ship versions which are not usable.
The various Windows' OSes are especially notorious for this.

My opinion is that this item is not very applicable, except for the last
days of a development version, when one should make a date for a feature
freeze, bug-fix freeze, release announcement, etc.

7. Do you have a SPEC?

Very important for most projects.

8. Do programmers have quiet working conditions?

Since the programmers are scattered around the globe, and each one can
dictate his own preferable working conditions, this item should not be
taken into account.

9. Do you have the best tools money can buy?

I think it can be skipped, since every programmer can use his preferable
tools. If the project can only be compiled with specific tools, then there
might be a problem.

10. Do you have testers?

The OS assumption is that the developers and to lesser extent the users
are the testers. Therefore, I'm not sure if it's entirely applicable.

Having dedicated testers is a good idea, but so is having some users who
will be willing to use the CVS version.

11. Do new candidates write code during their interview?

Obviously not applicable. Since an OS project can use any help it can get,
it does not distinguish between developers who wish to join it. What can
be done is that if a developer becomes a troll, he can be banned from it.
Usually, it does not happen.

12. Do you do hallway usability testing?

One should distinguish between what the program is:
1. A command-line one.
2. A GUI one.
3. A library.

(did I miss anything?)

If #1 - then one should make sure the program uses more-or-less standard
flags, behaves in a UNIXish way, has a manual etc. And the fact is, that
wannabe users, are bound to ask you how to operate it. ;-)

If #2 - obviously it should be done, or else users will have a hard time.

If #3 - make sure that the library follows command coding conventions, and
has a consistent API, and you should be OK.

--------

Other than those the following should be added:

1. A mailing list, IRC channel, and E-mail address for questions, etc. -

A project cannot scale without them

I had some other things in mind, but I forgot.

Regards,

	Shlomi Fish

----------------------------------------------------------------------
Shlomi Fish        [email-masked]
Home Page:         http://t2.technion.ac.il/~shlomif/
Home E-mail:       [email-masked]

"Let's suppose you have a table with 2^n cups..."
"Wait a second - is n a natural number?"

---

From: "Nadav Har'El" <[email-masked]>
Date: Tue, 22 Jan 2002 16:48:56 +0200

> dictate his own preferable working conditions, this item should not be

I'm not sure I agree. If your open-source programmers are poor students
forced to work in noisy, busy, computer labs, or live with their parents
and have to wait until their parents/siblings get off the single computer
that they share, then your open-source project will not progress as well
it could have.
Even open-source projects will benefit from programmers having a lot of
quiet and free time. (in a commercial setting, the "free time" is implied,
since your employer buys it from you).

> tools. If the project can only be compiled with specific tools, then there

Not quite... I bet you're only thinking of software tools. But what about
hardware? I've seen in so many occasions comments in free software projects
like "This would run on a Sun machine if we only had access to one...", or
"We don't support SMP because none of our developers can afford an SMP
machine", or things like that. You can say similar things about expensive
networking equipment, fat network connections, and things like that - most
free-software developers don't have access to these.

>...

Yes, but watch out: input from some testers is worthless (they don't know
what they're talking about and confuse their own misconfigurations and such
with bugs in your program), while input from others is valuable. If you
can't choose (or hire) your testers, you'll need to read a lot of crap
input, and seperate the wheat from the chaff yourself.

> Usually, it does not happen.

Again, your project can't just "use any help it can get". Some people
code so badly, and require so much "babysitting", that you'd be better
off without their help. Usually, you'll trust someone whose code you've
seen before - e.g., they sent you a patch previously. So in that case,
you do ask candidates to write code during their "interview"!

-- 
Nadav Har'El                        |      Tuesday, Jan 22 2002, 9 Shevat 5762
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Unix is user friendly - it's just picky
http://nadav.harel.org.il           |about it's friends.

---

From: "Nadav Har'El" <[email-masked]>
Date: Tue, 22 Jan 2002 23:06:56 +0200

> free-software developers don't have access to these.

Here's one example (out of many I've seen). Question 8.12 of the OpenBSD
FAQ:

   8.12 - Will OpenBSD run on multiprocessor machines?

   No. OpenBSD/i386 will not make use of multiple processors, but will run
   using one processor on a multi-processor system board. OpenBSD/sparc will
   not run on a multi-processor sparc system at all. No other platforms
   currently support multi-processor capabilities.

   The reason is quite simple: there are not enough developers who have access
   to MP machines. If you want to donate MP hardware, please refer to
   http://www.openbsd.org/donations.html. Our developers are currently asking
   for i386, sparc, hppa, and other multi-processor hardware donations to
   begin SMP support. A project to bring multi-processor support to OpenBSD
   is underway, and progress can be check at http://www.openbsd.org/smp.html.

By the way, do you think that CVS is the best collaboration tool that money
could buy? Sure, it does the job well, but sometimes you want to make a
phone call, do a video-conference or even fly over for a few days of
intense discussions with your coworkers. This takes real money, not just
good will, unfortunately.

-- 
Nadav Har'El                        |     Tuesday, Jan 22 2002, 10 Shevat 5762
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |How long a minute depends on what side of
http://nadav.harel.org.il           |the bathroom door you're on.

---

From: Shlomi Fish <[email-masked]>
Date: Wed, 23 Jan 2002 08:35:01 +0200 (IST)

>    is underway, and progress can be check at http://www.openbsd.org/smp.html.

Speaking of which - it reminds me that both the IP-Noise simulator my
partner and I wrote and Haifux' Sys-Call-Track were not yet tested on an
SMP machine. The reason is that we don't have access to a dual (or Quad)
Pentium. We tried to pull some string but we are not successful yet.

I don't see any inherent problem in the code that should result in
corrupting the kernel. But of course it should always be tested.

In any case, I suppose a project could always use more money in order to
support its developers. But usually, projects can get by with very little,
and should not depend on any positive money flow.

> good will, unfortunately.

Assuming we are talking about an OS projects whose developers are
scattered around the world, then CVS (or subversion or whatever) is not a
complete answer. With mailing lists and IRC channels, the situation
becomes somewhat bearable, but most large projects find a need for
real-life conferences and meetings in which many developers meet together
and discuss where the project is going to.

I again had an idea for something to add, but I again forgot it. :-)

Regards,

	Shlomi Fish

> To unsubscribe from this group, send an email to:
> [email-masked]
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>

---

From: Adi Stav <[email-masked]>
Date: Wed, 23 Jan 2002 13:42:04 +0200

Let's say that Joel's rules represent "common wisdom" in the 
commercial software development world. They are not necessarily
uncontroversial in the academic or open source worlds, and not 
universally applicable, but most /commercial developers/ try to 
follow them. Where they do not it is usually due to external 
pressure (e.g., the marketing people / customers keep on changing the 
spec on us). Distributed projects, in contrast, are often less 
uniform in their understanding of what "good practice" is.

I think the rules can be divided into four categories for our 
purpose. More in the end.

> developers are not concentrated in one place.

Why would this make CVS more important? It can just as well make it
less effective, because it is more difficult for developers to talk
about the changes.

> it can be ignored. Otherwise, CVS or whatever is a must.

Large and thriving projects such as Linux don't use CVS. If I reckon
correctly, Perl doesn't either (Shlomi?). Overall, I can both think 
of reasons why it is more important for /distributed/ projects to use 
source control and why it is more important for /centralized/ projects 
to do so. So I don't really know, and I'd say the importance of source
control is about the same for centralized and distributed projects.

> it tedious to keep up with the various releases of the programs.

Yup. Open source works by source distribution, so users will have a 
hard time to use the software if not for this. It is also important 
to make it easy to join the project.

> compile it, to see if it works OK. So the concept still stands but with a

"Release early, release often". I agree. Open source methodology 
takes daily builds a step further by saying that there is no 
fundamental difference between a daily build and a release, because
they have the same, or similar, target audiences.

> Very important. (need I say more)

I do :) A bug database is less important in a distributed project,
since all communication is recorded and searchable anyway. In a
centralized project, on the other hand, you very often exchange
information in person which will be forgotten the next day and lost.

> release a set of patches that fixes the bug-fixes that were found in the

Not applicable at all. I think Joel's point referred to the 
allocation of resources, i.e., when developers should concentrate
on which. In an open source project, it is the developers who decide
what itch they want to scratch, and all the project leader can say
is when and where they should be integrated. 

> freeze, bug-fix freeze, release announcement, etc.

Yeah. I don't think it is applicable, for the same reasons as 5. 
Developers choose what they want to develop and when, all the project
leader can decide is on schedule of integration, which I think is not
all of what the original point covered.

> Very important for most projects.

I disagree. Completely inapplicable. Joel said that you must have a 
spec before you start writing code. In the open source world, the 
leader often does not even know of a possible feature until they 
have a patch. There's no one publishing a project-wide spec of what 
developers should work now, they work on whatever they feel like.
The project leader can decide which features he wants to integrate
for a certain release, but that comes long after the code is 
written.

> 8. Do programmers have quiet working conditions?
.
.
> 9. Do you have the best tools money can buy?

I agree these are not applicable.

> will be willing to use the CVS version.

Many projects have those people, and they are generally considered a 
blessing. I'd say it is about as important for open source projects
as for traditional development.

> Usually, it does not happen.

I don't think so... A person will not be considered a part of the 
project until he has contributed code. Not only do the open source
projects follow this, but it is impossible for them to work 
otherwise. Joel's point was that contributors (developers) should be
judged by their contributions (code), but open source contributors
are often not even judged by themselves -- it is their contributions 
that are judged directly.

> One should distinguish between what the program is:

This is mostly not done in open source projects even though they 
could benefit from it. A lot of people download the code and try it, 
but that is a self selecting group, which defeats the purpose.

> 1. A mailing list, IRC channel, and E-mail address for questions, etc. -

Colaboration tools, together with CVS, daily builds and a bug 
database. I'd say that collaboration is as important in open source
projects as it is in traditional projects, but sometimes different
collaboration mechanisms are used (e.g., a searchable mailing list
can replace a bug database / source control to some extent). Possily
also a one-step build can be put here, although it might also be 
considered a quality issue.

Another category of these rules is about rigorously planning the 
development. All the rules belonging to this category (spec, 
schedule, bugs before features) are not followed because there's 
no planning at all due to a lack of central control.

A third category is the quality of development resources -- quality
of tools, of conditions and of developers. Open source project 
leaders usually don't know about any but the last. They are all 
important always but can't be measured. I wouldn't consider a target 
platform a development tool, though -- it is a bit more basic than 
that...

What remains are two rules emphathizing testing. These are important,
but not always followed.

---

From: Shlomi Fish <[email-masked]>
Date: Wed, 23 Jan 2002 15:16:15 +0200 (IST)

> control is about the same for centralized and distributed projects.

Hmmm, I checked and indeed the current perl5 VM does not use CVS. There's
seems to be someone who is considered the "pumpkin" owner who is allowed
to integrate patches into the code. This pumpking changes from time to
time as can demonstrated by the "perlhist" man page. I suppose the
everything passes thru Linus way of doing things in the Linux kernel works
much the same way (without taking human factors like Linus' lack of time
into consideration).

Note that Parrot which would be the future VM for perl5, perl6 and if its
hackers have their way many other high-level languages, is maintained
using CVS, to which several people have write access. From following the
perl6-internals mailing list this seems to break things on occasion, but
at least people can check in various unrelated changes in parallel. Some
of the patches that are applied are sent to the mailing list, which makes
me a bit trigger happy when deleting them in pine.

Regards,

	Shlomi Fish

---

From: Shlomi Fish <[email-masked]>
Date: Wed, 23 Jan 2002 15:22:43 +0200 (IST)

Judging by the commentary I received it seems that even we have
disagreements on what the OS equivalents for the Joel Test Rules are. Of
course, AFAIK, Joel Spolsky formulated the Joel Test alone, and eventually
people took it for granted, because of his prestigue. But I believe many
commercial software veterans will be able to formulate their own tests,
which may differ from the Joel Test.

Regards,

	Shlomi Fish

There is no IGLU Cabal! Its members could not agree on the contents of
its manifesto, thus giving way for Microsoft formulating it. The
Microsoft-formulated manifesto contained only one rule: "There is no IGLU
Cabal!". The rest is history.

---

From: Adi Stav <[email-masked]>
Date: Wed, 23 Jan 2002 19:43:28 +0200

> which may differ from the Joel Test.

But of course... There is not always concensus on what are good 
development practices, judging even by the limited set of 
contradicting attitudes that I am aware of.

Which is an interesting question (at least to me...) Is "software 
engineering" any more of a science than economics or theology is?
Personal taste and a vague idea of what /doesn't/ work?