Subject: RE: [hackers-il] An observation about standards compliance in clo sed software vs. open source

From: Chen Shapira <[email-masked]>
Date: Sun, 8 Jul 2001 09:51:55 +0200

> Possible reasons:
> 1. When source code is available, people can modify it until it fully
>    complies with standards (example - Posix compliance of Linux).
> 2. When starting to implement for a new platform, people don't need to
>    code from the beginning, but can port existing code base to the new
>    platform, while modifying it.  In this case, modifications are
>    relatively local and can be verified to preserve 
> compatibility (as much
>    as the platform allows this).
> 3. No commercial reasons to lock users into incompatible variants.

4. Open-Source development is notorious for not looking after the interests
of the user, under the lame excuse of "he has the source". This is yet
another example, along with the lack of decent integrated IDE's. As shoe
cobblers, we are walking barefeet.

---

From: "Nadav Har'El" <[email-masked]>
Date: Sun, 8 Jul 2001 10:57:58 +0300

> cobblers, we are walking barefeet.

Walking barefeet?

I don't think this is true, exactly. Open-source / free-software *is* user-
friendly, but only for more advanced users (i.e., it is not newbie friendly,
but as an experienced computer user I find it much more friendly that
MS-Windows, actually, because I can do *everything* - it's not easy, but
I can do it. In Windows there are stuff that are not easy to do, and some
stuff which is outright impossible to do unless you're a Microsoft employee).

Think of it this way: free software programmers often program something
they need for themselves; They design it themselves, code it themselves,
and test it themselves. These people are experienced computer programmers
(a newbie probably won't start by writing free software...), so their program
tends to be friendly to people like them, rather than being friendly to
clueless beginners.

If you ask me, that's how I want my computer system to be. A computer is a
tool of my profession, and I want it to be an empowering tool useful to me
and experienced people like me - and let the beginners spend some time to
learn to use it. Think of the tools of another profession - doctors. Would
you want surgery tools to be user-friendly so that any joe of the street
could pick it up, cut you up and operate on you? I didn't think so... Instead,
you want surgery tools to be good, reliable and accurate in the hands of
an experienced and professional doctor. Doctors study almost 8 years to use
these tools (and the theory around it which tells them how to use these tools),
so a computer professional having to spend 6 months learning how to use his
tools isn't too bad.

As the old Finnish proverbs go,

	Unix is user-friendly - it's just picky about its friends!

	Unix is simple - but it takes a genious to understand its simplicity.
and
	Linux *is* user-friendly. Not idiot-friendly, but user-friendly.

P.S. Of course, what I wrote doesn't preclude the availability of a free
beginner-friendly system for simple tasks like word processing, and stuff
like that. Just like somebody can use a steak-knife without studying how
to use a scalpel. However, these simplified systems will have their limitations
and will not empower the user as much as the more complex systems will. Just
ask anybody who tried to perform a liver transplant using a knife and fork :)

-- 
Nadav Har'El                        |      Sunday, Jul  8 2001, 17 Tammuz 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |In Fortran, God is real unless declared
http://nadav.harel.org.il           |an integer.

---

From: Adi Stav <[email-masked]>
Date: Sun, 8 Jul 2001 21:06:54 +0300

> cobblers, we are walking barefeet.

How is the lack of IDEs relevant? It seems to contradict your point; 
OSS developers working hard catering to users' needs while walking barefeet. 
In any case, do you consider such a test suite user- or developer-oriented 
feature?

I suggest the CatB explanation:

5. By "release early, release often", OSS projects acquire a very large 
testing base and a very long testing period. This makes formal testing
suites or procedures by developers less important, or at least to appear
less important.

---

From: Adi Stav <[email-masked]>
Date: Sun, 8 Jul 2001 21:17:59 +0300

> > cobblers, we are walking barefeet.

[...]
> tends to be friendly to people like them, rather than being friendly to

Actually I don't completely agree. There's lots and lots of really bad
free software around that's written by newbies, even such that is often 
used. Take a look at the source code of some ICQ clients. Probably the 
more critical the code, the higher the quality (my unproven guess).

---

From: "Nadav Har'El" <[email-masked]>
Date: Sun, 8 Jul 2001 11:36:12 +0300

> more critical the code, the higher the quality (my unproven guess). 

I guess there are different levels of newbie-ism. There are the real
newbies (hey, where is the "start" button? What's a shell, and where is
my trash can?). The newbies you describe can't be real newbies, if they
already know how to run a compiler, a shell, write a Makefile, use Unix-
specific C features, and stuff like that. That doesn't mean that 1 year
of experience lets you write *quality* code, so I guess you're right.

-- 
Nadav Har'El                        |      Sunday, Jul  8 2001, 17 Tammuz 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Cigarette: tobacco wrapped in paper, fire
http://nadav.harel.org.il           |at one end, and a fool at the other.

---

From: guy keren <[email-masked]>
Date: Sun, 8 Jul 2001 12:59:02 +0300 (EET DST)

On Sun, 8 Jul 2001, Nadav Har'El wrote:

> tool of my profession

but its not _only_ a tool of your profession. its a bootstraping mechanism
- you use a tool in order to create a product that uses the same tool.
thus, a computer is a tool of many different professions (and
non-professions as well - e.g. an entertainment tool).

>, and I want it to be an empowering tool useful to me
> an experienced and professional doctor.

the analogy is a bit invalid - surgery tools are directed only at
experienced doctors. computers have a much wider target audiance. perhaps
this calls for two different types of computers. one for programers, and
another for _users_ ?

> Doctors study almost 8 years to use these tools (and the theory around
> it which tells them how to use these tools), so a computer professional
> having to spend 6 months learning how to use his tools isn't too bad.

this reminds me of an ISP (in italy, i think) that didn't just give
internet services to everyone - only to people which it licensed to
"drive" the internet - they had a licensing exam, and if you wanted, you
could also take a course and prepare for the licensing exam.

shall we have a "computer users license", like we have a driving license?
or only for situations where one's computer work might have
life-endengaring effects (e.g. computer operators at nuclear reactors or
operational military bases - actually, they do tend to have some kind of
"licensing" - althought its an internal course an an internal set of
exams).

> like that.

a few billion potential users just sighed in relief ;)

how about, instead, having a system that is easy on beginners and yet
powerful for advanced users? the way linux desktops are built, it is
getting there. currently, i don't see that the existance of these desktops
took away any of the power i used to have over the system - i just had to
learn the locations and formats of newer configuration files, and as long
as the source is there, and 'strace' is there - i seem to manage to keep
up with them.

--
guy

"For world domination - press 1,
 or dial 0, and please hold, for the creator." -- nob o. dy

---

From: guy keren <[email-masked]>
Date: Sun, 8 Jul 2001 13:05:16 +0300 (EET DST)

> In any case, do you consider such a test suite user- or developer-oriented

lack of IDEs is relevant if you consider the set of "users" nadav seem to
have unconciously talked about - the set of users is 'programmers'. a
newbie is a 'newbie programmer'. newbie programmers seem to rely on the
usage of an IDE more then experienced _unix_ programmers - who have gotten
used to the lack of a borlandish-style IDE, and made up their own combos
(be that using emacs for everything, or using a combination of tools
without a common interface, e.g. vi+lots of xterms).

> suites or procedures by developers less important, or at least to appear

only makes it APPEAR less important, i believe. if you make proper testing
suites, you eventually will get a better product. and there are
open-source projects that went some ways to making testing suites
automated to some level. look at the 'gmake check' in postgresql. or at
the 'gmake test' of many perl modules. or at the gigantic test site of the
ACE library. regression testing helps you avoid re-inserting old bugs when
adding new fetures or fixing newer bugs.

--
guy

---

From: Adi Stav <[email-masked]>
Date: Mon, 9 Jul 2001 00:32:59 +0300

> without a common interface, e.g. vi+lots of xterms).

It doesn't matter whether the users are developers or not; it matters that
they have different software needs than the developers who created the 
software they are expected to use. You can still create software either
for yourself or for your users in this setting.

> suites, you eventually will get a better product. 

Of course a product that has a test suite will be of better quality. But
it is still less crucial for a product that is going to get eighteen 
months of thorough testing by 10,000 experts than for the same product if
it is going to be released in one month and expected to be correct. 

> and there are

True. Would you say Linux is quite different from most open source 
projects? If so, are there at all general characteristics to open source 
development?

What I think makes answering Omer's question difficult is that
there aren't that many successful open source programming languages 
with more than one implementation. AAMOF the only one I can think of is
Python, and even there the two implementations (Python and Jython) are 
mostly developed together.

Even among proprietary-originated languages, isn't Java's choice
of releasing the specification before/together with the implementation 
the exception rather than the rule? In both cases I know anything about 
(and I don't know many -- thoughts?), C and C++, standardization, and
therefore standardized test suites, only appeared after several
implementations existed. From my very limited set it seems that this is
the normal developemnt path of a language. This would make:

6. Open source languages very rarely have more than one implementation,
so a standardized test suite for compliance is irrelevant, as the 
standard is defined by the implementation.

---

From: Tzafrir Cohen <[email-masked]>
Date: Sun, 8 Jul 2001 14:53:11 +0300 (IDT)

> adding newfetures or fixing newer bugs.

Maybe. But the original post was about a test suite for the java
_language_, not for sun's implementation of the language. [IIRC the kaffe
JVM isn't a sun-java compliant JVM] kaffe can have an extensive set of
regression tests, that can check that many features don't break.

Just to make
this point clear: if I decide to derive some code from the postgresql
codebase (perhaps for making a competing product) this product need not
pass 'gmake check' successfully (maybe I decided to drop a couple of
features). The users will judge which "product" is better and use it. I
think that this was the original point. They don't need a
pgsql-compliance-test for that.

My example may be falwed, but I figure that you get the point.

-- 
Tzafrir Cohen
mailto:[email-masked]
http://www.technion.ac.il/~tzafrir

---

From: Omer Zak <[email-masked]>
Date: Sun, 8 Jul 2001 15:11:51 +0300 (EET DST)

I would like to take issue with the statement that regression test suite
is not important in open source projects.
While such projects have tons and tons of users, most of the users want
just to do their work, and have no time to test new releases.  Even the
experts would be willing to test only that new feature, which is important
to their work, rather than waste days testing all possible interactions
and side effects of that new feature.
And they wouldn't be willing to spend a lot of time testing the various
releases, which exist due to the RERO ("Release early, Release often")
methodology.

Therefore automated test suites, which cover say 95% of the functionality
are very much in order, just to save the time of expert users.

Other considratiosn:
1. If people find it easy to run a test suite, people with rare system
   configurations would be more willing to help test the software in their
   special systems, so there is more confidence that the software would
   work in a future new environment.
2. A developer, who wants to modify an existing software package
   (especially one with which he is not so familiar), will have his job
   easier if he can test quick and early his changes for any possible
   adverse side effects.

> it is going to be released in one month and expected to be correct.

                                             --- Omer
There is no IGLU cabal.  They got bogged down in testing daily releases
of Linux kernel, test versions of gcc, and daily builds of perl (not to
mention fribidi and isapnptools) - because they tested them manually,
rather than using methods of automated regression testing.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html

---

From: Adi Stav <[email-masked]>
Date: Mon, 9 Jul 2001 01:47:28 +0300

> is not important in open source projects.

I'd like to clarify again. I never said regression test suites are not
important for open source projects. I only said they are LESS important.
Consider the difference in public response to a new kernel prepatch that
trashes your filesystem versus that to a new Solaris version that does
the same. Naturally the Solaris version would face much harsher PR.
Why? Because a new kernel prepatch is not a expected to be a finished
products anyway. People are not supposed to install them on production
machines unless they really need them and after they waited a while
to see if they have any issues. Preferably, wait for a new certified
Red Hat release. In other words, the responsibility for finding bugs
shifts somewhat from the developers to the users, and the responsible
sysadmin had better test kernels before installing, or wait for others
to do it for them.

Of course filesystem-trashing releases are really, really bad, and 
maybe more likely to happen in Linux prepatch (as opposed to 
major distribution releases) than Solaris. But I am not 
making a value judgement in favor of either alternative; I am 
saying that RERO-oriented projects have a weaker incentive to develop
sophisticated testing tools.

> releases, which exist due to the RERO ("Release early, Release often")

I think most people who install the latest prepatches do it because
they want to help, or live on the edge, rather than because they actually
need the features crucially.

>    easier if he can test quick and early his changes for any possible

I agree completely. I remember vaguely a project that wanted to develop
organized test suites for Linux, but I don't remember much about it.
There's also the Stanford checker -- but it hasn't been released yet :)

---

From: mulix <[email-masked]>
Date: Sun, 8 Jul 2001 16:09:00 +0300 (IDT)

> organized test suites for Linux, but I don't remember much about it.

a quick google search showed this:
http://linuxquality.sunsite.dk/articles/testsuites/

i think the project you are referring to is "The Linux Test Project", at
http://oss.sgi.com/projects/ltp/

> There's also the Stanford checker -- but it hasn't been released yet :)

the stanford checker is actually a 'validity checker'. more about it
here: http://sprout.Stanford.EDU/SVC/. it performs its checks by
analysing the source code during compilation.

could anyone recommend a good book on 'software testing'? my summer
reading list only weighs 30 kgs at the moment :)
-- 
mulix
http://www.advogato.com/person/mulix

linux/reboot.h: #define LINUX_REBOOT_MAGIC1 0xfee1dead

---

From: Adi Stav <[email-masked]>
Date: Mon, 9 Jul 2001 19:59:12 +0300

> reading list only weighs 30 kgs at the moment :)

What an incredible conincidence, KT covered the Linux testing project
today at http://kt.zork.net/kernel-traffic/kt20010709_125.html#1 ...