Subject: Automatic Test Suites

From: Shlomi Fish <[email-masked]>
Date: Fri, 3 Jan 2003 13:57:48 +0200 (IST)

I know that many projects write automatic test suites to test their
programs for bugs before each release. Aegis has over 140 tests now and I
was told Peter Miller (its project head) added a test for every new bug he
encountered so it won't be reproduced. Subversion also has a lot of tests,
many of them unit tests that test specific modules. The Syscalltrack
Homepage mentions that it passes a lot of tests before it is released.

Now Chen asked me if I have prepared such automatic tests for Freecell
Solver, and I answered that I did not encounter it. OmerZ took the
opposite approach and claimed open-source developers should not bother
with automatic tests and should rely on their users for them.

Before each new release of Freecell Solver, there is a long development
branch (2.x.y where x is uneven) in which I gradually implement new
features, and towards the end document everything, run some test manually
and release. The recent development version is placed on the site and the
various stages are available on CVS, but I don't think many people
download them.

Usually, after I release a stable version 2.x.0 it is eventually followed
by 2.x.1 and others, that fix some bugs. It is possible I encountered the
same bug more than once, but I don't remember because I fixed so many of
them in the course of time. (although the vast majority during the
development branches)

My question is: what experience have you had with automatic test suites?
How much do they help identifying bugs and broken code? What kind of tests
would you recommend?

Regards,

	Shlomi Fish

----------------------------------------------------------------------
Shlomi Fish        [email-masked]
Home Page:         http://t2.technion.ac.il/~shlomif/
Home E-mail:       [email-masked]

"Let's suppose you have a table with 2^n cups..."
"Wait a second - is n a natural number?"

---

From: Omer Zak <[email-masked]>
Date: Fri, 3 Jan 2003 14:54:25 +0200 (EET)

> with automatic tests and should rely on their users for them.

Where&when did I say the above?

My position about tests in open source projects is that tests should be
considered to be part of the source code, because they help certify that
nothing unexpected was broken by an attempt to modify the software.  In
order to be confident at a modified version of complicated software, such
a certification is sometimes necessary.
                                             --- Omer
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html

---

From: Shlomi Fish <[email-masked]>
Date: Fri, 3 Jan 2003 17:23:27 +0200 (IST)

> Where&when did I say the above?

http://groups.yahoo.com/group/hackers-il/message/1491

Although it could have been a joke...

> nothing unexpected was broken by an attempt to modify the software.In

OK. Sorry for misinterpreting you.

Regards,

	Shlomi Fish

>                                            --- Omer
> WARNING TO SPAMMERS:at http://www.zak.co.il/spamwarning.html
>
>
>
> To unsubscribe from this group, send an email to:
> [email-masked]
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>

---

From: Omer Zak <[email-masked]>
Date: Fri, 3 Jan 2003 19:26:38 +0200 (EET)

> Although it could have been a joke...

It was indeed a joke.  The above message contained two contributions of
mine to Shlomi Fish's "The Rules of Open-Source Programming".

                                             --- Omer

---

From: Shlomi Fish <[email-masked]>
Date: Sat, 4 Jan 2003 10:49:21 +0200 (IST)

On Fri, 3 Jan 2003, Omer Zak wrote:

> It was indeed a joke.The above message contained two contributions of

Well, then I may have taken the joke too seriously and believed it
reflected your opinion. Naturally, some open-source projects can rely on
their users to locate bugs for them. This is especially true of a project
which does not have very good security concerns, or a GUI one that is hard
to test automatically.

Still, if you can write test suites, than maybe you should, despite the
fact that a large number of projects (including mine) don't bother to.

Regards,

	Shlomi Fish

---

From: guy keren <[email-masked]>
Date: Sun, 5 Jan 2003 09:45:47 +0200 (IST)

> How much do they help identifying bugs and broken code? What kind of tests

i've seen several good test suites, and i've participated in writing 2 
others. i distinguish between correctness tests and stress tests.

correctness tests deal with checking that a specific function is done 
properly (gives the expected reuslts). this is mostly useful when you 
re-write your code and want to make sure it didn't break, or when you add 
new features to a large feature-full project.

stress tests deal with bombarding your code in some manner, and making 
sure it does not crash. this is useful both in early development stage, 
and when the project is mature.

i've seen stress testing doing a lot of good, both for the developers, and 
for users. as a developer - it helps finding instability bugs, races and 
the like. its not somehting that comes instead of manual testing - but 
rather complements it.

when a project is small, or does something small, it is rather easy to 
test it. when it becomes large, and contains a lot of features, manual 
tests simply don't cope well with the task. when i was working on large 
industry project, which was first only being tested manually, i saw that 
we got bugs re-crawling into the system (sometimes the same bugs, 
soemtimes new bugs with the same symptoms). after a while, the testing 
team started writing automatic tests for the code (it was driven by a GUI 
on Unix, so they used X-runner for the testing) - and eventually, a single 
machine was dedicated to run automatic regression tests on the latest 
'internal release' version of the code all the time - and this helped us 
find bugs faster. finding a bug close to when you create it is very 
helpful - since it usually lies in the new code you just added (thought 
someitmes its a bug in the old code, that the new code now exposed due to 
using the old code in new ways).

thus, i think my (current) conclusion is that automatic tests should be 
developed along side the development of the software, and the ammount of 
effort spent on it, should be proportional to the size and complexity of 
the software, as well as to the maturity stage of the software (the more 
mature it is - the more tests you need in order to make sure it keeps
ticking).

-- 
guy

"For world domination - press 1,
 or dial 0, and please hold, for the creator." -- nob o. dy