Subject: The Philosophy of Automated Tests

From: Shlomi Fish <[email-masked]>
Date: Fri, 14 Jan 2005 23:38:19 +0200

It is a common idiom that in order to prevent bugs from appearing in a 
software in the first place, and from re-appearing, the developer should 
write automated tests to test that the program's code behaves the way it 
should.

It took me some time before I learned about this methodology, and even more 
until I decided to consistently implement it. Nevertheless, I had, and my 
recent CPAN module (HTML-Widgets-NavMenu) has 224 individual tests so far, 
and has more LOCs dedicated to tests than those in the main program logic.

Now a common paradigm in writing tests is to write the tests before you write 
the code or perform the bug-fix, watch the test fail, and only then change 
the main program. This is done so the test will serve as a contract for the 
program.

As I went on adding features, refactoring and writing tests, I encountered 
some dilemmas which I'd like to discuss.

The first one is what to do when refactoring. Let's suppose I've done some 
refactoring, like extracted a method and replaced all relevant places in the 
code with a call to it. Should I write a testing framework for this method 
before I do the refactoring? Should I do it afterwards? Or are internal 
utility methods should not be thoroughly tested, as long as the interface to 
the code has good test coverage?

Another dilemma is what to do when I'm adding a feature. I wrote some tests to 
cover the feature, and afterwards when I got to work on the code, I 
discovered that it will require making the internals more modular. So I go on 
messing with the internals a bit, which adds more functions or classes, that 
are not tested. Should I write more tests to this new functionality as I go, 
and see that, once it is added, these tests pass?

Another thing I noticed myself doing, which may be considered undesirable, is 
that I sometimes noticed a function that existed in my code, has become 
important enough to warrant its own unit tests. So I wrote tests for it. But 
naturally, it would have been a better idea to write the tests before the 
function was written.

Possibly, part of these problems are caused by the fact that the code was 
first written without too many tests, when just trying to get it to work 
using a module that someone else wrote. Only later an extensive test coverage 
was added. Perhaps in a code that was written with a test-driven methodology 
from the beginning, such problems are encountered less frequently.

Regards,

	Shlomi Fish

---------------------------------------------------------------------
Shlomi Fish      [email-masked]
Homepage:        http://www.shlomifish.org/

Knuth is not God! It took him two days to build the Roman Empire.

---

From: Adi Stav <[email-masked]>
Date: Sat, 15 Jan 2005 11:09:10 +0200

Hi Shlomi,

What I can contribute here is that there is no "should". Engineering 
is not an exact science, and software engineering more so than others.
Experiment and see.

As for myself, I like my unit tests to test more than a single 
internal function.  They usually test an entrire module or 
functionality of a program, but give it such input that make sure
that all code paths are covered. So refactoring doesn't really change
things, unless it creates a new code path, requiring a modification in 
the tests.

---

From: Shlomi Fish <[email-masked]>
Date: Fri, 28 Jan 2005 22:07:25 +0200

> is not an exact science, and software engineering more so than others.

Hmmm... I guess writing the tests _after_ writing the code if the new code 
should be written to again pass the tests, cannot hurt too much. It's better 
than not writing any tests for it.

> internal function.  

Some of my unit tests test more than one internal function. However, sometimes 
a function is important and self-contained enough to warrant its own unit 
test. (is a test a single verficiation of expected value vs. result? Or is it 
an entire set of them?) 

> They usually test an entrire module or 
> that all code paths are covered. 

That's what I'm aiming for as well.

> So refactoring doesn't really change 

Can refactoring actually create a new code path? As you may know, refactoring 
is improving the internal design of the program without modifying its 
external behaviour.

Do you write the test to test this new code path before or after the 
refactoring?

Regards,

	Shlomi Fish

---

From: Omer Zak <[email-masked]>
Date: Fri, 28 Jan 2005 22:52:44 +0200 (EET)

> On Saturday 15 January 2005 11:09, Adi Stav wrote:
[... snipped ...]
> Do you write the test to test this new code path before or after the

I would say - write tests before and during refactoring.
Do the refactoring at several steps, such that at the end of each step,
the piece of software has the same external behavior.
Run the tests on the original software and on the refactored software
at each step.
Add new tests as necessary to exercise cases which you did not think of
previously.

By the way, I cannot think of a case in which refactoring would add an
extra code path.  At least, not in languages having adequate expressive
power (such as Python).
                                             --- Omer
My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html