Subject: RE: [hackers-il] Maintenance of 10,000,000 lines long application s?

From: Omer Musaev <[email-masked]>
Date: Sun, 6 May 2001 20:37:22 +0200

> agreed, to both accounts. on a tangent, the use of the 
> debugger, i would
> be happy to have your opinions on the following post by Linus:
> 
> http://lwn.net/2000/0914/a/lt-debugger.php3
> 
> i find that when i try to avoid the debugger, the resulting code is
> higher quality, because i have to real _think_ about the code i'm
> writing. however, in some cases (a deep call stack, an obscure bug
> which only happens every n'th iteration), a debugger is very, 
> very usefull.

The main point of Linus is that as long as the project do not have 
deadlines, avoidance of debugger is a good thing.

However, when you are programming for a living, ability to find a problem 
and fix it in a short time is a crucial quality.

I would like to state it that way: If you are surprised by debugger output 
you are in deep problem. But if you use debugger to make sure that your 
intentions were clear to compiler, you ability to express yourself will
grow with time. 

In that way a debugger is similar to a spelling checker. That brings me to 
interesting thought - what a thesaurus analogy would look like?

> 
> > > i also find that beginning programmers often compile 
> after absurdly
> > > small changes, even as small as one line of code.
> > >
> > > so, a question, o' il'y hackers: how often do you 
> compile? approx, per
> > > how many lines of code?
> >
> > I'm not sure it is a big "gluck" to be able to write long programs
> > and only then compiling them. I saw one example where a 
> (rather beginner)
> > programmer wrote some function, compiling it only after the 
> entire code
> > was done. When she finally compiled it and tried checking it, it was
> > HORRIBLE. The function did not work correctly in ANY case. 
> It had many
> > cases and sub-cases, and when it was all written it was 
> almost impossible
> > to understand what was wrong, or be sure if all the 
> problems was fixed.
> > So I rewrote that function from scratch. I wrote the 
> function case-by-case,
> > testing each case (by compiling and actually running the 
> function) after I
> > wrote it, and thus made sure that all the cases worked 
> correctly. All-in-all
> > it didn't take me much longer to write the function, but 
> the difference was
> > that mine worked :)
> 
> actually, nadav, the difference is that you wrote the code...

There is another "goodness" of Nadav's approach: 
By splitting the big and evil code into pieces, the use of debugger on 
the second program would make much more sense.

> --
> mulix
> http://www.advogato.com/person/mulix

---

From: Adi Stav <[email-masked]>
Date: Sun, 6 May 2001 21:25:15 +0300

> intentions were clear to compiler, you ability to express yourself will

I find a debugger extremely useful in answering the "when", "where"
and "what" questions, and in some rare cases even "who", but worthless
in answering "why". The debugger can give such a feeling of control
and omniscience that I can imagine myself forgetting there are some
questions it cannot answer -- and using the answer to "when" I really
need the answer to "why". That would lead to me writing workarounds
and not fixes. Maybe that was what Linus meant.

> interesting thought - what a thesaurus analogy would look like?

An optimizing compiler? :)

---

From: guy keren <[email-masked]>
Date: Sun, 6 May 2001 23:14:35 +0300 (EET DST)

folks, you're doing wrong thiking by bashing debuggers. they are just
tools, and just like any other tool, they can be used properly, or not
used properly. how one uses them is completely a personal experience.

when i have a duzzling bug in the code, sometimes i think about things in
order to solve the problem, and other times i launch a debugger to try and
find what is going on.

as omer said, the problem with inexperienced programmers is that they
DON'T use the debugger, not that they don't use it wisely. that's
partially cause they are never taught how to debug code properly. i
haven't seen debugging (and logging) techniques being properly taught in
programming courses.

adi - if you find the debugger causing you to write workarounds instead of
proper fixes, that's a problem with you, not with the debugger. when i
step through a code and see something didn't work properly, i try to
figure it out. sometimes its a completly local bug that can be easily
fixed. sometimes its a non-local bug that requires more thinking. if i
write a workaround, its in cases where i didn't manage to solve the
problem, and using the workaround is sometimes better then not fixing the
problem at all (btw, there are cases where i understand the real problem,
and yet write a work-around, for various practical reasons, such as time
constraints, and the wish not to introduce new bugs when a real fix isn't
very simple - again, that is due to time constraints).

and btw, "writing code without using a debugger" doesn't strike as good
hackerism. and always knowing everything that is going on isn't quite
human. the programs are too large to be able to know everything about it,
and the interactions between its different modules. thus the debugger is a
tool that helps me, and i use it when its convinient. what next? i
shouldn't use purify cause i'm not supposed to write code that has memory
leaks or that has memory corruptions in the first place? ta! that's the
kind of speak of people who never realy wrote proper code in large
quantities. shu, shu! :)

i have a good friend who had the habbit of going over every line of code
he wrote using a debugger, to make sure he didn't make stupid bugs. as he
now has quite a few years of experience, he now goes with the debugger
over smaller parts of the code, but he maintains that this is a good
habbit, and IMO he's a good programmer. and he's not the only one working
this way.  we had discussions about this issue several times over the past
years, and in our latest discussion we came to some agreement that it is
important to teach inexperienced programmers to use the debugger in such
cases. it helps you easily avoid all the silly bugs of '=' isnide an if,
off-by-one errors, missing parantathes causing the wrong 'if' to be
executed, and so on. and don't tell me it comes in expense of thinking.
programmers are not robots, and thus are bound to make those stupid
mistakes, and the only ways i know of solving those problems (again, they
cannot be avoided unless you're a robotric) is by re-reading the code, or
using a debugger. and the second choice makes more sense. and btw, the
reason many programers avoid using the debugger is pure lazines, and
thinking that "it'll just make me work slower, if i go over every line of
code using a debugger, so why wate my time?"

so don't make the debugger a "special" tool. just like you use syntax
highlighting in your editors (you do that - don't you? shame on you!
:P~~~), you might as well use the debugger where it can help you, and not
spend your time on finding reasons why not to use it even in cases where
it'll make the job easier and the code moer robust.

--
guy

"For world domination - press 1,
 or dial 0, and please hold, for the creator." -- nob o. dy

---

From: mulix <[email-masked]>
Date: Sun, 6 May 2001 23:46:32 +0300 (IDT)

> used properly. how one uses them is completely a personal experience.

ah, guy, all tools can be used properly or not, but some tools have
a propensity to do evil. please explain to me how one uses a flame
thrower "properly" not to do evil, or, to return to chen's analogy, how
one uses a bra improperly (no, no need to answer that ;)

> haven't seen debugging (and logging) techniques being properly taught in

debuggers are tools. just like we dont teach how to use a specific
compiler, or a specific editor, we shouldn't be teaching debuggers.
debugging techniques are a different matter, but i guess that's
something that comes with experience.

> kind of speak of people who never realy wrote proper code in large

actually, guy, yes, one should strive to write code without memory
leaks. maybe it's a close to impossible goal (i dont think it is), but
it's a worthy one. such is the price of being a good programmer, taking
care of the little details. or, using garbage collection, but that's
another discussion.

i have written large projects, i have worked with and without debugers
and with and without purify. when i _consciously_ try to minimize memory
leaks and smears, instead of relying on automated tools, i write better
code. dont you?

> executed, and so on. and don't tell me it comes in expense of thinking.

lack of attention to details? lack of running gcc -Wall? lack of code
review, even if it's by the same person that wrote the code? lack
of simple unit tests? there's a reason they are called 'stupid bugs'...

> using a debugger. and the second choice makes more sense.

this strikes me as an unqualified statement. how does using the debugger
make more sense then reading the code?

linux/reboot.h: #define LINUX_REBOOT_MAGIC1 0xfee1dead

---

From: Omer Zak <[email-masked]>
Date: Sun, 6 May 2001 23:45:29 +0300 (IDT)

I would like to get back to the original ideas of the thread of 10,000,000
lines long applications.

No matter what software engineering methodology, pattern, key practice or
tools one uses, there comes a time at which the software to which one's
responsible is too big to fit into one's head all at once.

What do you do when this happens?

For example:
You started a small software project.  You incrementally developed it,
adding features all the time.

Suddenly, one day, you lose control over the project, because it is just
too big for your head (it has grown to be more than 100 modules, say).

Can the veterans among us tell how they coped with this kind of
complexity?

My own favorite method was to invoke grep to find all places affected by a
change (identifiable by using the same variable name and/or another
textual marker, if one was wise enough to choose grep-friendly variable
names). 

                                             --- Omer
There is no IGLU Cabal.  Their collective head exploded when trying to
deal with the complexity of software which consisted of 1,000,000 modules.
WARNING TO SPAMMERS:  see at http://www.zak.co.il/spamwarning.html

---

From: "Nadav Har'El" <[email-masked]>
Date: Mon, 7 May 2001 01:08:48 +0300

> responsible is too big to fit into one's head all at once.

I don't have experience with that thing, but I have an analogy to offer.

Consider medicine. When you're a doctor, you're basically the maintainer
of Human bodies. Human bodies are at least as complicated as your 10,000,000
lines of code (probably more). No human doctor could ever hope to understand
the *complete* human body. This is why you have a GP (general practitioner,
"rofe mishpacha") which is supposed to have some basic (or not basic, but
certainly not deep) knowledge of all parts of the body. When he sees you
have an ear infection he can recognize it's a problem with your ear and say
"I don't know much about ears to I'll send you to this other guy, an ear
specialist". This ear-specialist knows almost everything about ears, but
even he cannot understand everything about them (say an ear corresponds to
1,000,000 lines in your code). So if he thinks there's a problem with your
(say) inner ear, and there's some famous inner-ear doctor with some new
supper-dooper testing equipment (debugger?), he'll send you to that third
doctor, hoping that he can fix your problem.

Sometimes none of these doctors know how to fix a problem, and the patient
goes on to live with the problem indefinitely (in our example, say, the patient
continues to hear ringing sounds constantly, but still lives). In very bad
cases, when a serious problem isn't fixed, the patient dies.

I think the same thing happens to huge software project. If either a GP-like
programmer isn't available (which can guess which part of the huge project
is responsible for the problem) or a specialist-like programmer isn't
available for that specific broken part, the broken part remains broken
indefinitely. If the users learn to live with it, fine (e.g., consider the
"blue screen of death" of Microsoft Windows 95). If the problem is too
serious to be ignored, the program will eventually "die" (i.e., nobody will
be using it any more).

-- 
Nadav Har'El                        |       Monday, May  7 2001, 14 Iyyar 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |If you tell the truth, you don't have to
http://nadav.harel.org.il           |remember anything.

---

From: Oleg Goldshmidt <[email-masked]>
Date: 07 May 2001 01:11:22 -0400

> Can the veterans among us tell how they coped with this kind of

I just sent a long posting focused mostly on the tools that I had the
fortune (sometimes misfortune, due to implementation deficiencies) to
use developing a big system.

My solution is to work the same way on any software project,
regardless of size. Whatever I do, I make a point of using CVS
(tagging liberally), GNATS, a set of boilerplate makefiles, automatic
dependency tracking and prototype generation, etc. All that for
myself, on my home computer, without collaborative development.

The same goes for coding practices. I often write throwaway 5-10 line
programs to check whether my understanding is right. I cd to ~/tmp,
open a new file (foo.c), and type a few lines of code. I make a point
of writing such a file the same way that I would code a piece of a
large project. To give an specific example, I go to the extreme of
"return EXIT_SUCCESS;" (with "#include <stdlib.h>) rather than "return
0;" in my 5 line "int main()".

I firmly believe that this is a very good habit to get into, and is
well worth the extra seconds it takes me to type a throwaway program.
The upside is that the habit prevents me from cutting corners when I
write real code. I have paid dearly for almost every corner I cut in
my experience, and I have paid even more for every corner that someone
else cut developing code that was dumped on me for maintenance. I do
think that doing things "right" by habit helps you as your project
grows.

-- 
Oleg Goldshmidt | [email-masked] 
"I'd rather write programs to write programs than write programs."

---

From: Omer Zak <[email-masked]>
Date: Mon, 7 May 2001 00:41:14 +0300 (IDT)

> use developing a big system.

Yes, and it was worth its weight in gold.

> think that doing things "right" by habit helps you as your project

OK.  But this only postpones the moment at which one loses control over
one's project due to its having grown beyond all bounds.  Then, what can
one do to retain semblance of control over the project?

(BTW, there is another obvious advantage to Oleg's approach:  one can
 reuse the code from his throwaway programs in production code.)
                                             --- Omer
WARNING TO SPAMMERS:  see at http://www.zak.co.il/spamwarning.html

---

From: guy keren <[email-masked]>
Date: Mon, 7 May 2001 09:47:53 +0300 (EET DST)

On Mon, 7 May 2001, Omer Zak wrote:

> OK.But this only postpones the moment at which one loses control over
> one's project due to its having grown beyond all bounds.Then, what can

there are two cases. one in which you're currently tackling a project that
has goen out of control. another is that you are tackling a project of
this size, and afraid that it might get out of control.

in the later case - don't allow it to get out of control, by gradually
introducing configuration management issues such as those oleg described.
if the project gets too large for one person's head - have few persons
that knows the general layout of the project, and that know which areas
each of them knows in a deeper sense. then, together, as a group, they
have "all the project in their heads". make sure if one of them leaves,
they pass the material to the others and/or to their repacement in an
orderly manner, and give their replacement time to get introduced into the
code gradually, by assignning them many small tasks that'd make them go
over around the project and looking around it.

in the former case (project was out of control when you boarded the
ship), letting such a large project get out of control implies some very
very bad management decisions. perhaps key personel left the company
without doing a proper passing of the material to their replacement.
perhaps nobody bothered much with working procedures in the first place.

in the former case, you have several options. note that i haven't yet got
into such a case myself - i'm trying to deduce fromwhat i've seen in
either smaller badly managed projects, or the one large project i was
involved in for the past 4 years, so take my advices in a limietd manner.

1. run away from the sinking ship. if the management of the company
   running such a project let it get out of control, you'll have hard time
   working with them, and making them change their ways.

2. if you decide to stay on board for the challenge, then first check out
   what working procedures and historical events lead to the current state
   of the project. doing that is good for two reasons. first, you'll know
   what need to be changed, and what procedures are worth keeping (even in
   lousy-managed projects, some things might still be worth keeping, and
   its good to allow people in the project to feel there is something in
   the old ground that's stable and they can safely stand on).

   then have several _good and experienced_ people in the project start
   scanning the configuration management system, and from there try making
   a map of the terain, listing which models exist in the system, shortly,
   what each of them contains, and make a general graph of their
   interdependencies. in such a lousy-managed project, i'd expect you'll
   find too many dependencies, incuding many cyclic dependencies. that'll
   make the mapping process harder to make, and you'll have to take note
   that there might be various connections via side effects. for example,
   i saw a project (not very large, only about 200,000 lines of code)
   that was built using a client-server model, with a database as well.
   the server would access the database - you'd imagine. it did, but the
   clients accessed the database directly as well, all over. that causes
   various indirect relations between code modules that might seem
   seperable at a first glance.

   then, learn how the beast is being fully recompiled, make a new copy of
   the sources, and try to build it, until you get into a state where the
   newly built code appears to work like the older built code. that step
   alone could take quite a while to perform. if this step goes as badly
   as i think it would, then by the time you'er done, you'll be quite
   familiar with the build system of the project.

   after that, prepare a list of very very small changes to be made to
   "outer" modules, and allow several programmers to perform them. choose
   a set of "representative" modules and a set of "representative"
   changes, for this to make sense.

   then take another step, and try doing the same for one of the
   mid-internal modules. using the grep solution you've suggested, and
   using compiler warnings (its probably that the old code gives many many
   compiler warnings during compilation, cause nobody bothered making it
   cleanly compile, so the only way to make sense of them is diffing them
   with the compiler warnings given compiling the older code, and focuing
   only on the changes).

> (BTW, there is another obvious advantage to Oleg's approach:one can

you're talking about 10M lines of code, and you want to reuse a few 10s
of source lines from these test programs? you must be kidding :) you're
also not talking about reusing - you're talking about copy&paste(!!) - the
mother of all evils...

--
guy

---

From: guy keren <[email-masked]>
Date: Mon, 7 May 2001 10:30:49 +0300 (EET DST)

> one uses a bra improperly (no, no need to answer that ;)

you're making a bad analogy here, IMO. a debugger is no more deciving then
an editor. think about it - the editor has a built-in copy and paste
mechanism. that will deceive programmers into using the copy&paste
paradigm. maybe we should remote that feature then from our programmers
editor? only allow for cut and single paste operations, with paste erasing
the contents of the cut buffer (to avoid the ctrl-K ctrl-Y ctrl-Y method
of duplicating lines in emacs) ?

> something that comes with experience.

using this method, you can say we sohuld also not teach programming
techniques - people wil learn them from experience. well, software
engineering is mostly about debugging one's code, cause that's what you
mostly do when writing software. writing the code ammounts to around 25%
of the coding time (i'm not counting here the time for making software
design, btw).

> leaks.

i didn't say one sohuldn't strive for that. i said that it's impossible,
cause you are a human being, not a machine. and hence, you should have a
strategy for handling the cases that slip by.

> leaks and smears, instead of relying on automated tools, i write better

no. i always make bugs. its part of the work, and i cannot write bugless
code by thinking i can write bugless code. thus, i have to have the tools
and the strategy for fixing bugs as well.

plus, i also fix bugs in other people's code, either directly, or
indierctly (by giving them consulting when they strike bugs they fail to
fix on their own). i also sometimes ask other people to help me fix my
bugs.

you should remember that while you gain experience that helps you make
fewer bugs in simple code modules, you also write more complex code units
and systems, which cause bugs stil to exist - only they are larger bugs,
and less localized (i.e. caused by interaction between various system
modules).

> > over smaller parts of the code, but he maintains that this isa good
> > this way.we had discussions about this issue several times over the past

don't expect inexperienced programmers not to make stupid bugs. and they
are called stupid cause when you find them, you say "ouch, what a stupid
mistake". but face it - inexperienced programmers tend to have bugs, not
cause they are stupid, but rather cause they don't have enough experience
to show them that they have better methods of working out.

> make more sense then reading the code?

when i _read_ code, i see what i _think is written there. that is how the
eye and the brain work, biologically, and was proved time and again by
experiments. so while by reading the code, i _think_ i am seeing a '{' on
the 'if' line, when i run it using a debugger, i _realize_ that this '{'
wasn't there, and it was a figment of my tired brains. teh eye-brains
system has easier time to see _changes_ then to analize _static_ input,
and in a debugger it sees changes, while in an editor it sees static
data.

the same issues goes in other places. for example, i ask someone to check
if they have this and that file in a directory. they perform an 'ls', and
start scanning the output for the file's name. they say "no, i don't have
it", when they simply missed it in the list of tens of files with rather
similar names. when someone asks me this question, i ask ls to show me
that specific file's name, and i use file name completion. this way, i
either get a _single_ file listing (and i _always_ use ls -l, not just
ls), or i get a 'no match' message. thus, i let the computer do the work
it does much better then i do, which is text-based pattern matching.
much the same way is how i consider a debugger. it helps me avoid the
delusions that the eye-brain system might make me see. and its not out of
pure lazyness - i have experience with scanning large log files by eye,
and i think its a useful skill. but it shouldn't be done instead of using
grep. hence, reading code and being able to grasp and parse it in one's
head is a usefull skill - but it should not come instead of using a
debugger.

--
guy

---

From: Adi Stav <[email-masked]>
Date: Mon, 7 May 2001 15:03:38 +0300

> used properly. how one uses them is completely a personal experience.

Guy, I was not trying to bash debuggers, and I don't think anyone was
saying "I don't recommend using debuggers" or anything of that
sort. Like I said in my post, I find debuggers irreplaceable for
answering many questions -- I consider a debugger one of the
development tools I'd absolutely not be able to work without.
Afterall, you can write programs with cat (given incentives
sufficently high or a system state sufficiently incredible), but you
can't get a core backtrace with od, no matter what (war stories
notwithstanding). I (and, I think, others) was merely trying to
understand the what risks Linus sees in lazy reliance on debuggers and
understand his (admittedly paternalistic) point.

> proper fixes, that's a problem with you, not with the debugger. 

Specifically, that would be a problem with my use of the
debugger. Trying to use it to answer the wrong questions.

> when i

Obviously, one shouldn't say "I'll skip those memory management
functions on the code review, because purify will tell me if there's
anything wrong with them anyway". As in "I'll skip reading this file,
it looks complicated, I'll just pass it through the debugger and see
what it does". At least this is how I understand Linus' point's
logic. But I can't really prove this logic, and running it through a
debugger would be recursive :)

> code using a debugger, so why wate my time?"

I find a debugger useful especially with those kinds of errors. For
more "program logic" or "interesting" bugs I usually like really
detailed logs. Maybe that's because a debugger lets you see evrything
about a program state only in a single moment at a time, while a
detailed log gives you a certain aspect of the logic during all of (or
part of) the program's execution.