Subject: RE: [hackers-il] Maintenance of 10,000,000 lines long application s?

From: Chen Shapira <[email-masked]>
Date: Sun, 6 May 2001 13:07:00 +0200

> 1. how many programmers per integrator?

Approximatly 50.
Although some projects are managing without an integrator at all (having
programmers do that on rotation)

> 2. approx. how many lines of code per 'baseline'? or to ask 
> it differently
>    - how long does a full compilation of a 'baseline' take? 
> during this
>    build - can other programmers continue working on their code?

Complex question.
We have interpeted languages, which take about 5 minutes.
Real build will take about 15 minutes, and if we want to have setup read too
it will take longer.

We do have very small baselines (Like those belong to DBA's) with less than
5000 lines, larger ones take around 30,000. We try not to have more than
50,000 per baseline.

When your baseline is on build you can't do checkins, but you can do
everything else.

> 3. when a programmer works on a 'baseline' - do they copy all 
> of its code
>    to their own private work space, and compile their own 
> complete private
>    set of libraries and binaries? or does this private work 
> place contain
>    symbolic links to sources/libraries/binaries located in 
> the 'public'
>    baseline, with only files they changed/compiled being kept as real
>    copies in the private work space?

Exectly.

> 4. how often (you said "not often" - that's a very subjective 
> term) do you
>    have problems with code changes coming from libraries generated by
>    other teams in other baselines? and on the "not often" 
> cases when this
>    happen - how long since the problem is first discovered, until the
>    programer in your team that found it can continue work?

I remember it happening around 6 times in the last year in all the teams in
my product.

They all happened the same month though, when two major infrastructure teams
did major changes simultaniosly.

It took around 4-5 hours until we could continue working.

> 
> btw, note that any kind of configuration management environment
> works - eventuallly, the problems are solved (until the next 
> time), and
> projects get done, and tested, and delivered. the question is how much
> this costs. naturally, mercury is a profitable company, so we 
> can assume
> that their methods work well. but i've seen at least one 
> company that was
> profitable, and yet had a night-merish configuration management system
> (which was also built in house, developed and maintaned by a group of
> 4 or so programmers ....)

I think that the "1 click building" is very important, but the fact that
programmers can easily build their own baseline (for that one small last
change) without notifying anyone can backfire. 

I think most programmers here like the build system. It has a very large
overhead when starting a new baseline (especially if its in a new language,
like I did with Java), but I think it is mostly enjoyable.

Then again, I enjoyed many inadequant tools until I used a better
alternative. Manual gear in a car for example :-)

---

From: guy keren <[email-masked]>
Date: Sun, 6 May 2001 13:20:26 +0300 (EET DST)

> > 1. how many programmers per integrator?

50???? that's a lot of programmers, then. the overhead of the integrators
is then negligable. that's good.

> Real build will take about 15 minutes, and if we want to have setup read too

what do you mean with 'setup read too'? also, since you probably ahve a
lot of baselines, i'd gather that when a baseline changes, then all
baselines that use that baseline have to be recompiled as well?  the
ammount of noise would most likely change based on the deepness or
shalowness of the dependency tree.

> 5000 lines, larger ones take around 30,000. We try not to have more than

ok. then how many lines of code does your baseline (if there are few -
pick an average) _depend_ on (i.e. code in baselines your baseline depends
on) ?

> When your baseline is on build you can't do checkins, but you can do

and when you checking (until a build is make) - other programmers can keep
on compiling their code, or they have to wait until you compile (cause
having a header file inconsisent with the matching class's object file
causes useless binaries) ?

> >  copies in the private work space?

exactly what? this was a 'choose the option' question. which option in the
two options above (full copy vs. mostly symlinks to public baseline) is
correct?

> I remember it happening around 6 times in the last year in all the teams in

these teams are made of how many programmers? 50?

> like I did with Java), but I think it is mostly enjoyable.

how often do you create new baselines? and how long does it take to create
a new baseline?

> alternative. Manual gear in a car for example :-)

i also seen a place where it took them a long time to compile the system
after changes, but no one complained - they used this time to go have
another looooong cup of coffee. hope that's not the case there ;)

--
guy

"For world domination - press 1,
 or dial 0, and please hold, for the creator." -- nob o. dy

---

From: "Nadav Har'El" <[email-masked]>
Date: Sun, 6 May 2001 13:24:05 +0300

> 5000 lines, larger ones take around 30,000. We try not to have more than

15 minutes to compile 50,000 lines? Why?
Actually, you can lower this figure to anything you want, depending on how
much hardware you have to compile. You can have an 8-CPU SMP and run "make
-j 16" and have the compilation finish over 10 times faster. Better yet, you
don't have to invest any more money if you're running Linux: Just use Mosix
on each developer's workstation, and run "make -j", and have the compilation
finish in less than a minute using all the workstations. [Actually, I'm not
sure Mosix will handle "make -j" very well, because it starts a whole bunch
of short-lived processes which will not migrate quickly enough. Maybe make -j
can be improved to start each compilation on a different node (using rsh,
for example) - then Mosix will not even be needed]

Moreover, if you keep object files around, and only compile stuff that has
changed (this is what 'make' is for), I don't see why you can't lower the
build time to something in the order of 1 minute (most of it being the linking
phase, that usually can't be parallelized).

> When your baseline is on build you can't do checkins, but you can do

Why can't you do checkins? Can't the compile process make a copy of the
sources before starting to compile? This means the sources will be locked
only for the duration of the copy, which is only several seconds.

> change) without notifying anyone can backfire. 

I can't imagine 50 people doing this 1-click-building all the time, and
every time everyone else is locked out of checking-in changes for 15
minutes... "I better press this button quickly, before Tom, Dick or Harry
press theirs. I think I'll check in my changes before checking them, otherwise
I'll miss this opportunity".

-- 
Nadav Har'El                        |       Sunday, May  6 2001, 13 Iyyar 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |Drink varnish and you'll get a lovely
http://nadav.harel.org.il           |finish.

---

From: "Nadav Har'El" <[email-masked]>
Date: Sun, 6 May 2001 13:38:55 +0300

> another looooong cup of coffee. hope that's not the case there ;)

When I came to my previous employer, they had there a piece of software around
100,000 lines that had to be compiled often (because running it always involved
changing some code - don't ask why...). Anyway, the compile script they had
(written in a weird combination of Fortran and Csh, argghh!!) had to decide
what to compile based on what depended on what, and they did it in an
incredibly inefficient manner: the source files were being read dozens of
times looking for "include" statements, rather then being read once and that
information kept in memory.

Each time the program was compiled, it took around 15 minutes, which was
a real hindrance to its interactive use. And yet, nobody complained - they
just took it as a fact-of-life. One of the first things I did there was to
convert this build process to C, and doing this efficiently (i.e., traversing
the source files only once, keeping the dependency tree in memory, and even
keeping it cached on file so next time I will not need to look at the source
files at all). I also changed other things on how the build was done (e.g.,
keep the temporary library files in /tmp, rather than the current directory
which was typically NFS-mounted, for a much better response time from the
compiler and linker).

After my change, the typical build was reduced to around 2 minutes, on the
same computer. People were stunned :)

Reminds me of the Mo'ach vs. Co'ach proverbs.

-- 
Nadav Har'El                        |       Sunday, May  6 2001, 13 Iyyar 5761
[email-masked]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |I would give my right arm to be
http://nadav.harel.org.il           |ambidextrous.

---

From: guy keren <[email-masked]>
Date: Mon, 7 May 2001 09:58:31 +0300 (EET DST)

> phase, that usually can't be parallelized).

i assume they normally use dependencies systems. however, in order to
avoid getting out-of-sync code, you sometimes perform full system
rebuilds. for example, when common basic baselines below you change
significantly, its better to have your complete baseline rebuilt, and
building only part of it, and later finding yourself chasing phantom bugs,
that resulted from de-synchronized object files.

> I'll miss this opportunity".

if that was the case there, people wouldn't be too happy about it. you are
taking a few things out of line here:

1. you don't perform code check-ins very often. you work on your code for
   several days before checking things in.
2. when you do have smaller cycles (cause you fixed a small bug), you
   don't make a full recompilation, so the 'lock-out' period is much much
   shorter.
3. if you made changes to code files (for C, C++ or Java), then you only
   need to ercompile that file and relink it - hence the lock-out period
   is very short as well.

it is possible to have the complete baseline copied to a "real public
workspace" after it was compiled in teh "private-public workspace". i
suggest when you build your next project, you indeed use this method. from
seeing cases when this was NOT done this way, i can see that this is a
good solution (in the largest case where this was not done, people
eventually started doing so anyways, by creating copies of the public
workspace of other projects, for each team of programmers.

--
guy