Subject: Maintenance of 10,000,000 lines long applications?

From: Omer Zak <[email-masked]>
Date: Sat, 5 May 2001 11:50:40 +0300 (IDT)

Is there anyone in the group with experience in the above?

What key practices and insights helped you in maintaining extremely large
software applications?

Any interesting war stories (especially about projects which were
successfully maintained)?

                                             --- Omer
The IGLU Cabal Theorem:
     "The only software engineering methodology known not to fail
      in very large projects is the principle of modular design."
WARNING TO SPAMMERS:  see at http://www.zak.co.il/spamwarning.html

---

From: guy keren <[email-masked]>
Date: Sun, 6 May 2001 06:52:01 +0300 (EET DST)

> Is there anyone in the group with experience in the above?

more like a 2,000,000 lines long project (more or less), and it wasn't a
single application - it was rather a large set of processes. and it was a
long-lasting thing (original code base was started at 1991, the project
still runs up-to-date). hope that's good enough as another point of
reference.

> What key practices and insights helped you in maintaining extremely large

keeping everything in one's head. :) i.e. you have to have at least
someone in the project that has an idea where everything is placed.
documentation is a good thing as well, in getting new people into the
project. i know it saved a lot of time for people on many cases, where
instead of having to dig into the source to find something, they could ask
someone to tell them where that part is, who who knows that part properly.

another aspect that is important is having more then one person to know
(from the inside, not just on the surface) how to maintain each part of
the code. this is very important since people take vacations, and people
leave the project, etc. a good way to achieve this is by either doing code
reviews (not in large groups! only 2-3 people at a time), or moving code
modules to new programmers and letting the original ones move on to other
code parts.

one other important thing is dividing maintanance properly between
different people, to avoid having a bottleneck. in my case, there were
times where too much of the "middleware" and communications code was
maintained by me, and that caused delays in responding to people on the
project.

finally, don't waste too much time on finding a very good version control
system - spend it on getting a good configuration manager, and one that
can handle the code itself. one thing about such large projects is that
when people check-in their code changes, it sometimes causes people to get
stuck with the code for a while, cause someone forgot to check backwards
compatibility of their code. thus, you'd need to have many integration
environments, in which people check-in code changes gradually, while not
yet checking it to the main source branch, and you have to train people at
being very very acrefull about how they make changes to the code.

thus, beginner programmers should be allowed to work mostly on external
modules, that are not used by other modules, and when you see they are
good and can be trusted, gradually let them work on more internal modules.

one way i found that was good to overocme this problem, is by having
interfaces being backwards-compatible in a binary level. you can achive
this in several methods:

1. never change interfaces, just add more methods to them.
2. when an interface returns data, have it return it as a set of
   attributes, rather then as a list of compile-time known parameters.
   tihs way, if you ned to erturn new data, you don't break source-level
   or binary compatibility for modules that use this interface. in order
   to make it yet easy to work with these interfaces, write thin-wrappers
   that sit on top of these attribute-level interfaces. if you want, i
   could try giving a more concrete example of such an interface.

you get more flexibility and easier path of enhancing interfaces, at the
price of loosing performance (attribute-level interfaces take more memory
resoruces for data transfer then regular parameter-level interfaces do,
and take more CPU cycles to process). note that i've used such interfaces
in distributed applications of that project, and usually they didn't cause
too much overhead, and they made our updates life much easier then another
part of the project, that used parameter-level interfaces, and had messes
at least half the times they enhanced their interfaces.

> successfully maintained)?

i think everyone that dealt with such projects would have endless war
stories to tell about them. i'm not sure it's to interesting for anyone
outside the project, thought :0

--
guy

"For world domination - press 1,
 or dial 0, and please hold, for the creator." -- nob o. dy

---

From: Oleg Goldshmidt <[email-masked]>
Date: 06 May 2001 10:35:15 +0300

> Is there anyone in the group with experience in the above?

All by myself or within a small group of people - not here. A
30,000,000 plus lines system in a large organization, maintaining
significant chunks of the above - yes. Relevant?

-- 
Oleg Goldshmidt | [email-masked] 
"I'd rather write programs to write programs than write programs."

---

From: Omer Zak <[email-masked]>
Date: Sun, 6 May 2001 11:08:39 +0300 (EET DST)

> significant chunks of the above - yes. Relevant?

Since I am the one who asked the question, I get to rule whether Oleg's
experience is relevant to my question.

Accordingly, I am laying down the rule that the criteria for determining
relevancy is one's ability to contribute to the discussion and offer one's
insights.

Therefore Oleg's experience is hereby declared as relevant, and Oleg is
invited to offer his observations and insights.

                                             --- Omer
There is no IGLU Cabal.  Due to lack of consciousness of weaknesses of
methodologies, they got bogged down when trying to maintain a software
package having only about 1,000,000,000 code lines.  This happened even
though they rigorously applied the very best software development
methodologies that money, ego or wisdom can buy.
WARNING TO SPAMMERS:  at http://www.zak.co.il/spamwarning.html

---

From: Oleg Goldshmidt <[email-masked]>
Date: 07 May 2001 00:55:07 -0400

> > significant chunks of the above - yes. Relevant?

<snip>

> invited to offer his observations and insights.

OK, I couldn't afford to take the time while at the office... Check
the length of what follows - you'll see why. You've been warned.

I have had some (not insignificant) experience developing parts of a
big (30M lines of Fortran and C, not counting tools, utilities, etc is
big) system in a big (hundreds, probably of the order of a thousand,
developers) organization. I was not responsible for managing the 30M
lines, but I had a lot of responsibility for developing a piece of the
heap, and I had to deal with the issues in the 30M lines as a
developer, often to my great frustration.

I will be assuming UNIX or Linux and free software development tools
mostly (they are adequate) - feel free to substitute your favourite
commercial tool if you feel like it.

I will assume many developers (10M lines and more are not likely to be
developed by a single or a very small number of programmers).

I will assume that production code releases are very frequent - much
more frequent than a CD once in 6 months. That's my experience, and I
suspect that treating snapshots as "production" often enough is not
such a bad idea even if the real release is months away.

I will assume that all of the code is mission-critical.

What follows are random bits and pieces in no particular order. I will
not cross-reference the bits and pieces, if you indend to flame please
read everything first. I used all the tools I describe below, and
developed some of them. It is not a fantasy. Some of the tools I used
were poorly implemented, but I believe the ideas were right. [Needless
to say, the tools I developed were designed and implemented
brilliantly - trust me. :-)]

I do not claim it's the best way to run a big development shop. I am
simply describing my experiences.

- A team is needed, consisting of several people responsible for the
  whole process of building the whole shebang, including custom tools,
  scripts, policies, etc. These guys and gals should be intimately
  familiar with the development process, and be aces as far as
  development tools and scripting (for creation of custom tools - they
  will be needed for a system of this size) are concerned. These
  people, together with the most experienced developers, should take
  care of various build- and maintenance-related policies. They will
  also handle all the inquiries from developers regarding the build
  process and tools.  I'll call them "the librarians". I don't envy
  them.

- The scripts, tools, utilities to be developed by the librarians are
  crucial. They should be of "production quality" for all intents and
  purposes - well designed, extensible, maintainable, robust, clearly
  documented, extensively tested, open source (for the developers). It
  is a job for experienced software engineers, not for high school
  kids. Well, maybe exceptionally talented high school kids or
  students will do, but one of the main requirements for such a job is
  discipline, and the ability to impose development discipline on
  programmers. Since normally programmers will not be subordinate to
  the librarians, it is essential that the former have a lot of
  respect for the latter, on merit.

- Introduce a discipline into the program design and coding. Some
  system of design and code review for any change in production code
  is essential. Among the 30M lines of code I've seen pieces that not
  only could not be changed, but could not even be re-written from
  scratch. I was willing to undertake a re-write, but it was
  absolutely impossible (you requested war stories, Omer?).

- Introduce the developers to the tools of the trade, both common
  (compilers, debuggers, profilers, revision control, make, etc)
  and custom, developed internally in your organization. Teach them
  how to use them correctly and efficiently.

- Modularize. It is unlikely that you'll put all the code in one huge
  library. You will have many libraries, each built from many source
  files (separate projects that will most probably be
  interdependent). The reasons should be obvious. Code modularization
  for each library is a separate issue.

- Each library will probably have its own makefile. While you can make
  the makefile a responsibility of the librarians, letting the
  developers maintain the makefile makes more sense IMHO. In the
  latter case though, the developers will be responsible for the
  makefile in the same way (only more so) as they are responsible 
  for the code. Assuming each library sits in its own directory tree,
  the top-level makefile (which the librarians will use to build the
  whole thing) should be able to do "$(MAKE) -C foo" to build libfoo
  in directory foo. Well, something of the kind - you get the picture.

- The librarians can (I recommend it) provide boilerplate makefiles
  with a number of standard targets useful during development that
  can be easily included in each library's makefile. The latter will
  be reduced to a source listing and other project-specific stuff.

- Everything - source code, makefiles, utilities, tools, scripts,
  docs, etc should be under revision control. CVS is quite adequate.
  I'll assume CVS in what follows. Adjust to your favourite tool.

- Teach the developers to use CVS tags and branches. The developers
  should be encouraged to make small incremental changes, whether or
  not they compile. When a developer is satisfied that a revision
  he/she is working on is "good enough" (deliberately vague, but at
  least it should compile and link with the rest of the stuff cleanly)
  it can be committed with a "development" tag. A version ready for
  production (definitely works properly, thoroughly tested, etc) can
  be committed with a "production" tag. This will allow development
  and production versions to be built automatically (from scripts)
  based on CVS tags. 

- A mechanism to roll the tags (to be able to return to the previous
  production or previous development versions, or to the version that
  was in production on June 31, 2000) is needed. CVS provides what is
  needed, but the librarians will have to do some scripting to make it
  easy. It is possible to provide make targets for that (see
  boilerplate makefiles above).

- Automatic builds can and should be used (meaning extensive
  script-writing by librarians): cvs update (with the appropriate
  tags) followed by make, you fill in the details. The developers
  should know the schedule (e.g. development builds nightly at 3AM,
  production builds weekly, etc), and adhere to deadlines.

- The automatic builds should produce extensive logs that should be
  examined by the librarians for any errors. Automating this task is
  up to the librarians. Problems should be reported to developers, the
  extent of public fingerpointing is up the the librarians and the
  corporate culture. Finding out who checked broken code in should be
  easy with CVS tools (cvs {status,log,annotate}).

- A broadcast should be sent (by email) to notify the developers when
  a build is complete, and the new production build should be
  available for final testing before actually moving to
  production. Easy access to the previous build should be provided,
  so that the new and old ones could be compared.

- The CVS commits (tagged as development or production) should be
  customized. The more checks are made at commit time the better. 
  Certainly one can rig CVS to verify clean compilation (more
  extensive script-writing by librarians). Forcing the committer to
  examine the diff before proceeding is generally a good idea. 

- Programmers should be able to receive notifications about changes
  made by others. Forcing everybody to get emails about changes made
  by everybody else is probably unreasonable in a big organization.
  It should be possible though to enable developers to subscribe to
  CVS update messages on a module-by-module basis. Developers should
  be educated in writing clear and useful log messages (emphasis on
  why, not what).

- A developer working on a piece of code (typically just a few files)
  should be able to easily build a shared library out of it and link
  it to the rest of the system for testing purposes. Linking millions
  of code lines is not time well spent. Linking a tiny .so is much
  more efficient. A tool (based on make, I guess) making this easy
  should be developed by the librarians.

- In a system of this size undefined objects will inevitably appear
  from time to time. It is possible (though non-trivial) to create a
  tool that will identify the undefineds and attempt to add the right
  libs to the link statement and re-link. At the same time, a tool
  that will stub the undefineds after several unsuccessful attempts to
  resolve them is needed. The build completion notification (see
  above) should list the stubs and the responsible developers should
  fix the problems.

- Use GNATS or another bug-tracking system to report bugs, keep track
  of ideas, etc. GNATS is a reasonably convenient database where one
  can store information, ideas, email echanges, etc pertaining to a
  project or a problem, and keep track of the status. Copy emails to
  [email-masked] liberally - later you'll be glad that you did.

- Use compilers with extensive debugging capabilities. We've had very
  good experience with Insure from ParaSoft (commercial). ParaSoft
  offer tools for TCA, automatic unit testing, etc. Do use stuff like
  that, from whatever vendor.

- A crucial element is tracking dependencies. This absolutely must be
  automatic - there is no chance in hell anyone can maintain it by
  hand across a big system *effectively*. While gcc -M? provides the
  basic functionality for C/C++, it can be improved significantly. Do
  it through make - make is a tool for tracking dependencies, after
  all. For other languages, such as Fortran, tools need to be found,
  bought, or developed. If you do it right, your biggest headache will
  have been taken care of.

- If you write C code, another thing that must be automated is
  prototype maintenance. A tool like cproto provides the
  functionality, but juggling the dependencies between prototypes,
  dependency files, lex/yacc code, etc may be non-trivial. It all can
  be done through make, of course. be careful - it *is* non-trivial.

- Pay special attention to a uniform method of error checking and
  handling throughout the code.

- Do trust your programmers to a reasonable extent. It's no use to
  forbid them to do certain things (malloc and free were forbidden -
  without management authorization - in one development environment I
  encountered, under the excuse that they were dangerous). The
  smartest of them will find a way to sneak forbidden stuff in anyway,
  if they find it useful. They'll teach the rest of the bell curve how
  to do it, too. Instead, do educate your programmers. None of them is
  stupid (you don't hire idiots, do you?), and all are able to learn.

- Run a mailing list or some sort of chat forum for all the developers
  to discuss - as close to real time as possible - development,
  maintenance, build issues.

Are you bored yet? G'nite.