Maintenance of 10,000,000 lines long applications?
- Is there anyone in the group with experience in the above?
What key practices and insights helped you in maintaining extremely large
Any interesting war stories (especially about projects which were
The IGLU Cabal Theorem:
"The only software engineering methodology known not to fail
in very large projects is the principle of modular design."
WARNING TO SPAMMERS: see at http://www.zak.co.il/spamwarning.html
- On Sat, 5 May 2001, Omer Zak wrote:
> Is there anyone in the group with experience in the above?more like a 2,000,000 lines long project (more or less), and it wasn't a
single application - it was rather a large set of processes. and it was a
long-lasting thing (original code base was started at 1991, the project
still runs up-to-date). hope that's good enough as another point of
> What key practices and insights helped you in maintaining extremely largekeeping everything in one's head. :) i.e. you have to have at least
> software applications?
someone in the project that has an idea where everything is placed.
documentation is a good thing as well, in getting new people into the
project. i know it saved a lot of time for people on many cases, where
instead of having to dig into the source to find something, they could ask
someone to tell them where that part is, who who knows that part properly.
another aspect that is important is having more then one person to know
(from the inside, not just on the surface) how to maintain each part of
the code. this is very important since people take vacations, and people
leave the project, etc. a good way to achieve this is by either doing code
reviews (not in large groups! only 2-3 people at a time), or moving code
modules to new programmers and letting the original ones move on to other
one other important thing is dividing maintanance properly between
different people, to avoid having a bottleneck. in my case, there were
times where too much of the "middleware" and communications code was
maintained by me, and that caused delays in responding to people on the
finally, don't waste too much time on finding a very good version control
system - spend it on getting a good configuration manager, and one that
can handle the code itself. one thing about such large projects is that
when people check-in their code changes, it sometimes causes people to get
stuck with the code for a while, cause someone forgot to check backwards
compatibility of their code. thus, you'd need to have many integration
environments, in which people check-in code changes gradually, while not
yet checking it to the main source branch, and you have to train people at
being very very acrefull about how they make changes to the code.
thus, beginner programmers should be allowed to work mostly on external
modules, that are not used by other modules, and when you see they are
good and can be trusted, gradually let them work on more internal modules.
one way i found that was good to overocme this problem, is by having
interfaces being backwards-compatible in a binary level. you can achive
this in several methods:
1. never change interfaces, just add more methods to them.
2. when an interface returns data, have it return it as a set of
attributes, rather then as a list of compile-time known parameters.
tihs way, if you ned to erturn new data, you don't break source-level
or binary compatibility for modules that use this interface. in order
to make it yet easy to work with these interfaces, write thin-wrappers
that sit on top of these attribute-level interfaces. if you want, i
could try giving a more concrete example of such an interface.
you get more flexibility and easier path of enhancing interfaces, at the
price of loosing performance (attribute-level interfaces take more memory
resoruces for data transfer then regular parameter-level interfaces do,
and take more CPU cycles to process). note that i've used such interfaces
in distributed applications of that project, and usually they didn't cause
too much overhead, and they made our updates life much easier then another
part of the project, that used parameter-level interfaces, and had messes
at least half the times they enhanced their interfaces.
> Any interesting war stories (especially about projects which werei think everyone that dealt with such projects would have endless war
> successfully maintained)?
stories to tell about them. i'm not sure it's to interesting for anyone
outside the project, thought :0
"For world domination - press 1,
or dial 0, and please hold, for the creator." -- nob o. dy
- Omer Zak <omerz@...> writes:
> Is there anyone in the group with experience in the above?All by myself or within a small group of people - not here. A
30,000,000 plus lines system in a large organization, maintaining
significant chunks of the above - yes. Relevant?
Oleg Goldshmidt | ogoldshmidt@...
"I'd rather write programs to write programs than write programs."
- On 6 May 2001, Oleg Goldshmidt wrote:
> Omer Zak <omerz@...> writes:Since I am the one who asked the question, I get to rule whether Oleg's
> > Is there anyone in the group with experience in the above?
> All by myself or within a small group of people - not here. A
> 30,000,000 plus lines system in a large organization, maintaining
> significant chunks of the above - yes. Relevant?
experience is relevant to my question.
Accordingly, I am laying down the rule that the criteria for determining
relevancy is one's ability to contribute to the discussion and offer one's
Therefore Oleg's experience is hereby declared as relevant, and Oleg is
invited to offer his observations and insights.
There is no IGLU Cabal. Due to lack of consciousness of weaknesses of
methodologies, they got bogged down when trying to maintain a software
package having only about 1,000,000,000 code lines. This happened even
though they rigorously applied the very best software development
methodologies that money, ego or wisdom can buy.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html