Subject: [hackers-il] Re: Reading Open Source Code

From: "Tzahi Fadida" <[email-masked]>
Date: Mon, 29 Dec 2003 23:07:59 +0200

> -----Original Message-----
> From: Shlomi Fish [mailto:[email-masked]]
> Sent: Sunday, December 28, 2003 3:54 PM
> To: Hackers-IL
> Subject: RE: [hackers-il] Re: Reading Open Source Code
>
>
> On Sun, 28 Dec 2003, Tzahi Fadida wrote:
>
> > > > >
> > > > > > Aside from that, i think many open source projects would
> benefit from
> > > > > > a good design.
> > > > >
> > > > > Naturally. But so would a great deal of in-house or even commercial
> > > > > software, some of which was heavily designed and written with a lot of
> > > > > "red tape". If your project suffers from bad design, you
> should refactor
> > > > > it, so it will have a good one.
> > > > >
> > > > > Otherwise, (I may have misinterpreted you) it is always a good idea to
> > > > > design and plan a change before you start hacking on it.
> Unless of course
> > > > > we are talking about mindless logical transformations, where you just
> > > > > change stuff in the code.
> > > >
> > > > yeah,
> > > > before, during and make sure that the maintainers also maintain the
> > > > design.
> > > >
> > >
> > > Or make it even more modular to form an even better design. Software is
> > > not static and what is an adequate design for a software that does A,B and
> > > C, may not necessarily be good for a software that needs to do A,B,C and
> > > D.
> >
> > well, scalability is always good. the reason UML, which is the most widely
> > used, doesn't seem to be loved by many is because they fail to
> > understand it does not scale. the modularity issue is just an illusion,
> > and the "package" attempt at one of the models is pathetic.
> > A good scaling must come with more attention to details.
> > again, i don't say to use it with open source because of patents and
> > licensing issue, but see OPCAT at the IE IS department.
> > I wish we had something more free, or maybe there is and i don't know
> > about it. if not, please enlighten me.
> >
>
> I felt UML was too nitty-gritty as well. Too many diagrams to represent
> the micro-management of the code instead of the really important stuff.
> Thanks for enlightening me in this regard!
>
> But you can do good design without any official diagrams and with only
> a-posteriori diagrams that are given to illustrate various points (and
> need not necessary be anything official like UML).

well, this is often misleading. I am now using methodologies for about
10 years and i can't live without them now.
at the start ERD and DFD was all that i needed and all that i knew about.
after that, i discovered UML class diagram, and now i mostly use
the UML business class diagram with the three tier model as stereotypes.
these are very handy nowdays.
Anyway, i since started to work on Agent technology and so i evaluated
several agent design methodologies. GAIA strikes out as the best for
a good role design, however it has many shortcommings. others are
lagging behind but more are comming.

>
> > > > where one of the doctoral works was presented. it was on behavioural
> > > > stuff so that part isn't important here, however, they talked about
> > > > how they do it in companies and what are the limitations and offers to
> > > > make it better etc...
> > > > Now, here comes the interesting part - While i liked it in general for
> > > > small to medium term projects the inhouse staff of Information Systems
> > > > almost kicked them down the stairs. I myself feel that of course, for
> > > > big and "important" projects where you need a good design to refer
> > > > to where there is a problem this kind of methodology is out of place.
> > >
> > > Perhaps. Albeit a great deal of large scale open source projects have
> > > emerged recently and have been developed in a Bazaar way. Usually, you
> > > divide such projects into sub-components and define well-defined
> > > interfaces for making use of each component. Then the internal design of
> > > each component is left as the "private zone" of its core developers, and
> > > the other developers of the project need not be concerned about it.
> >
> > yeah, the blackbox idea. however, this is an ad hoc methods. i.e.
> > if for example someone decided to use one driver, and then someone
> > decided to use a different one, and now you have to install 2 drivers.
> > etc... you need to have an overall look. you can design using for example,
> > the class model in UML and as you probably know, its not getting to
> > specifics there, and so 90% is left for the "private zone" to design.
> >
>
> Well, this is one problem that is prominent in the open source world any
> way you look at it. There's duplication of work anywhere. What's good for
> Linux needs to be duplicated in the BSDs. What's good for GNOME needs to
> be implemented in KDE as well. We have a gazillion Wikis, window
> managers, editors, soundwave editors, bug tracking systems, CMSes, etc.
> etc. Of course, this problem is not restricted to the open source world.
> For instance Microsoft have released a large number of database packages:
>
> ODBC, RDO, DAO, ADO, OLEDB, ADO.NET
>
> All by the same company under the same roof. So I guess you can't really
> avoid it, albeit can do your best to minimize it. Perhaps the only true
> way to do so is to break backwards compatiblity if necessary.

right, but you can at least anticipate this in advance and provide
appropriate api's instead of years of angry mails to developers.

>
> > > > systems that must have a good design. one example would be the black
> > > > boxes in airplanes or in the robots they send to mars. This can be done
> > > > using mathematical models that can be checked by a computer. Of course
> > > > it would be impossible for large scale projects to be built like this.
> > > > So, the next best thing is to take something like GAIA for agents,
> > > > or UML for small projects, etc... to each appropriate field of software,
> > > > and make sure it works.
> > > >
> > > > THE MAIN PROBLEM i see here is that in CS and any other faculty in the
> > > > technion, i never heard of the IEEE/ACM Code of Ethics(c IEEE/ACM
> > > > 1999). this code is like the doctors - not to do harm and negligence.
> > > > see a Book:
> > > > Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1
> > > >
> > >
> > > I saw the IEEE code of ethics once in writing. Can't remember it all.
> > > However, I'm not sure it says anything specific about which
> > > software engineering methodologies to use.
> >
> > of course not, its a code of ethics, but it certainly defines what is
> > acceptable.
> > it even starts: "those who contribute by direct participation or by
> > teaching, to the analysis, specification, design, development,certification,
> > maintainence...."
> > http://www.acm.org/serving/se/code.htm
> >
>
> Right. But I'm not sure XP will be frowned upon it in much regard. After
> all it has test-driven development, pair programming (and so catching a
> lot of bugs, and good peer review), refactoring and other elements that
> increase the quality of the code. I'm not sure if it's entirely suitable
> for developing software to guide missiles, but it does stress quality.

I am just saying its no good for big projects like SAPing :) the navy force
or a govrenement branch.

>
> > >
> > > >
> > > > >
> > > > > > However, this projects suffer from problems that comes from
> > > mistakes just
> > > > > > like that. Mistakes, that are apparent when the project
> needs to grow
> > > > > > beyond the 12 people stage.
> > > > >
> > > > > Luckily, most projects don't need to grow beyond the 12
> people stage. Most
> > > > > projects have a core team of 1 or 2 developrs, and a halo of
> a bit more.
> > > > > I'm personally involved in Subversion, where the number of
> contributors is
> > > > > very large, and yet it's in pretty good shape. Still, the
> number of people
> > > > > doing most of the work is relatively small.
> > > > >
> > > > > I think open source has proved that its projects can scale to a large
> > > > > number of developers, without grinding to a halt, or its code
> > > > > deteriorating into a very low quality mess. The reasons why
> this is the
> > > > > case were discussed in the "Go! Linux" panel, and also in many other
> > > > > places and occasions.
> > > >
> > > > I'll have to check that. I guess good will and "this is my baby
> and i know
> > > > anything about it"  is a good part of that.
> > > >
> > >
> > > Some of the points that were raised (by the Go! Linux Panelers and
> > > others):
> > >
> > > 1. By splitting the code into components it is possible to divide the work
> > > and make sure it scales.
> > > 2. A great deal of code is not automatically accepted into a project and
> > > its correctness, style and quality is criticized by many reviewers before
> > > it is accepted.
> > > 3. In many software shops, programmers are terriorial about their code.
> > > Those shops in which programmers encourage others to take a look at their
> > > code, look for potential bugs and suggest or implement improvements,
> > > eventually see a boast in their productivity. Open Source is like that
> > > only on steroids.
> > > 4. (my own) In many open source projects there is less "red tape" than in
> > > those projects that are run by software houses. Thus, a programmer can
> > > achieve more as he is not restricted by factors induced by his workplace.
> > > 5. (Homesteading the Noosphere). It was shown that incentive pay (bonuses,
> > > etc.), actually reduce the output of programmers, and workers of other
> > > professions in which there is a genuine mental effort involved in the
> > > work. In fact, what can encourage them is letting them work on things
> > > they desire to work on. Thus, open source hackers who receive little
> > > monetary incentive and work on what they like embody the ideal of this
> > > approach.
> >
> > the last,5, is a known phenomena in psychology, or at lest that's what
> > we learned in one of the cognitive courses i took.
> > I am a bit skeptic about the conclusion, because it does not solve anything
> > but avoid the phenomena. on the same respect, if you lower a person's
> > salary or cancel a bonus, you will get the effect of more productivity.
> > should that mean that we should start dissing the work of open source
> > developers? :)
> >
>
> Maybe you should read "Homesteading the Noosphere" to get more
> understanding. Lowering a salary of a programmer who gets paid to do
> programming, will likely be a demotivator as well, because he'll feel that
> he's no good. An increase in salary, can in fact motivate if presented
> correctly, or if done as part of a desireable promotion.
>
> I agree that when ESR is taking this phenomenon to extreme, it may not
> necessarily be a correct thing to do scientifically. But it's might as
> well have a grain of truth to it.

Any psychology prof. will tell you not to take their studies too seriously as
i can assure you many companies do. i.e. many extend psychological
principals from lab experiments to real life when they should ultimately
conclude that it only worked in the lab, how can they be sure it would
work in other situation in a non controled environment...

>
> >
> > >
> > > > >
> > > > > > I didn't suggest this before because i don't have the time, and
> > > the usual
> > > > > > response would be to do it yourself but i think that someone should
> > > > > > round up a group of people that would design projects based on
> > > > > > requirements for teams that do open source projects.
> > > > > >
> > > > >
> > > > > I don't quite understand this last sentence, but I'll try. I think you
> > > > > mean that they would design tools to aid in quality control, software
> > > > > management, etc. of open source projects. Like
> SourceForge/GForge, only on
> > > > > steroids. Well, that may be a good idea, but of course will
> involve a lot
> > > > > of work.
> > > > >
> > > > > If you want these tools to be programming language-aware,
> then you'll need
> > > > > to handle all the different languages out there. Good luck to whoever
> > > > > choose to go this way.
> > > >
> > > > Though a really good idea, and i don't think its really possible using
> > > > regular methodologies like UML for reasons that would take a course time
> > > > to explain. Mainly due to the many models. Orthogonality plays a crucial
> > > > roles in this matters.
> > > > Anyway the IE IS area has a new methodology called OPM, which
> > > > I must state here first that its patented and thus i have
> personal feelings
> > > > about it, though it maybe continues our last discussion of universities
> > > > internal funding sources. this methodology is the new wave suppose
> > > > to supercede UML using one model for all, and supposed to be able
> > > > to one model - > every programming language.
> > > > And I for one, believe it is possible.
> > > > Ok, comming back to the issue, i was talking about OUTSOURCING
> > > > open source style. i.e. people with design skills at different areas
> > > > with different methodologies would design projects with the base
> > > > requirements and maybe a meeting place between designers and
> > > > project developers that will commision designs and refactoring from
> > > > them.
> > >
> > > This could be a problem for one good reason: as ESR identifies in:
> > >
> > > http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar
> > > 01s10.html
> > >
> > > You have to start a project by writing a working version of the code. It
> > > may not have to do everything the project needs to do in its final
> > > version, but it has to prove that the source code has a future in it. Most
> > > projects that start by forming a mailing list and writing a future arch
> > > doc or whitepaper wither away into oblivion, without having too much to
> > > show for.
> >
> > well, this is a murphy's law kind of thing. usually inexperienced or
> > experienced people from different reasons don't have the time or
> > streangth to continue a project and what the quoted reason means is
> > that the left overs can be continued by someone else, but if there aren't
> > any then its a problem.
> > I think its negligable, the same can be argued for someone who leaves
> > a message about an idea without implementing it because he doesn't have
> > the time. (like i did)
> >
>
> Right.
>
> > >
> > > I can testify from experience that this was the case for my projects.
> > > Thus, I always make sure that if I wish to release a project into the
> > > wild, I first sit down and write a version that achieves a small subset of
> > > the planned featureset.
> >
> > i know the feeling, but its a psychological thing and not a
> scientific reason.
> > i even skipped a few myself, but not before i told them in some mailinglist
> > so others, if they have the streangth, can implement.
>
> Well, you are right that it is mostly psychological and not scientific.
> Some projects that started with a mailing list and a planningware
> announcement are flourishing now. Many others which still have a working
> codebase are now neglected.
>
> In any case, be aware that as nice ideas are. They don't require a lot of
> effort. An individual needs much more effort in developing a good product
> out of it, mass producing it and mass distributing it. (or whatever is the
> equivalent for software).

right

>
> > an example, is the idea that not using an open source documents
> > and software in govrenment facilities is against the freedom of information
> > law before the committe about this thing convened for the first time. i
> > corresponded with rabia and others and even got the law copy.
> > but of course i didn't try to get on the amuta wagon when a few of us
> > raised the issue of amuta several times in linux-il. i don't have
> the time, but
> > it doesn't mean someone else can't do it.
> >
>
> Well, if you're not going to do it, either someone else will and get most
> of the credit, or it won't get done at all. We all share the same
> existence.

I have a rule, if i am not going to do it, i don't care someone else will
take the credit. again, the psychological thing.

>
> > >
> > > However, your idea may hold for re-designing projects, refactoring them or
> > > improving them. I know that Subversion started from a vapourware design
> > > that a certain RedHat employee designed, and let others implement. The
> > > latter is well-known for this tendency.
> >
> > well, my idea is about who ever wants a design for whatever reason can
> > commision one from the community. so yeah.
> >
>
> Well, I suppose you can post a message to mailing lists or web forums
> asking for a good design for a program that does A,B and C. Many people
> will be happy to suggest and discuss such good designs.
>
> If you want to set up a site, then a rudimentary one using some web-log
> system (Slash, Post-Nuke, Scoop, etc.) should be very easy to do and
> you'll only need hosting. It may actually prove to be quite popular.

yes, it may be so.
I just have to find the time. maybe we could integrate this into haifux.
i.e. close in person development support for open source projects
or even wannabe, etc..

>
> > >
> > > > The main problem i see that this is supposed to solve, is the
> experienced
> > > > designer issues, and the knowledge time - to - develop issues.
> > > >
> > > > >
> > > > > > anyway, back to the main issue. I think that reading an open source
> > > > > > would be much easier if it was well designed using a proven
> methodology
> > > > > > and "enough" documented so that you can at least pinpoint the
> > > > > > whereabout of a glitch and fix it in half the time it took you.
> > > > > >
> > > > >
> > > > > "proven methodology" - do such beasts exist? I think you've
> spent too much
> > > > > time being an industrial engineer. Have you ever read "The Mythical
> > > > > Man-Month"?
> > > >
> > > > yes there are, though rare. mathematical models which are "safe" can
> > > > be proved, however these are for small scale. Mostly a few
> lines of code.
> > > > No, i was talking about how much time they exists and if they have a
> > > > record track of successful usage.
> > > >
> > >
> > > Well, the Bazaar model so far proved itself spectacularily, at least
> > > according to the eyes of observers. I don't know about XP, though there
> > > are reports it had been successful for certain projects in its
> > > target niche. It reminds me of the book "Facts and Fallacies of Software
> > > Engineering" which aims to analyze which trends increase productivity and
> > > which don't based on research and statistics:
> > >
> > > http://www.amazon.com/exec/obidos/tg/detail/-/0321117425/joelonsoftwar
> > > e/102-6670515-1781755
> >
> > i personally doubt the empirical research of such things because they
> > tend to be biased to a certain methodology. usually because, for example,
> > XP was concived by companies (I think IBM or HP, can't remember) for
> > companies. Companies usually like to cut corners, and so i would refer
> > those companies to the ACM code of ethics.
> >
>
> Well, this book is supposed to be based on a great deal of researches
> conducted. I did not read it, so I cannot testify for how well it is
> based.
>
> > >
> > > > >
> > > > > Also, define "adequately documented"? Do you want a Literate
> Programming
> > > > > Style book documenting the program? Do you want a detailed
> architecture
> > > > > document? Is a user's guide enough? Do you agree with what XP
> says that
> > > > > even comments should usually be unnecessary and the code should be
> > > > > self-explanatory?
> > > >
> > > > don't count of self-explanatory, this rarely happen when a multitude of
> > > > people write different bits of code.
> > >
> > > Probably true. That's why many software projects have style guidelines and
> > > review every contribution before its inclusion.
> >
> > right.
> >
> > >
> > > > i can't define here what is adequate precisely because every project
> > > > is different. I can only say that it should be adequate in the sense
> > > > that further down the line the code could be maintained using this
> > > > documentation.
> > > > This is the XP way of thinking - "adequate enough".
> > > > I am still ambivalent about this so i can't be accurate.
> > > >
> > >
> > > OK. Eventually, by experiencing enough with the code, you can eventually
> > > understand even the most obfuscated one. Even a code with tons of
> > > comments, meaningful variable names, a consistent style convention, an
> > > organized architecture document, API documentation, etc. may not
> > > necessarily be easy to understand. And like I said I many times
> > > encountered code that had only a small subset of these things, and was
> > > very clear and easy to modify.
> > >
> > > I cannot testify I've been a saint in this regard. Between version 2.4.x
> > > and 2.6.x of Freecell Solver, I added a great deal of optimizations
> > > some of which I feel may have decreased readability a great deal. But they
> > > did made the code 1.5 times faster, which is still significant.
> >
> > right.
> >
> > >
> > > > >
> > > > > I'm not saying a code should not be documented at all. However,
> > > > > documenting a working code, and keeping the documentation in
> sync with the
> > > > > code is very difficult. The best way to understand a code is to ask a
> > > > > programmer who worked on it extensively.
> > > >
> > > > yes. again you are quoting XP, and i'll requote traditional
> methodologies.
> > > >
> > >
> > > I am not quoting XP as I'm not entirely familiar with it. I'm speaking
> > > from experience and intuition. Maybe my opinions may seem XPish to you,
> > > but maybe that's what my experience has taught me.
> >
> > XP was developed from experience rather than science, and so it seems
> > to integrate perfectly into one's practicies. however, it should be
> carefully
> > checked when used.
> >
>
> "experience rather than science". Isn't science supposed to be
> experimental? If someone has a theory, he needs to prove it actually works
> in the real world. This is very difficult to do in software engineering
> and team management where the success of the team is dependant on a great
> deal of many factors. But it is still necessary to show it is.
>
> I don't really care if a theory was deduced from experiments and then
> explained theoretically or vice versa. I know that in physics, both thing
> s were witnessed. And XP was eventually explained why it is working.
> Albeit, with such sciences that involve a lot of human interaction, it is
> hard to satisfy all researchers.

however, many researchers, even prominent ones, sometimes after
experimenting conclude they found a life principle and immideately
goes on to write an article about it. next thing you know, a company
was adviced to use it. next thing you know **get ready for exageration**
pal kal is falling. but you know what i mean.
its like if you have a million black horses in a continent and thus conclude
that using induction that the base 0..1000 are all black horses, and goes
on to prove that all horses are black. meanwhile in another continent
all the horses are white.

>
> Regards,
>
> 	Shlomi Fish
>
> ----------------------------------------------------------------------
> Shlomi Fish        [email-masked]
> Home Page:         http://t2.technion.ac.il/~shlomif/
>
> You are banished! You are banished! You are banished!
>
> Hey? I'm just kidding!
>
> To unsubscribe from this group, send an email to:
> [email-masked]
>
>
>
>
> Yahoo! Groups Links
>
> To visit your group on the web, go to:
>  http://groups.yahoo.com/group/hackers-il/
>
> To unsubscribe from this group, send an email to:
>  [email-masked]
>
> Your use of Yahoo! Groups is subject to:
>  http://docs.yahoo.com/info/terms/
>
>
>

---

From: Shlomi Fish <[email-masked]>
Date: Sun, 4 Jan 2004 18:35:29 +0200 (IST)

Hi Tzahi!

For next time, please prune sub-threads that you do not wish to comment
on. Otherwise these messages become very long.

> lagging behind but more are comming.

Well, to each his own. If I can find a good tool for creating and
manipulating diagrams, I may give it a try.

> I am just saying its no good for big projects like SAPing :) the navy force

Right. I'm not sure it was ever tried in these contexts, or that its
authors claim it is suitable for that.

Right.

> take the credit. again, the psychological thing.

So do I. However, I was trying to point at the fact that it is ultimately
you who is responsible for taking an idea forward. (yours or someone
else's)

> i.e. close in person development support for open source projects

Again, it's up to you. Note that Scoop (the kuro5hin back-end) boasts the
fact that it needs minimal maintainance. What it does is let users vote
for what will appear and get submitted.

> all the horses are white.

True.

Regards,

	Shlomi Fish

Hey? I'm just kidding!