Subject: RE: [hackers-il] Rules for Open Source development (the serious v ersion)

From: Omer Musaev <[email-masked]>
Date: Sun, 20 Jan 2002 17:13:53 +0200

> -----Original Message-----
> From: Chen Shapira [mailto:[email-masked]]
> Sent: Sunday, January 20, 2002 5:04 PM
> To: [email-masked]'
> Subject: [hackers-il] Rules for Open Source development (the serious
> version)
> 
> 
[ snip snip ] 
> BTW. Does anyone know of a software project that has 
> professional QA team,
> with the usual practices (detailed test plans, regressions, 
> forcing the
> developers to stop adding features and fix a showstopper)?
> Do QA teams in open-source projects have enough political power over
> developers ("Head of QA said we can't release until we have these bugs
> fixed")?

XEmacs ( http://www.xemacs.org/Develop/jobs.html ).
Not really professional QA, but there is a head of QA, and he has 
a word.

> 
> 
> Thanks,
> Chen.
> 
> ------------------------ Yahoo! Groups Sponsor 
> ---------------------~-->
> Tiny Wireless Camera under $80!
> Order Now! FREE VCR Commander!
> Click Here - Only 1 Day Left!
> http://us.click.yahoo.com/WoOlbB/7.PDAA/ySSFAA/saFolB/TM
> --------------------------------------------------------------
> -------~->
> 
> To unsubscribe from this group, send an email to:
> [email-masked]
> 
>  
> 
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/ 
> 
>

---

From: Omer Musaev <[email-masked]>
Date: Sun, 20 Jan 2002 18:15:45 +0200

> Not really professional QA, but there is a head of QA, and he has 

We discussed the matter a bit.
Here comes a log.

chenshap: You mean "bug tracker" is XEMACS head of QA? It isn't the same
thing at all. QA should search for bugs, not just track them.
chenshap: Still its a start.
omussaev: Well, the interesting point is a statement that "bugs do not go"
omussaev: Another interesting case is clear separation of "bug tracker"
position from development team.
omussaev: Those points are not present in many projects.
omussaev: I should check Ximian for existence of QA teams.
chenshap: Thanks.
....
chenshap: OH. mozilla has QA.
chenshap: little help it is.
omussaev: yes.
chenshap: But I can't find their test-plan
omussaev: Mozilla has "super-reviewers".
omussaev: How interesting. McConnel says that reviews are one of most
efficient ways to ensure quality.
omussaev: This means, reviews give more bugs revealed than unit testing. 
omussaev: However, McConnell states that reviews alone are not enough.
omussaev: He tells that unit-tests and reviews should solve most problems.
omussaev: But the point is that Mozilla has both, as long as nightly builds
can be considered unit-tests.
omussaev: So, McConnel rules do not apply to Open Source. This is an
interesting twist.
omussaev: What do you say?
chenshap: I think McConnel is talking about very specific kind of reviews. 
chenshap: Reviews I had in Topaz weren't that effective.
chenshap: Besides, I don't believe him.
chenshap: Reviews rarely find sync bugs
chenshap: Unit tests only find old bugs
chenshap: How effective can this be?
chenshap: What about UI problems? Things misplaced on screen are usually
find only at QA. Especialy if the problem exists only in a specific
netscape.
omussaev: Reviews we have in Mercury are not really reviews, in McConnel
terminology, but a "stepthroughs".
omussaev: Which, IIRC, have lesser efficiency than "formal reviews".
chenshap: Are you sure mozilla reviews are MacConnel style?
chenshap: Best QA practice: Show code to VP.
chenshap: Demo to customer at a large meeting
omussaev: Unit tests can find new bugs too. If you are have a clear spec,
and an unit test written according to that spec, the unit test can be really
eye-opener.
chenshap: Omer, you sound like a new graduate. clear spec? where are you
living?
omussaev: :-)) yes. Running code before VP will reveal too many bugs for
date estimation to remain clear.
omussaev: Ha. We usually have one. The DTP - first there was a spec, and
then there was implementation, and the unit test was written by QA team. 
omussaev: However, you are right - this case is a wonderful exception, good
for optimism. 
omussaev: I do not really believe that I could afford writing all my code
that way.
chenshap: And in this ideal case, you feel that Unit-tests were enough?
chenshap: You didn't need any manual "bang on the keyboard" thingy?
omussaev: No. But I would feel more confident with my code. 
omussaev: I mean, no unit tests were not enough, and I would like to have
that chair-keyboard connector intact.
chenshap: Unit tests are intended to make a developer happy with his code.
especially during refactoring. Macconell is turning them into the ultimate
QA tool, and I disagree.
omussaev: No. He is not. He actually says that unit tests are quite often
not appropriate. Never a substitute for a good design, design reviews, code
reviews, clear specifications et cetera. They are good for conformance
tests, and for extreme conditions tests. No more.
chenshap: I believe you said that he said that reviews and unit tests are
enough?
omussaev: Something like that. He said that combining 2 or more techniques
give good results.
omussaev: as an example, code reviews ( formal ones ), combined with unit
tests.
omussaev: According to his survey, using unit tests only reveals 25% ofd
bugs in best case.
omussaev: While formal reviews go for 60-70% uin best case.
omussaev: 2 of those combined go for 80% and more.
chenshap: I'll stick with my belief that developers are simply not good at
finding bugs, its a mindset thing and while good practices help, they can't
be good enough for production.
omussaev: I agree. Developer must have an ability to view code in selective
manner. QA person must see it as a whole.
omussaev: Like in DIP - I see refactoring the whole bunch of those thousands
lines of code as very important step. And if I will push some asserting to
the /dev/null during refactoring, I will never know - it is not in my
priority list.
omussaev: However, for QA, the fact that DIP crashes on some not-so-wierd
input is a fact that they can not ignore. 
omussaev: I think that the best situtation is when QA is planning unit
tests. However this is not possible situation.