Dale Scheetz <dwarf@polaris.net> writes: > Testing is not closed! I actively recruited both from developers and users > for members of the testing team and ended up with what we got. Oops. I mostly meant that it didn't involve everyone. Poor choice of words. > My problem with your proposal is that it assumes that you (retorical you, > meaning someone) can get all of the developers on board with all the > controles you would need to manage verification of tests. I don't think the "organized developer testing" has to be too controlled. But I think we could probably find and prioritize a huge bunch of bugs if we could organize things so that every package is looked at. Granted, the developers are always doing this - but I believe there are still plenty of bugs that go unnoticed, or unreported. I guess I want to use the developers to decide which packages pass/fail our criteria for going into stable - which is too big of a job for a small testing job. > Yes, we need to encourage all developers to do extensive testing of their > individual packages, but the reason for centralized testing is that none > of the developers have the same machine. Last time, we all built base > systems and then installed all of "standard". This gave us identical test > benches for testing X and the other "ancillary" programs that we were able > to focus on. Expecting developers to do this is probably just as hard as > holding them to stable after a release. (which causes untold problems with > post-release fixes) It is my intention to provide every tester for 2.0 > with a 1.3.1 CD. This will allow us all to do our "upgrade" testing on > identical systems. While this doesn't deal with all the problems we are > certain to see from systems that are "partially" hamm, it will be far more > informative about integration problems within hamm than the scatershot > testing by developers. I don't think the "scattershot testing" (good description) is going to be comprehensive at all - but I think it would definitely lighten the testing load on the release team. It won't catch all the bugs, but it would catch the really obvious ones, which unstable is full of. I'd estimate that 20-30% of the packages in unstable aren't ready to go into a stable release. In the past, we released them anyways. We shouldn't this time around. > Basically the QA team did very little of the testing. I must have been confused, I thought the QA team was the testing team. > > The Release Team's job should be easy, if we have a round of developer testing > > first, which uses the bug system to identify broken packages. The release > > team can monitor this, and prioritize what needs to be fixed. > I think that this is the fundamental part where your idea breaks down. > > We are supposedly doing "developer testing" every time a package gets > released. The only thing that the testing team did last release that was > complete was "integration testing". If all the developers could do > complete regression testing on their packages individually, the your > statement about the easy job the release team has would be correct. I don't think we can rely on each individual maintainer to do a good job of testing. Some will, some won't, and some are on holidays... The "integration testing" stuff is tricky, since that has to mostly come at the end of the release process. That has to be done by a small team, like what you organized last release, for sure. > > There is a lot of work in organizing the developer testing - I'll volunteer > > for that job, if this plan goes ahead. > > > I have absolutely no objection to your doing this! > > In fact I encourage you to do so. You will then get some first hand > experience with the difficulties I have been (somewhat poorly) trying to > communicate to you, and we would get some kind of improvement in the > testing process. > > YES! Go for it! Ok, I will, boss. :-) > > We should write a "Release/Testing Strategy Manual" to codify everything so > > there aren't any misunderstandings this time around. > While this is not a bad idea (outside of who writes the damn thing) it > doesn't come close to eliminating misunderstandings (see current DFSG > threads ;-) > > While mistakes were made in the last release, most of the ones I can think > of had to do with mistakes in judgement rather than misunderstanding of > proceedure. I found that most of the misunderstandings I saw were based on differing interpretations of testing terminology. That's why I think a manual might help. > > So, when are we going to release 2.0, if ever? We should announce a date, > > and stick to it. Frankly, I think we've been in good shape since August > > to make a release. > > I see no reason that we can't make code freeze by the first of the year > and release by 1 Feb. This seems to fit your schedule, if not in the > details. With Thanksgiving comming up and Christmas close behind I see no > way to organize anything at all tight before then. > Personally I don't see what a quick release turnaround has to do with the > development model, but I'm known for being dense at times ;-) Less time testing/releasing (no fun) = more time developing (fun) Maybe we should aim for Feb. 15? Cheers, - Jim
Attachment:
pgpIz8LBz40fF.pgp
Description: PGP signature