Well, after some discussion on cooker and, quite frankly, needing a resource myself to speed up security update testing (particularly looking for regressions), we’ve re-implemented “Testzilla”. Sorta.
Instead of integrating with bugzilla, we’ve used the mediawiki framework. The reasons for this many. The biggest reason is to make the “Testzilla” as accessible as possible to as many people as possible. It is also to lower the bar to usability… anyone can look at information on the wiki, but dealing with Testzilla can be trickier (perhaps). Also, Testzilla itself hasn’t been updated in many years, which means a lot of hacking to make it work. Other solutions were either too complicated (Testopia) or didn’t integrate with anything else we currently had (using mediawiki allows us to re-use our authentication from my.mandriva.com).
Anyways, the basic idea is to have a page per package (urpmi, eclipse, evolution, whatever) and on this page will be testcases — descriptions of ways to test programs in that package. Where possible, automated or semi-automated testcases should be written and these get committed to subversion and referenced from the wiki page (the downside here is we need a mechanism to get testcases from people without commit access in, but the upside is it is faster, provides versioning, and won’t bog down the wiki with (hopefully!) numerous testcases).
I’ve already implemented the main page, which can be found at wiki.mandriva.com/en/Testing, and from there you can get to the various testcases (there’s only one on there right now). There is a cookie-cutter template to be used to start new testcases pages, and there are some new macros that make integrating with subversion and bugzilla a little easier/nicer.
All in all, I think this has the potential to do really good things for Mandriva, and probably other distributions as well. With the ability for anyone to create testcase pages, testcases, send an email with an automated testcase attached to our new testcases_@_mandrivalinux_dot_org “exploder” to get committed to subversion, it should be extremely easy and straightforward for people to get involved.
This is something that quite a few people on the cooker list expressed an interest in, so I’m hoping this will nicely take off and become yet another useful resource for the Mandriva community.