Several times since last year I heard a complain about the lack or insufficience of QA in small FOSS projects. Judge yourself the example of libwpd.
Recently we had a crasher regression discovered just some days after a release and your servant was not happy at all about it. To prevent this kind of situation to arrise again, I approached sum1 of AbiWord fame and he agreed to QA the cvs head tree after a major commit and before a release. Here is his methodology:
- First, he crawls the web and, using a nice firefox add-on called DownThemAll!, he downloads all WordPerfect documents he finds.
- Like that, he created himself a stock of about 40k real life documents in different WordPerfect formats. He throws 10k - 15k of these documents on wpd2raw, a nice small utility that is distributed a s part of libwpd-tools package and is used by libwpd developers in their regression test suite. This QA run takes about an hour.
- The documents that crash the tool and/or throw an exception are filed as bugs in AbiWord bugzilla, so that they can be dealt with.
A simple plan that has a lot of benefits. First of all, the library is stress-tested against a considerable quantity of real-life documents, so that the developers can deal with any crasher situation found. Second, this QA testing helps to find quite complicated documents in rare WordPerfect formats and renders much easier to add new features in the list of converted features; especially where the documentation is a bit fuzzy or unavailable. Third, the documents that produced crashes that were really hard to fix / not trivial to see take place in the above-mentioned regression test suite so that they do not bite anymore.
Just a little question: How much stress-tested are the proprietary, binary, stale and unmaintained WordPerfect filters distributed with StarOffice 8?
