Testing - an afterthought for opensource?

When I was working at Thiokol I found an article online about the development process for the software that runs on the shuttle. As you can imagine they have a very careful development process. I'll see if I can find it.

I found your blog entry by following a link from my referrer log. I think there's plenty of open-source tools for testing, but testing itself is seen as a second-hand activity by most non-agile shops. Testing does get first class citizenship in agile (i.e. XP, Scrum, etc.) environments, although there's a big emphasis on unit tests there and less on system/integration/load/performance/stress tests.

As for your question about performance/stress/load tests being repeated by end users, I very much doubt this is the case. On most projects, you need people with a deep understanding of system architecture/network infrastructure/database optimizations to perform these types of testing. So I'm afraid they're simply not done.

Grig- Thanks for the comments. I guess I should have clarified what I was saying by "end users". My bad. I meant your ISV/fortune 1000 type companies using open source, who often times will have a "performance engineer" in house.

I am pretty sure your average 16 year old linux 3l33t user could care less about testing.

A few years ago I decided I wanted to contribute to development of free Common Lisp implementations (there are a half dozen or so of these.) Looking at what they needed, I decided testing was the biggest lack.

I ended up putting together a set of black box tests driven by the ANSI CL specification. The suite now has over 20,000 tests in it. I also wrote a random program generator to find compiler bugs. This tester has generated hundreds of millions of random (but legal) programs and has found bugs in every implementation on which it's been tried, free or commercial. This test suite is in the GCL CVS tree.

I think the presence of a high power test suite has noticably changed the dynamics of development of the free lisps. Standards compliance is up and the developers are more willing to make daring, dangerous changes knowing that testing is more likely to catch regressions.

I encourage anyone wanting to get involved with a free software project to consider testing. High quality testing is a very creative activity, and can be just as personally fulfilling as development of the software product itself.

> developers are more willing to make daring, dangerous changes knowing that testing is more likely to catch regressions.

I abolutely agree. A good suite of unit tests really feels like a great safety net if you want to do big refactorings.

I abolutely agree.

The Perl community, and the CPAN in particular, has a very strong culture of testing. Older CPAN modules sometimes tend to have less tests, but the newer modules (and some heavily used modules like DBI, etc) tend to have very good tests suites. There is even a CPAN Testers group which will run these automated test suites on multiple platforms and report back to the module authors on their results. I would suspect that a lot of this comes down to the fact that Perl has an excellent assortment of testing tools/modules available and the community itself places a high value on using these tools.