Testing

Posted: 6 months ago (2010-01-11 21:19:38 UTC )

Tests often have significant dependencies between them, and those can cause all sorts of ugly problems in testing. That's not news, of course.

Test suites tend to have varying approaches to minimizing the problem, but usually boil down to "run the tests in a basically fixed order and let the programmer sort out the problems."

Unfortunately, that encourages all kinds of bad habits on the programmers' parts.

Languages, with a few exceptions (heya, Haskell!), lend themselves to ugly global state change, making it very hard to run many tests without testing things subtly different than you think you're testing -- two tests touch the database, and leave changes behind, while the other tests mess with the /tmp directory, and a fifth test does both.

Test frameworks should exist to run each test individually, and then run combinations of them, and to mark various tests as "troublemakers" (known to touch global state in a way not fully restored), individually or in cliques. Test suites need the equivalent information to the phrase "this test tends to fail when run along with tests under tests/functional/database".