zephyrfalcon.org :: labs :: The Testipy framework

Testipy is a simple, lightweight testing framework, that I wrote for personal use. There is no reason for you to use it. emoticon:smile Under normal circumstances, you'll want to use PyUnit instead.

Why did I write it, then? ... For my work, I wrote and maintain a large(ish) body of code. Currently, there are around 450 tests for this, using unittest. Over time, it became painfully clear that I missed two features:

1. Methods (non-tests) that run before and after the whole test case. Currently I'm using an ugly hack, that abuses the fact that tests are run alphabetically. So test_000 will be run first, and I use this to create temporary directories, etc. test_zzz will be run last and is used to clean up these directories. Not only is this ugly, but it has several drawbacks; for example, if an error occurs in test_000, all the other tests are run anyway, probably causing errors because things were not set up properly.

2. Methods that run before and after a specific test. I suppose I could use a try..finally construct, but this isn't ideal.

The setUp() and tearDown() methods don't cut it, because they are run before and after *every* test, so they don't quite do what I want. Setting up things in the test case's __init__ isn't good either, because this isn't always executed at the right time (for example, in a test suite with multiple test cases).

Enter Testipy. This is a very minimal framework, it has only ~250 lines of code, and it's not complete or perfect, but it does what I want. Here is what it does:

There's a TestCase class, much like unittest's. You stick test methods in it:

class MyTests(testipy.TestCase):
    def TestThis(self):
        ...blah...
    def TestThat(self):
        ...

All methods with names starting with 'Test' are considered to be tests; anything else is "ignored".

The actual tests are done using various methods, much like unittest's. Their usage is obvious. Here's an example:

def TestSomething(self):
    x, y, z = getsomevalues()
    self.AssertEquals(x, y)
    self.Assert(x == 2)
    self.AssertNotEqual(x, z)
    self.Fail("too bad!")

You run a TestCase using the Run() method. This executes all tests, in alphabetical order.

t = MyTests()
t.Run()
# or: MyTests().Run()

You can override the Init() method to add arbitary attributes to the instance. No need to override __init__ and call the parent constructor with obscure parameters.

Before the tests in a TestCase are called, the BeforeTestCase() and AfterTestCase() methods are called. These methods are optional. You can use them to set things up or do cleanups. So, the order is:

- BeforeTestCase()
- all the tests
- AfterTestCase()

Before any specific test, a similar procedure happens. Let's say we have a method TestFoo. If there is a method BeforeTestFoo, then this is called before TestFoo is executed. If it's not found, but there is a method BeforeTest, then that is called. Both are optional, so neither of BeforeTestFoo or BeforeTest needs to be present.

Similarly, AfterTestFoo or AfterTest is called afterwards. So, the order is:

- BeforeTestFoo; or, if not present; BeforeTest; or, if not present, nothing
- TestFoo
- AfterTestFoo; or, if not present, AfterTest; or, if not present, nothing

Note that if BeforeTestCase fails, we exit with an error; no tests are run. Similarly, if the "before" method fails, the actual test isn't run.

BeforeTest and AfterTest are similar to unittest's setUp and tearDown. The other "special" methods don't have an equivalent in unittest, as far as I can tell.

There's also a TestSuite class. You feed it testcases and run it:

suite = testipy.TestSuite()
suite.Add(testcase1)
suite.Add(testcase2)
suite.Run()

Or, you can use the Gather method to scan files for TestCases:

suite.Gather(".", "test_*.py")

(this looks in the current directory for files matching the pattern test_*.py, scans them for TestCase subclasses, and adds them to the suite.)

Note: The Run() methods (currently) have two optional parameters, print_summary and print_report. The "report" prints a traceback in case of errors (after all the tests are done). The "summary" just lists the names of the tests, and whether they passed (OK), failed (FAIL) or had errors other than failed assertions (ERROR). Sample output:

TestTestipy.Test001 ... OK
TestTestipy.Test002 ... FAIL
TestTestipy.Test003 ... ERROR
TestTestipy.Test004 ... OK
TestTestipy.Test005 ... OK

[tracebacks snipped]

Tests run: 5 (0.31s total, 0.31s in tests)
Passed:    3
Failed:    1
Errors:    1

That's mostly it. It may not be feature-rich, but it's easy to use.

If you happen to know any useful enhancements that do not compromise its simplicity, I'd like to hear about them.