2011-03-22 Email6 Development RestartΒΆ

Thanks to the supportive folks at QNX, I now have funding to do the full email6 development work. I’ve already discussed the general plan on the mailing list. The last two weeks have been a whirlwind of dealing with a flooded basement, followed by Pycon. This week I finally get to start on the project, and that makes me very happy.

So yesterday I dug in. The first project is going to be the policy framework. But before that there’s some preliminary work to be done.

The overall plan calls for doing the development of email6 pretty much in the Python main repository, with opportunities for some testing of major patches via feature branches before they land in the mainline. Once there are enough features implemented for it to be worth wider testing, I’ll start periodically rolling releases of a PyPI package based on a snapshot of the development. My intention is that what is in the CPython repo will be the intended-for-release code, and the package will be constructed via a little script that pulls the pieces together into a standalone package.

A little while ago the core developers had a discussion about the location of test files in the development tree. Most test files are located in the ‘test’ package, but there are a few located in subpackages of particular other packages: json, unittest, ctypes, importlib, sqlite3, tkinter...and email. Stub test_xxx files in the test package handle the chore of pulling in these separate test packages when tests are run, to give the illusion that all tests are in the ‘test’ package.

Most of these packages are this way because they were originally developed as standalone packages (and some are still maintained that way, for good or ill). Most of us using the tests would find it more convenient if all the tests actually lived in the test package (it makes grep and automated test discovery easier), but the package maintainers often find it easier to have the tests as a subpackage of the main package.

The conclusion of the discussion was to let it up to the individual maintainer where to keep the tests, but to encourage placement of tests in the main test package. The json tests were moved into test in 3.2.

As (current) maintainer of the email package, I would myself prefer them to be in the main test package. So that was my first project: move the tests. Barry Warsaw (the former maintainer) is a little bit freaked out by this move, but since I’m the one who is keeping the tests up to date, he’s fine with it.

Doing this move turned out to be an interesting experience, forcing me to learn quite a bit more about how mercurial works. My first attempt resulted in an incorrect file layout and some failing buildbots. With the help of our resident mercurial experts on #python-dev, I got it all sorted out in the end.

In the process of moving the tests, I also modernized the test support framework. The Python test suite has been in existence a long time, and has evolved over the years from no-test-suite, through a run-tests-and-check-the-output test suite, to a unittest-based set of tests wrapped by a custom test runner. In Python 3.2, unittest has seen a number of improvements via its current maintainer (Michael Foord). This allows for a considerable simplification in the amount of boilerplate code needed to hook tests in to the test runner. It is also now possible to use unittest directly to run tests, which allows nice things such as running an individual test method of a given test class.

All told, I deleted a net of 47 lines of boilerplate code during the move. Deleting unneeded code can be just as satisfying as adding new code.

Today I had planned to start on updating my existing implementation of the Policy Framework, but the reality of other commitments got in the way. Tomorrow, however, looks much more promising.