Executable Documentation

Yet another reaction blog post, although they seem to get me the most traffic :) This time I’m posting in reaction to “Software Engineers should keep lap notebooks”. The general premise of the article, if I’ve got it right, is that software engineers should keep notes while developing and fixing software. While good in theory, this falls victim to the biggest problem in software documentation – rot.

One of the things that came out of the agile movement when it first reared its head was the theory of executable documetation. The general idea is that the documentation you write can also be interpreted and executed by some computer process to check for validity. By doing this, you can essentially prove that your documentation is up to date with your application (or vice-versa), thereby pointing out when something needs to be updated and preventing (or at least notifying of) documentation rot.

Tests as Documentation

This has manifested itself in several forms, mostly in some sort of test context. Unit tests are great forms of code-level documentation and can be a great resource of training and documentation for new team members or developers who are unfamiliar with a particular part of the project. Integration tests work well as “feature documentation” – when I click this button on the UI, it should save this record in the database. Still technically targeted at a developer-level, but much more high up compared to unit tests. And finally, “behaviour tests”, where a non-developer writes requirements in a certain format, and they’re interpreted and executed against the application (usually in conjunction with an integration test of sorts).

There’s a lot of tooling built around this concept. Test runners can be executed by any developer on the project to ensure the current “documentation” is up to date and accurate any time they make a change or add new functionality. Code-coverage tests can show if the feature or code you’re working on has been documented yet or not – if not, you have every opportunity to add it yourself, as the author of the code. And BDD frameworks like Cucumber provide wiki-like tools for non-developer types to create executable specifications easily.

Document Rot

If we look at the proposed solution from the original article, having a notebook, it’s easy to see how these notes can become out of date. I can agree that writing things down and keeping notes while you develop is good for personal reasons – I think I read somewhere that you remember 70% of what you write down. The thing I don’t agree with is using those notes for future reference or collaboration. You can’t have any certainty that the documentation is up to date at all, especially if you’re working with other team members. The notes you took down when you wrote that function 2 weeks ago could be entirely out of date. And if you’re relying on them as a start point to make changes you’re probably going to set yourself back more than you help yourself.

Not to mention, if you do work with other team members, I’d hardly expect them to flip through your personal notebook for access to your documentation, much less update it for you. Someone proposed in the comments that using a wiki would be a viable strategy – while this would definitely be better for being able to search for and update documentation, there’s still the risk of rot as having an up-to-date document isn’t verified or checked by anything. Especially in a new project or feature, when code and architecture is changing on a regular basis, having manually-updated documentation will probably be pretty useless simply because there’s no maintainable way of ensuring its up to date.

While the original thought of keeping a personal lab book handy while coding is a good idea, I don’t think it addresses the needs of a collaborative software development environment. Taking an “executable document” approach has a much better chance of being relevant and useful in the future, and ensures that all team members can contribute and maintain the documentation.