Test-Driven Documentation: Turn Your Test Suite into an Executable Spec
Designing effective strategies for both testing and documenting a codebase is hard.
Beyond the challenge of writing a single passing test or a clearly communicated document, every local decision sets a precedent which affects the system as a whole. Each test and document communicates something about the entire strategy.
If everything were a unit test, would failures that only emerge end-to-end be missed? If documentation always used this level of detail, would the amount of content make understanding the system harder, not easier? None of these issues is obvious in isolation. It is only as these decisions accumulate—when the overall strategy has already emerged—that the second-order effects become visible and expensive to correct.
Enter Test-Driven Documentation!
Test-Driven Documentation: the use of tools to generate documentation artefacts from automated tests.
This is a paradigm that combines the testing and documentation systems to inform our strategies for both. It isn’t a specific implementation or technology, but a technique based on a mental model that helps us make decisions and scale our tests and documentation.
To illustrate this concept in concrete terms, an example implementation of Test-Driven Documentation might take the following Playwright test suite and generate a documentation website for the users directly from the test definitions with Docusaurus.
import { test, expect } from '../fixtures'; test('Create a project and add your first task', { tag: '@doc' }, async ({ page }) => { await test.step('Sign in', async () => { // ... }); await test.step('Create a new project', async () => { // ... }); await test.step('Add your first task', async () => { // ... }); });
This is a working example, for which the code is available here, with both a live application website and the full documentation website. At this point, it’s slightly more than a deployment of an end-to-end test report; however, a more complete feature set could be developed over time. This version could replace a corporate pre-release run-book and might be extended with a variety of potential features:
- Composable, non-linear, linked user journeys: multiple navigation arrows at the bottom of the web page, which lead you through a “choose your own adventure” for documentation. Just watched “How to Sign In”? See everything you can do next, or all the prerequisites you need to do before that!
- Users can subscribe to specific features to get a warning when something has changed and view a diff of user flows side-by-side to highlight the steps that have changed since the last release.
- Only show guides to the relevant users based on their in-app user roles and allow them to comment on or approve the preview of the current/future state of internal tools/ applications.
Here, the tests and documentation supplement each other.
As the documentation is autogenerated, the overall maintenance burden is reduced, and the output is constrained by passing tests, dramatically reducing the surface area for drift. Making the tests public and user-facing means that users/ product managers have a direct feedback loop with the developers. The documentation can only be updated by updating the tests, which means questions raised about the documentation lead to test maintenance and a more aligned understanding of user needs from developers.
As the tests now capture precisely what results and edge cases the users care about, we can verify the correct behaviour in a way that isn’t coupled to the structure of the code; in theory, we could completely refactor the underlying codebase without breaking a single test with the same level of confidence, reducing the cost of improving the codebase.
Test-Driven Documentation is not limited to end-to-end test suites. The produced documents should be genuinely useful to end users; if a different type of documentation is requested, it suggests that a different level of testing is also required, as this reflects the tangible result being called for by users. Developers may require more technical documentation, which may relate to integration/ unit tests. Diagrams generated from the codebase show a similar intent of keeping a single source of truth that remains up-to-date (despite tooling for generating these diagrams often not being quite there yet).
So, why does this work?
While it may initially seem that testing and documentation are unrelated, they are, in fact, two sides of the same coin–both describe the behaviour of a subset of the system. While documentation records the author’s understanding of the software behaviour, testing is the verification of that behaviour. This allows the two to sit hand in hand.
The key insight underlying the Test-Driven Documentation system is:
Tests verify without validating; documentation can be validated, but not verified.
Here, to verify means “to (automatically) check that the software’s behaviour matches the designed specification”, whereas to validate means “to confirm that the behaviour meets the needs of the user”.
Software testing, in this context, is the executable verification of a subset of system behaviour; however, tests are only a heuristic for what users actually care about–developers encode their own assumptions into their assertions, which can lead to green tests with software that doesn’t meet the users’ needs. The goal is to quickly and reliably prove that the software is working as expected to continue evolving the system in reaction to changing user needs.
Meanwhile, documentation, in this context, is a record of the author’s (unverified) understanding of a subset of system behaviour; it can be placed before a user to confirm (validate) how the system should behave, but there is no automatic way to prove (verify) it works as intended. Documentation is often disconnected from the reality of the software and, as such, fails to stay up-to-date as the system evolves. The goal is to accurately capture and efficiently communicate an understanding of significant behaviour to build a common understanding which allows us to make informed decisions about the software.
This dissonance between verification and validation is the fundamental challenge that must be addressed to build well-functioning, self-sustaining testing and documentation systems.
Test-Driven Documentation acknowledges this and attempts to reconcile the flaws of both systems by finding ways to couple them together, creating a tight feedback mechanism using the strengths of each to supplement their weaknesses.
Verification automatically ensures “correctness”.
Validation ensures the real-world value of the software.
Both together ensure a self-regulating system.
Test-Driven Documentation isn’t a silver bullet–simply an expression of the underlying limitations of tests and documentation, with an attempt to design tools that enable the systems to serve their intended purpose well for longer.
While the illustrated example uses Playwright and Docusaurus for testing and documentation, which are excellent and widely used tools, I leave it to the reader to explore new ways of turning automated tests into artefacts that people can actually rely on.