Test Driven Development is a development methodology that encourages you to write tests before writing the code that is being tested. Following it completely strictly means that you never make a functional change to the code (that is, adding a new feature or fixing a bug) until you have a failing test that implementing the feature or fixing the bug will cause to pass.
For those who are not used to writing tests, such a strict methodology can feel quite daunting, potentially unproductive and perhaps fun-spoiling. I can empathize with all three points of view, but at the same time it's worth trying almost everything at least once to see how it works out (though I'm still not so keen on trying a bungee jump).
In this article I'm going to look at the idea of automated testing more generally. I'll share a couple of my experiences of it, then I'll look at some thoughts for giving it a go in your own environment.
Why Automate Testing?
Testing involves working out whether something you have implemented behaves as expected. We all test software at some level. For example, if you are implementing an e-commerce site, you will have a shopping cart where people can add products. You will probably try adding some products to the basket, updating them, deleting them and so on, in the web interface, to see that it actually works before shipping it off to the client for them to play with.
As development continues, you may find that you have to do some changes that in turn affect the shopping cart. You may or may not be conscious that the changes will affect it; in the case where you are not, going and checking that the shopping cart still works will likely not be something you think of doing.
This is how bugs can creep in to systems without getting noticed. It means that you will get interrupted at some point in the future when the bug is discovered to go and fix it. It also means that you may well not know what change introduced it, making finding the problem and resolving it somewhat more difficult.
It is too costly to go and manually check every feature of the site after every change. If, on the other hand, you could issue one command or press one button and run a whole set of automated tests that will exercise many features of the site, you can afford to do testing after every change. Therefore, automated testing enables you to quickly detect bugs introduced through changes.
Whenever you add a new feature, you need to test it too. If it is complex, you will want to test many use cases. Writing an automated test means that you can see, at a glance, what use cases are and aren't being tested. Keeping track of that in your head, or trying to remember if you tested a certain corner case manually, is much harder than just being able to look and see.
Study 1: My Bytecode Translation Project
One of my areas of interest is Virtual Machines for high level languages, such as the JVM, the .Net CLR and Parrot. For my degree project I looked into the problem of virtual machine interoperability: how can you use a library written to run on one VM on top of another. The solution I implemented was bytecode translation: you take the program in one bytecode format and translate it to another.
It is of great importance that the translated code has the exact same behavior on the target VM as the original code has on the source VM. I quickly realized that as I was going to add more and more features to the translator, so it could translate more and more complex programs, I needed to be able to check that the latest change I had made did not break things that had worked previously. Therefore, right at the start I implemented automated testing.
It was pretty trivial to implement. I'd call the translator to translate the bytecode representing a class on the source VM, then write tests that made method calls on the translated class in the target VM. If the method calls returned the expected values, then the test passed.
This served me very well throughout the project. There were more than a couple of times when I made a change or introduced a new feature and the tests pointed out that I had broken something that had worked previously. Furthermore, looking at which tests had failed often gave a large hint as to where the bug was to be found, cutting the amount of time spent debugging.
With the advantages of tests becoming quickly apparently, I vouched to never, ever check a new feature into source control without writing a test for it. The test didn't have to pass; I can cope with failing tests if they point out real bugs that will be fixed later. However, it meant I did not check in bad code without it being apparent the next time I worked on the project. I followed this model, and it served me well.
As I went on, I found that if I wrote the tests first then it made me think through the use cases and code paths that were going to occur. Having got that running through my mind, I found it was often a easier to write a good implementation of the feature that didn't miss so many edge cases. And of course, before you write the code you are thinking in terms of what the code should ideally do. After you've written it, you're aware of its strengths and limitations, even if you aren't consciously missing out the tests that it has a good chance of failing.
I think this story is pretty suggestive of the "conversion" process that you go through as you embrace automated testing. Some people can read about it, see it's a good idea and just start doing it. For most people - myself included - I think the only way to see its value is to give it a go and find out.
Study 2: An Outsourced Web Project
I was managing an outsourced project on behalf of one of my clients. After putting the project out for bidding and getting various offers, I looked through the profiles of the freelancers that were behind them. There was quite a mixture in there, but one that caught my eye mentioned that he used automated testing. Figuring this would give a better chance of quality code, I went with this bid.
The code was indeed supplied with a test suite, which I was very pleased to see. However, actually running the program quickly showed up a variety of bugs. In this case, there were tests, but either they were not comprehensive enough, or covered the wrong things.
Would there have been more bugs if the test suite hadn't been written? Very possibly, and very likely. However, I mention this example because it highlights an important issue. You can only consider passing the test suite to be evidence that there aren't problems with the software if it provides a high level of coverage (that is, it exercises most or all of the code paths), and covers a wide range of input values - both the expected and the unexpected.
Be careful not to be lulled into a false sense of security by the fact that all the tests are passing. It doesn't mean there aren't any bugs; it just means there aren't any bugs in the areas of the code base that get tested, for the inputs you are testing it with.
So Was It Worth It?
One question that comes up with automated testing is whether the time we took in writing the tests, most of which may well have passed straight away because our implementation was good, actually saved us time overall? That is, would it have been cheaper (time wise, and thus cost wise) not to write tests, but just to debug the code when we try to use it and find it doesn't work?
I think that the answer is, even if it does take a bit more time to write the code and a good test suite, it's still well worth it.
First, if you're writing tests alongside writing implementation, you'll have the code base in your head. Any bugs that the tests highlighted can be fixed while the code base is fresh in your mind. If, on the other hand, you discover the bug a week, a month or a year down the line instead, chances are that you'll have moved on to something else, and will have to dig through the code, having to work out what it does before you can identify the problem and work out a fix.
Second, it's good for project and time management purposes. Knowing that you have implemented a chunk of a project
and that it actually works because you've got tests to show that it does is a good indicator of your progress. Having a bunch of code that you know compiles, but have no idea if it really works, is a much less reliable measure of your progress. This is especially true when developing libraries and not having any user interface that makes use of them available to test with yet.
Third, it's an investment for future maintenance programming work. At some point, chances are the code will need fixes, have new features added or have to be ported to be compiled under a different compiler or run on a different platform. Chances are that the maintenance programmer may well not be you, or it may be a five year older version of you. Having a test suite that gives at least some assurance that the code still works with the changes or after the porting will be a big help in this situation.
Jumping Into It Yourself
There are plenty of frameworks out there for automated testing. I can't look at all of them, so I'm just going to list a few for some popular languages and platforms, and then look at some general hints for writing good tests. Note that these are just some of the options, and a little searching will reveal more options than these for the language or platform of your choice.
Test Organization
The first thing to think about is test organization. In most frameworks, you can spread tests across multiple files (which you can in turn organize in some kind of directory structure), and each file can contain many tests. Exactly how you organize your tests will depend on what you are testing and the structure of your project.
One way that can work well for some object-oriented software is to structure your tests in the same way as your source tree, with a test file for each class. A more general principle is to give each feature (or set of related features) its own test file, putting tests for that feature inside that test file.
The reason for this kind of division is that it helps you to see exactly what is broken by looking at which test files contained failures. It also means that you can run the tests for that individual feature alone more easily, which is useful when you want to see if a fix you applied to try and make that test pass again worked. Of course, you need to do a full test run at some point to make sure it didn't break other things, but if it takes a few times to get right then just being able to run a single test file rather than the entire suite of tests can be a big time saving. It also helps you to work out more easily where the test for a particular feature are, if you want to review them.
Writing Tests
When writing an individual test, try to make it test a certain aspect of an individual feature. If it depends on other features, they should also have tests. You may want to order tests in such a way as to make sure that features depended on by later tests are tested first.
A test should either have a descriptive name that makes the purpose of the test clear or a comment stating that (or both). Just like code, don't assume that some maintenance programmer staring at your test in five years time will know just know what the intention is. Tests are code too.
Additionally, a test should try to leave the overall environment as it found it, whether it passes or fails. This can be decidedly tricky (or maybe even impossible) in some cases, when code that modifies a database or changes files is being tested. In these cases, perhaps consider supplying a test database or test set of files. The key criteria is, if you run the test through once, and then run it again without any changes to the code or the test, does it always give the same result?
What To Test
It's quite obvious that you should write tests that check that a feature behaves as expected when given sensible input values. However, if there is any way that data can reach it from the "outside world" without being sanitized, or if you are not sure whether that is the case, you should also test it with invalid input and make sure it handles the situation gracefully.
The other important kind of tests to write are boundary condition tests. For example, if you are testing a function or method that processes a list, check how it handles an empty list. If it is meant to take a range of values, check the values at either end of that range. If it is handling a date or time, check it with the first and last day of the year, leap years, midnight and so on. If it is working with a string, check how it handles if an empty string is passed.
Closing Thoughts
If you weren't doing any testing already, I hope this quick look at automated testing has inspired you to give it a go. If you do, why not write in your PH blog about your experiences of it, or leave a comment here? I'll certainly be interested to read anyone's accounts of this, as well as success and horror stories.
This article is far from the definitive guide to testing. If it were, there wouldn't be books hundreds of pages long on the subject. I hope that it has given you an appetite for the area, and you'll find plenty more material out there on the web and in books, some general and some specific to individual testing frameworks or testing certain types of application. Good luck!
