Random header image... Refresh for more!

This is not a test. I repeat, this is not a test.

This is not a test:

public void Test1()
    DataService target = new DataService(); // TODO: Initialize to an appropriate value
    DataObject data = null; // TODO: Initialize to an appropriate value
    Assert.Inconclusive("A method that does not return a value cannot be verified.");

It is, in fact, a complete waste of time.

I spent part of this week neck deep in unit tests that the development team has written for our latest project. At the start of this project, there was a big commitment made to have unit tests written for every component. “Test Driven Development” was tossed around in meetings, despite the fact that most of the people using it were unaware that they were using that phrase completely incorrectly.1 Anyway, I’d offered my services to help the devs develop unit tests, but, for the most part, I stayed out of the way and let them do their own thing.  This week, I decided to take a trip through what had been done.  Due to corporate confidentiality agreements, I cannot discuss what I found there, however, let’s just say it inspired this posting…2

For many developers, testing is beneath them.  They view testing as the refuge of the software engineer who couldn’t cut it as a real dev.  Testing is a simple, braindead task, requiring limited skills or intelligence.  Yet remarkably, developers who hold that view are also the ones who are invariably unable to write any kind of useful test.  Sometimes they’ll resist, claiming that “It’s not my job to write tests”.  When they reluctantly give in to the radical notion that it is, in fact, their job to write code that, you know, works, they rely on wizards or recorders to tell them what tests to write and how to write them, and frequently brag about the fact that they’ve spent all week writing unit tests, without regard to the quality of those tests. 

I have an annoucement to make for the benefit of those people:

The letters “SDE” in my title are not honorary.

Writing good test automation is every bit about software development as is, say, writing a middle-tier WCF service.  You have to think of it that way or you will fail miserably.  In normal software development, you typically have a fixed set of requirements that you base your code on.  In test automation development, you also code to a requirement:  The requirement to make sure that the software works.  That particular requirement is infinite, highly dependent on context, and generally requires actual thought to implement.3  And somehow, many developers writing tests completely lose sight of that requirement.

Back to what I started with.  We’re using Visual Studio’s Unit Tests.  One of the much touted features is the fully integrated support of unit testing in the IDE.  You can even right click on a method you’ve written and generate unit tests!  Except…  Those tests are COMPLETELY WORTHLESS.  It doesn’t actually generate a test, it creates a stub and asks you to fill out the inputs and outputs.  It fails on a basic technical level, because it produces nothing useful and typically it takes you longer to delete all the crap they’ve put in the method than it would to have simply written the test the right way from scratch in the first place.   However, more disturbingly, it fails on a philosophical level for many reasons.  Among them:

  • It enables and encourages laziness.  A few clicks and look at that, you’re a tester!  Now you don’t have to think about what you’re doing at all.  Isn’t that easy?
  • It encourages bad testing practices.  It will create a single method for a function, implying that you only have to do one thing to make sure that your code is good.  It makes it easy to produce a whole set of test stubs at once, but, oh crap, they’re all Asserting “Inconclusive” and causing the build to report errors, so better comment that out, now the build looks good and you’ve got 80 passing tests — SCORE!
  • It gives you idiotic advice, like:  “A method that does not return a value cannot be verified.”  List<T>.Clear(); doesn’t return a value, but I’m sure you can think of a way to verify that. Array.Sort(something); also has no return, and also has a clear verification.
  • It creates a worthless “ManualTests.mht” and “AuthoringTests.txt”, which are about as useful as the MFC “Readme.txt” from VS6 and should be deleted just as fast.

And there’s more, but I’ll save it for some other time, because I think you’re getting my point.  Unfortunately, because it’s Visual Studio that’s doing it, people seem to think it’s automatically the right way of doing it, and don’t bother to think about it long enough to realize that they’re doing it wrong.  Therefore, I feel it is important to provide testing advice for developers who are attempting to write unit tests.  I think I sent most of these tips to my team at the start of our project, which means that they’re still in mint, unused condition for you!  In no particular order, but numbered for your reference:

  1. Unit tests are supposed to be self-contained.  They’re not supposed to run against external DBs, they’re not supposed to hit external services.  Use your DLL directly, don’t bother going through the service that exposes your DLL.  If you can’t check it in, you can’t use it.  Remember, the purpose of a unit test is to make sure your code works, not to make sure that the database server is configured correctly.  Read up on mock data providers to see how to design your code so that you can test it without talking to a DB or external service.
  2. Automation code is the same as regular code.  You can use helper methods and classes.  You can include libraries.  If you’re doing the same thing in fifteen tests, altering only a variable or two, then do the same thing you’d do in your regular code:  Write a method that does the work and pass those variables in as parameters.  It’s the same language you normally program in, and I’m assuming you already know how to use it.
  3. Always give yourself enough information in failure messages to find out why the test failed.  xUnit4 Assert methods typically have an optional “string message” parameter so you can put a note about what you’re checking.  As far as I’m concerned, that should not be an optional parameter.  Always use it.  If you’re catching an exception, print out the message AND the stack trace.  Add status writelines if you want.  Remember, you’re probably only going to look at the details of a test case when that test case is failing or acting strange, and when the test is failing or acting strange, you want to know why.  The more details and information, the better.
  4. The default status of a test should always be failure, whenever possible.5  Any time anything goes awry, it should fail.  If you have an if-else chain or a switch that’s supposed to cover every possible case, you should have the final else or the default of the switch make the test fail when you get an impossible case.  Basically, treat the code as guilty until proven innocent.
  5. Don’t forget the easy pickings!  Can you pass nulls?  Negatives?  Boundaries?  If your method has preconditions, try inputs that fail to meet those preconditions.
  6. You should generally have more than one test per method.  If your code has an if statement in it, then that’s a minimum of two tests right there.
  7. You should generally have more than one Assert per test.  For instance, if you have a function that fills a data object and returns it to you, then you should assert that the object isn’t null and you should assert that every field in the object is the value you expect it to be.  In some cases, it’s also prudent to make sure that it didn’t do anything you weren’t expecting it to do.  In the case of a delete method, for example, it’s often a good idea to make sure the thing you wanted to get deleted was deleted AND that the method didn’t go on a killing spree and delete everything else while it was at it.
  8. You should pretty much always end up with more code in your tests than in the code being tested, or you’re doing it wrong.
  9. You should pretty much always end up with more ideas for test cases than you can ever write, or you’re doing it wrong.
  10. Don’t write a bunch of unit test stubs for things you’d like to test.  It’s worthless.  Instead of test stubs, use comments to remember what tests you’d like to write.  In normal programming, you write a stub because you want to write some other piece of code first that will need a method, but you don’t want to actually write the function just yet.  That’s not the way it is in testing.  Nothing is going to call a test except the test harness.  A stubbed out test is not only a waste of time and space, it’s also dangerous.  If you stub out a bunch of tests that you’d like to write, something is going to happen that’s going to prevent you from getting to them right away, and you’re going to end up forgetting that you’ve left all the stubs there.  Then, sometime later, you’re going to review the tests that are being run, and you’ll see 20 test cases covering method X, and from the names of the test cases, you’ll assume that method X is adequately covered.  If, for some inexplicable reason, you feel you absolutely must have test stubs, then you MUST make them explicitly fail.  That way, you’ll know that they’re there.  Also, you’ll be annoyed by the large number of failing tests, and be less likely to put stubs in next time.  It’s better to have no test at all than a bad test.  At least you know where you stand when you have no tests.  If you have bad tests, they’re lying to you and making you feel good about the state of things.
  11. Use the most specific Assert available for what you’re checking.  The more specific the Assert is, the more specific information you’ll get about when when wrong when it fails.  And if there isn’t an Assert for something that you’re checking frequently, then I recommend writing one.  They’re not magic functions or anything.  They’re typically an if statement that’ll throw an exception with a details message in certain circumstances.
  12. Avoid using try/catch blocks in your test code.  Yes, I know that sounds strange, and like it’s a bad coding practice, but you’re doing it for a reason.  First, Asserts communicate to the harness using exceptions, so if you’re catching exceptions, you risk swallowing the failure exception from an Assert.  Second, if your test is throwing an exception, you generally want to know about it and want it to fail the test.  xUnit frameworks consider any exception to be a failure, unless you’ve marked the test with an attribute telling the framework that you expect a certain type of exception to be thrown.  Now, try/finallys, on the other hand, are perfectly acceptable.  Just be careful with catches.
  13. Always make sure you’re actually testing the right code.  Don’t worry about testing the third party framework, don’t test the external class library, and don’t copy large sections of the code from your project into the test project for convenience.
  14. Don’t rely on the code you’re testing to test the code you’re testing.  Don’t call your method to make sure that your method is correct.  If your method is wrong, there’s a good chance that it will be wrong the same way both times.  Verify the code using independent means when possible.
  15. If you have an automated build setup, then integrate your unit tests such that a unit test failure causes a build failure.  Every last unit test you have should be passing at all times.  (And if you don’t have an automated build, then you need to set one up.  Seriously.)
  16. If a unit test fails, fix the code or fix the test.  You should consider a failing unit test to be the same level of seriousness as a compliation failure.  Do not ignore it and do not comment it out to make it work for now.  A unit test failure means that your code is broken.  If the test is obsolete, then delete it.
  17. Name your tests something meaningful.  Don’t call them “LoginTest1” to “LoginTestN“.  Name them something useful, like “Login_NullUsername” or “Login_ValidUsernameAndPassword” or “VerifiesThatLoginDoesNotWorkWhenUserIsNotFoundInDataStore”.  Anyone should be able to look at a test name and have some idea about what it is supposed to be testing.
  18. Test your tests.  Make sure that they’re doing what you expect them to be doing, and looking for what you expect them to be looking for.  Sometimes I’ll forcibly alter an important value, just to make sure that the test will fail.  You can even step through your test cases in a debugger and watch what’s going on, just like normal code!  (Because it is normal code!)
  19. Your test cases should never depend on one another.  Test A should never set up something for Test B, because you’re not guaranteed that Test A will run before Test B, in fact, you have no guarantee that Test A will be run at all.  xUnit frameworks typically have some sort of ClassInitialize and ClassTearDown and TestInitialize and TestTearDown methods that you can use to set up and clean up your test cases, if needed.  Read the documentation on these for your framework to be clear on exactly when these will be called.

And finally, the most important reason to be diligent about writing good unit tests:

  • If you write good unit tests, then you’re more likely to keep those annoying testers away from you, because you’re less likely to be giving them code that doesn’t work.
  1. I’m sure I’ll have something to say about TDD at some point. []
  2. To be fair, one group had their act together, producing about 480 out of the 500 or so unit tests that were there, and I didn’t see any glaring problems.  So, they’re either doing it right, or they’ve tricked me into thinking that they’re doing it right, either way, they deserve credit. []
  3. I know when a developer is starting to understand the true nature of unit testing when they cry out “Testing is hard” in anguish. []
  4. nUnit, JUnit, MSTest, MBUnit, etc. []
  5. Unfortunately, the xUnit frameworks are completely backwards in this regard and will pass any test if there isn’t an explicit failure.  This is wrong, and in my view, broken.  In the obvious case, a completely empty test method will count as a pass, even though it doesn’t do a damned thing.  In a more insidious case, an unintended code path could cause a test to pass, even though there is a bug in the system.  For instance, I recently wrote a test to verify that a method would throw an exception with a property set to a certain value.  I put the method call in a try block, my Assert on the property in the catch, and called it good.  A few days later, I realized that my test would pass if the method I was testing didn’t throw an exception at all, which was obviously incorrect. []


There are no comments yet...

Kick things off by filling out the form below.

Leave a Comment