James Shore is in XP immersion coach. He spends months with each team which lets him get well beyond the basics. Jim believes strongly doing things right. He isn’t willing to half-ass, and he isn’t willing to work with a client who wants to half-ass.

Jim’s background includes both technical and people practices. When he coaches an agile transition, he brings in a full suite of practices and values.

I asked Jim the same question that I asked everyone else:

What are the characteristics of a good test suite?

James replied and then expanded his answer to a full blog entry. This page shows just his initial reply.

James’s Answer

A good automated test suite runs fast, provides a safety net for refactoring, documents the intent of the code, and alerts the team to regressions. It doesn’t prevent bugs on its own, but if bugs are escaping the team, that’s a sign that there’s a flaw in the team’s development approach–which includes the team’s approach to design, technical debt, requirements, and overall defect prevention as well as its approach to automated tests.

The test suite needs to run fast because a slow test suite leads to broken builds. When the test suite is too slow, people don’t run all the tests before integrating, and integration failures aren’t always fixed right away because the person or pair who made the mistake isn’t always available to fix it. This often compounds so that one build failure hides a second, and you end up with the team letting the build remain broken for hours or even days. This violates a fundamental principle of continuous integration, which is that the code in the repository always builds and passes its tests.

Further reading:

• Continuous Integration is an Attitude, Not a Tool
• The Art of Agile Development: Ten-Minute Build
• The Art of Agile Development: Continuous Integration

The test suite needs to provide a safety net for refactoring so that the team is able to improve the design and pay down technical debt. Without refactoring, technical debt steadily increases, leading to degraded design quality, which leads to defects.

Further Reading:

• The Art of Agile Development: Refactoring
• The Art of Agile Development: Simple Design
• The Art of Agile Development: Incremental Design and Architecture

The test suite also acts as living documentation. This is necessary because comments and requirements documents go out of date. Tests can document both programmer intent (at the class and method level) and business intent (at the business rules level). Although people often write end-to-end tests to document business intent, this isn’t necessary; it’s actually more effective to write “customer unit tests” that focus on specific business rules.

Further Reading:

• The Art of Agile Development: Documentation
• mumble test business rules mumble

The test suite needs to be comprehensive enough to alert the team to regressions. This is important because manual regression testing is too slow and expensive to do every iteration, and this is a burden that steadily increases as the application gets bigger. I typically want to see unit tests for every class and method, focused integration tests for every interaction with outside processes, files, and systems, and a small number of end-to-end tests for smoke testing. (This assumes your design doesn’t require end-to-end tests to ensure everything hangs together, which is also necessary to make your build run fast.)

Further Reading:

• The Art of Agile Development: Test-Driven Development
• Let’s Play: Test-Driven Development video series

Finally, an automated test suite is just part of an overall strategy of defect prevention. It’s an important part, but automated tests on their own aren’t enough to prevent defects. You also need to focus on refactoring and technical debt control, close interaction with business experts and other requirements donors, and finding and fixing gaps in the team’s thought process.

Further Reading:

• The Art of Agile Development: No Bugs
• Alternatives to Acceptance Testing
• The Problems with Acceptance Testing

---

Love (or hate) that answer? Want to see other answers?