Python Testing Tip #1

Why do we even test our software?

If you’ve read anything about software testing, I’m sure you’ve stumbled across the idea of “one assertion per test”. Sounds like a simple idea, right? Thinking about it raises quite some concerns:

  • Do I need hundreds of tests to verify a single method?
  • What if my object has multiple properties? Should I write a test for each of them?
  • I need to write less code if I do multiple assertions in one test. So why would I want to have a single assertion per test?

Before we address these concerns, we must ask ourselves why we would want automated tests for our software. I want to be part of a successful business. Therefore, I want to react quickly to changes in business requirements without breaking existing functionality. To do this, I need to be able to release changes as fast as possible and ensure that introduced changes don’t cause defects. Manual testing takes time and is difficult to repeat reliably. On the other hand, automated tests can be fast and repeatable. Consequently, deciding which one I want to invest in is easy.

Can we remove manual testing entirely? Usually, we can’t. Anyhow, we can reduce it to a minimum by having valuable automated tests.

Single assertion per test – rule or guideline?

Let’s focus on the following two aspects of high-quality tests:

  • They should be readable
  • They should help us to localize the defect quickly

So how does “One assertion per test” help us to achieve these two goals? Let’s say we have a simple API like this:

Copy

Tests with multiple asserted behaviors might look like this:

Copy

Tests with a single asserted behavior per test might look like this:

Copy

The first thing you might notice is that none of them has a single `assert` statement. The concept of “one assertion per test” is more of a guideline than a rule. Since tests are used to verify the behavior of our software, we can paraphrase it from “One assertion per test” to “One asserted behavior per test”.

Blindly following “one assertion per test” would bring us to a state where status is asserted in one test and response structure in another, etc. That’s not what we want.

With this in mind, let’s analyze the examples. In the first one, we have two asserted behaviors:

  • When the user exists, we should get user details with status code 200
  • When a user doesn’t exist, we should get a 404 status code with a proper error message

In the second example, we separated the two. Consequently, we comply with our new guideline. So why is the second example better? Suppose the following scenario – someone removes the handling of non-existing user case from the API.

Copy

If we run the tests from the first example, we will get the following result:

Copy

If we run the tests from the second example, we will get the following result:

Copy

Single asserted behaviour per test

As we can see, the “single asserted behaviour per test” approach makes it easier to localize the error. There are two tests, and only one of them fails. Just by looking at its name, test_user_details_not_found_when_user_not_exists, we know something is wrong when the user doesn’t exist. It’s still all good for a happy path scenario. Consequently, we know much better where to look for the defect – in the path when a user doesn’t exist. While this might not be as obvious in this simple example, it’s much more apparent in complex real-life scenarios. The same goes for readability. The first example mixes act and assert steps in the test. You need to read the whole thing to understand what’s going on. So, if you want to change the error message, you need to investigate the test to find the part that tests the “user does not exist” scenario. With the second example, you can look at the test name and know where to look for the assertion. You can update the message there without digging into the test. The more complex the endpoint, the more obvious the difference is. With the “single asserted behavior per test” approach, all your tests for a single endpoint will have roughly the same complexity/readability. If the endpoint gets more complex, you will just need to add more of them. In the “multiple asserted behaviours per test” approach, the complexity/readability of the tests will grow with the complexity of the endpoint.

Tests as executable documentation

When you combine “single asserted behavior per test” with meaningful test names, you can look at tests as executable documentation. You can just go through tests to figure out what the system does. Reading their name tells you what are the expected behaviors. And since they are tests, you can execute them. If tests are testing behaviors, they can’t go out of date (as opposed to comments or static text docs). If you want to alert a behavior, you need to alter tests. This includes updating/removing existing ones and adding new ones.

That doesn’t mean documentation as such is useless. This just means that you can make it even better by having valuable automated tests with a single asserted behavior per test.

Conclusion

As we’ve seen, it’s not about the number of assert statements but rather about the number of asserted behaviors. “One asserted behavior per test” helps with the defect’s localization and test readability. That’s one of the steps towards high-quality tests.

You can find code examples in this Github repository.

Subscribe To Python Testing Tips

Get Python Testing Tips to Your Inbox

Subscribe To Python Testing Tips

Get Python Testing Tips to Your Inbox

Share This Story, Choose Your Platform!