Careful with negative assertions

Sunday 4 November 2018

A cautionary tale about testing that things are unequal...

We had a test that was kind of like this:

def test_thing():
    data = "I am the data"
    self.assertNotEqual(
        modify_another_way(change_the_thing(data)),
        data
    )

But someone refactored the test oh-so-slightly, like this:

def test_thing():
    data = "I am the data"
    modified = modify_another_way(change_the_thing(data)),
    self.assertNotEqual(
        modified,
        data
    )

Now the test isn’t testing what it should be testing, and will pass even if change_the_thing and modify_another_way both return their argument unchanged. (I’ll explain why below.)

Negative tests (asserting that two things are unequal) is really tricky, because there are infinite things unequal to your value. Your assertion could pass because you accidentally have a different one of those unequal values than you thought.

Better would be to know what unequal value you are producing, and test that you have produced that value, with an equality assertion. Then if something unexpectedly shifts out from under you, you will find out.

Why the test was broken: the refactorer left the trailing comma on the “modified =” line, so “modified” is a 1-element tuple. The comparison is now between a tuple and a string, which are always unequal, even if the first element of the tuple is the same as the string.

Comments

[gravatar]
Kevin Reid 11:22 PM on 4 Nov 2018

Other ways to have an infinite space of unintended passes:

Asserting that an exception is thrown, without checking that the exception is the kind you expected.

Making an assertion within a callback, which is never called, or which is called by something that catches all exceptions including the one the test framework uses for "assertion failed".

[gravatar]
jonathan hartley 6:09 PM on 5 Nov 2018

This is why the TDD cycle specifies "red" then "green" : run the test and see it fail first, as a sanity check of your test, even if you have to add a throwaway line to your code to make that happen. Once you've seen the test fail, then replace the throwaway line with the real implementation to see the test pass.

Once you start doing this by habit, you will be *amazed* at how frequently you discover you accidentally create tests that pass *all the time*, no matter what the code-under-test says. Thus proving that this habit is vital.

[gravatar]
João Farias 5:56 AM on 6 Nov 2018

Mutation tools also comes at hand to spot these failures.

[gravatar]
Gordon 11:52 AM on 11 Nov 2018

That sort of tuple accident comes up in various contexts. We use this flake8 plugin to catch that sort of thing. https://pypi.org/project/flake8-commas/

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.