Testing time

Wednesday 29 April 2015

A recent pull request for coverage.py by Conrad Ho added a timestamp to the HTML report pages. Of course, it included tests. They needed a little cleaning up, because they dealt with the current time, and that always gets involved.

The original test looked like this:

def test_has_date_stamp_in_index(self):
    self.run_coverage()
    with open("htmlcov/index.html") as f:
        index = f.read()
    time_stamp = datetime.datetime.now().strftime('%Y-%m-%d %H:%M')
    self.assertIn("<p>Created on {}</p>".format(time_stamp), index)

Here, run_coverage creates the HTML report, then the test reads the HTML file directly, computes the expected timestamp, and then checks that the expected timestamp is in the file.

Seems straightforward enough, but there's a problem. Deep inside run_coverage is a call to datetime.now() to get the current time to create the timestamp. Then in our test, we call datetime.now() again to create the expected timestamp. The problem is that because we call now() twice, they will return different times. Even formatting to hours and minutes as we do, the timestamps could be different.

This test will very occasionally fail: it is a flaky test, which is a very bad thing. Some of the existing tests in the test suite weren't changed in this pull request, but they also become flaky. They looked kind of like this:

def test_html_delta_from_source_change(self):
    self.run_coverage()
    index1 = self.get_html_index_content()

    # ... change some stuff ...

    self.run_coverage()
    index2 = self.get_html_index_content()
    self.assertMultiLineEqual(index1, index2)

Here, we're creating two different HTML reports, and asserting that they are the same. But run_coverage() in each calls now() at different times, so the timestamps can differ in them. Some might say that the chances are really small, and a very occasional test failure is not worth the extra complexity. True story: the first time these tests were run on Travis, they failed because of different timestamps!

One way to solve time problems like this is to mock out datetime.now(), but that can be complicated. So I took different approaches.

The second tests were straightforward to make impervious to the time changes. In that case, I amended get_html_index_content to strip out the timestamp:

def get_html_index_content(self):
    """Return the content of index.html, with timestamps scrubbed."""
    with open("htmlcov/index.html") as f:
        index = f.read()
    index = re.sub(
        r"created at \d{4}-\d{2}-\d{2} \d{2}:\d{2}",
        r"created at YYYY-MM-DD HH:MM",
        index,
    )
    return index

Now the text of index.html doesn't have the timestamp, so the value of now() doesn't matter, and the tests aren't flaky. These are tests of other aspects than the timestamp, so it's fine to just remove the timestamp.

But the first tests were about the timestamp itself, we can't just scrub it from the output. For those tests, I chose a different approach: extract the timestamp from the HTML, and check that it is a very recent timestamp:

def test_has_date_stamp_in_files(self):
    self.run_coverage()
    with open("htmlcov/index.html") as f:
        self.assert_correct_timestamp(f.read())

def assert_correct_timestamp(self, html):
    """Extract the timestamp from `html`, and assert it is recent."""
    timestamp_pat = r"created at (\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2})"
    m = re.search(timestamp_pat, html)
    self.assertTrue(m, "Didn't find a timestamp!")
    timestamp = datetime.datetime(*map(int, m.groups()))

    age = datetime.datetime.now() - timestamp
    self.assertEqual(age.days, 0)
    # The timestamp only records the minute, so the delta could be from
    # 12:00 to 12:01:59, or two minutes.
    self.assertLessEqual(
        abs(age.seconds),
        120,
        "Timestamp is wrong: {0}".format(timestamp)
    )

Here I have a new method, assert_correct_timestamp. It takes the content of the HTML, extracts the timestamp with a regex, converts it into a datetime, and then checks that the datetime is recent. This fixes the flaky test: it will not fail due to shifting time windows.

But now the test method has a bunch of code for figuring out if the datetime is recent. And it has a bug: I used abs(age.seconds) < 120, which will pass if the datetime is in the near future as well as in the near past.

This test has two ideas in it: get the timestamp from the HTML code, and check if it is recent. Better would be to factor out that second part into its own datetime assert method:

def assert_recent_datetime(self, dt, seconds=10, msg=None):
    """Assert that `dt` marks a time at most `seconds` seconds ago."""
    age = datetime.datetime.now() - dt
    self.assertEqual(age.days, 0, msg)
    self.assertGreaterEqual(age.seconds, 0, msg)
    self.assertLessEqual(age.seconds, seconds, msg)

This assert method is purely about datetimes and their recency. We've fixed the bug with the near future. Now we can test this assert method directly to be sure we have the logic right:

def test_assert_recent_datetime(self):
    def now_delta(seconds):
        """Make a datetime `seconds` seconds from now."""
        return datetime.datetime.now() + datetime.timedelta(seconds=seconds)

    # Default delta is 10 seconds.
    self.assert_recent_datetime(now_delta(0))
    self.assert_recent_datetime(now_delta(-9))
    with self.assertRaises(AssertionError):
        self.assert_recent_datetime(now_delta(-11))
    with self.assertRaises(AssertionError):
        self.assert_recent_datetime(now_delta(1))

    # Delta is settable.
    self.assert_recent_datetime(now_delta(0), seconds=120)
    self.assert_recent_datetime(now_delta(-100), seconds=120)
    with self.assertRaises(AssertionError):
        self.assert_recent_datetime(now_delta(-1000), seconds=120)
    with self.assertRaises(AssertionError):
        self.assert_recent_datetime(now_delta(1), seconds=120)

And with all that in place, we can simplify our HTML report test:

def assert_correct_timestamp(self, html):
    """Extract the timestamp from `html`, and assert it is recent."""
    timestamp_pat = r"created at (\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2})"
    m = re.search(timestamp_pat, html)
    self.assertTrue(m, "Didn't find a timestamp!")
    timestamp = datetime.datetime(*map(int, m.groups()))
    # The timestamp only records the minute, so the delta could be from
    # 12:00:00 to 12:01:59, or two minutes.
    self.assert_recent_datetime(
        timestamp,
        seconds=120,
        msg="Timestamp is wrong: {0}".format(timestamp),
        )

Nice.

Comments

[gravatar]
Julian 9:33 AM on 30 Apr 2015

Do you ever run the test at 2am on Sunday mornings? This may fall over at Daylight Savings change-over. Probably not worth solving for tests, but worth considering by anyone re-using your blog post code for other purposes.

[gravatar]
Tomas 10:38 AM on 30 Apr 2015

There still seems to be much uncertainty when doing time range check – why not just wrap datetime.datetime.now() to a separate method in coverage.py and simply mock that method?

[gravatar]
Ned Batchelder 11:01 AM on 30 Apr 2015

@Tomas: you might be right that mocking is better. I was interested to try these techniques and see what I thought of them.

[gravatar]
Chris Warrick 5:29 PM on 30 Apr 2015

You could try the freezegun package from PyPI: https://pypi.python.org/pypi/freezegun

Does all the mocking for you, intelligently.

[gravatar]
Eric 11:45 PM on 30 Apr 2015

I've been using a pattern that records the time at the beginning of the test, and asserts that a given time is between then and the time of the assertion.

https://gist.github.com/eswald/098120b5aca16d01a080

Granted, in this case, you would have to create a datetime object from parsing the web page content, and the start time would need to be rounded down to the minute.

[gravatar]
Tiw 1:43 PM on 22 May 2015

This may not be appropriate depending on how the application is structured, but wouldn't it be better to inject the time into the function that renders the html and then assert that the output timestamp matches the injected one?

Add a comment:

name
email
Ignore this:
not displayed and no spam.
Leave this empty:
www
not searched.
 
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.