« | » Main « | »

Testing time

Wednesday 29 April 2015

A recent pull request for coverage.py by Conrad Ho added a timestamp to the HTML report pages. Of course, it included tests. They needed a little cleaning up, because they dealt with the current time, and that always gets involved.

The original test looked like this:

def test_has_date_stamp_in_index(self):
    self.run_coverage()
    with open("htmlcov/index.html") as f:
        index = f.read()
    time_stamp = datetime.datetime.now().strftime('%Y-%m-%d %H:%M')
    self.assertIn("<p>Created on {}</p>".format(time_stamp), index)

Here, run_coverage creates the HTML report, then the test reads the HTML file directly, computes the expected timestamp, and then checks that the expected timestamp is in the file.

Seems straightforward enough, but there's a problem. Deep inside run_coverage is a call to datetime.now() to get the current time to create the timestamp. Then in our test, we call datetime.now() again to create the expected timestamp. The problem is that because we call now() twice, they will return different times. Even formatting to hours and minutes as we do, the timestamps could be different.

This test will very occasionally fail: it is a flaky test, which is a very bad thing. Some of the existing tests in the test suite weren't changed in this pull request, but they also become flaky. They looked kind of like this:

def test_html_delta_from_source_change(self):
    self.run_coverage()
    index1 = self.get_html_index_content()

    # ... change some stuff ...

    self.run_coverage()
    index2 = self.get_html_index_content()
    self.assertMultiLineEqual(index1, index2)

Here, we're creating two different HTML reports, and asserting that they are the same. But run_coverage() in each calls now() at different times, so the timestamps can differ in them. Some might say that the chances are really small, and a very occasional test failure is not worth the extra complexity. True story: the first time these tests were run on Travis, they failed because of different timestamps!

One way to solve time problems like this is to mock out datetime.now(), but that can be complicated. So I took different approaches.

The second tests were straightforward to make impervious to the time changes. In that case, I amended get_html_index_content to strip out the timestamp:

def get_html_index_content(self):
    """Return the content of index.html, with timestamps scrubbed."""
    with open("htmlcov/index.html") as f:
        index = f.read()
    index = re.sub(
        r"created at \d{4}-\d{2}-\d{2} \d{2}:\d{2}",
        r"created at YYYY-MM-DD HH:MM",
        index,
    )
    return index

Now the text of index.html doesn't have the timestamp, so the value of now() doesn't matter, and the tests aren't flaky. These are tests of other aspects than the timestamp, so it's fine to just remove the timestamp.

But the first tests were about the timestamp itself, we can't just scrub it from the output. For those tests, I chose a different approach: extract the timestamp from the HTML, and check that it is a very recent timestamp:

def test_has_date_stamp_in_files(self):
    self.run_coverage()
    with open("htmlcov/index.html") as f:
        self.assert_correct_timestamp(f.read())

def assert_correct_timestamp(self, html):
    """Extract the timestamp from `html`, and assert it is recent."""
    timestamp_pat = r"created at (\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2})"
    m = re.search(timestamp_pat, html)
    self.assertTrue(m, "Didn't find a timestamp!")
    timestamp = datetime.datetime(*map(int, m.groups()))

    age = datetime.datetime.now() - timestamp
    self.assertEqual(age.days, 0)
    # The timestamp only records the minute, so the delta could be from
    # 12:00 to 12:01:59, or two minutes.
    self.assertLessEqual(
        abs(age.seconds),
        120,
        "Timestamp is wrong: {0}".format(timestamp)
    )

Here I have a new method, assert_correct_timestamp. It takes the content of the HTML, extracts the timestamp with a regex, converts it into a datetime, and then checks that the datetime is recent. This fixes the flaky test: it will not fail due to shifting time windows.

But now the test method has a bunch of code for figuring out if the datetime is recent. And it has a bug: I used abs(age.seconds) < 120, which will pass if the datetime is in the near future as well as in the near past.

This test has two ideas in it: get the timestamp from the HTML code, and check if it is recent. Better would be to factor out that second part into its own datetime assert method:

def assert_recent_datetime(self, dt, seconds=10, msg=None):
    """Assert that `dt` marks a time at most `seconds` seconds ago."""
    age = datetime.datetime.now() - dt
    self.assertEqual(age.days, 0, msg)
    self.assertGreaterEqual(age.seconds, 0, msg)
    self.assertLessEqual(age.seconds, seconds, msg)

This assert method is purely about datetimes and their recency. We've fixed the bug with the near future. Now we can test this assert method directly to be sure we have the logic right:

def test_assert_recent_datetime(self):
    def now_delta(seconds):
        """Make a datetime `seconds` seconds from now."""
        return datetime.datetime.now() + datetime.timedelta(seconds=seconds)

    # Default delta is 10 seconds.
    self.assert_recent_datetime(now_delta(0))
    self.assert_recent_datetime(now_delta(-9))
    with self.assertRaises(AssertionError):
        self.assert_recent_datetime(now_delta(-11))
    with self.assertRaises(AssertionError):
        self.assert_recent_datetime(now_delta(1))

    # Delta is settable.
    self.assert_recent_datetime(now_delta(0), seconds=120)
    self.assert_recent_datetime(now_delta(-100), seconds=120)
    with self.assertRaises(AssertionError):
        self.assert_recent_datetime(now_delta(-1000), seconds=120)
    with self.assertRaises(AssertionError):
        self.assert_recent_datetime(now_delta(1), seconds=120)

And with all that in place, we can simplify our HTML report test:

def assert_correct_timestamp(self, html):
    """Extract the timestamp from `html`, and assert it is recent."""
    timestamp_pat = r"created at (\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2})"
    m = re.search(timestamp_pat, html)
    self.assertTrue(m, "Didn't find a timestamp!")
    timestamp = datetime.datetime(*map(int, m.groups()))
    # The timestamp only records the minute, so the delta could be from
    # 12:00:00 to 12:01:59, or two minutes.
    self.assert_recent_datetime(
        timestamp,
        seconds=120,
        msg="Timestamp is wrong: {0}".format(timestamp),
        )

Nice.

How I make presentations

Saturday 25 April 2015

I like giving talks. I spend a lot of time on my presentation slides, and have a typically idiosyncratic toolchain for them. This is how I make them. Note: I am not recommending that anyone else make slides this way. If you like it, fine, but most people will prefer more common tools.

I generally favor text-based tools over WYSIWYG, and slides are no exception. For simple presentations, I will use Google Docs. But PyCon talks are not simple. They usually involve technical details, or involved explanations, and I want to have code helping me make them. I choose text tools for the control they give me, not for convenience.

HTML-based presentations are popular, and they suit my need for text-based tooling. Other options include Markdown- or ReST-based tools, but they remove control rather than provide it, so I prefer straight-up HTML.

There are a number of HTML-based presentation tools, like impress.js and reveal.js. For reasons lost in the mists of time, I long ago chose one that no one else seems to use: Slippy. Maybe someday I will switch, but Slippy does what I need.

To make a Slippy presentation, I create a .html file, open it in vim, and start typing. Each slide is a <div class="slide">. To see and present the slides, I just open that HTML file in a browser. If you want to see an actual artifact, click the "actual presentation" link on any of my recent talks, or take a look at the repo for one of them:

When I need more power than just my own typing, I want to use Python to produce content. In Pragmatic Unicode, I used it to produce tables of character translations, and to run the Python code samples. In Names and Values, I used it to write Cupid figures.

To run Python code that can create content in my HTML file, I use Cog, a tool I wrote that can execute Python code inline in a text file, and put the output back into the file. I originally wrote it to solve a different problem, but it works great here. It lets me stick with a workflow where I have one file that contains both the program and result.

Sometimes, I don't need Cog. Loop Like a Native is just static text, with no need, so it's not in there.

For explaining code, it's very helpful to be able to highlight individual lines in a snippet on the screen. I couldn't find a way to do this, so I wrote lineselect.js, a jQuery plugin to let me select individual lines. While presenting, I use a presentation remote with volume control buttons, and remap those keys to j and k so that I can manually move the line selection as I talk.

As I write the presentation, I like working out what I am going to say by writing it out in English. This helps me find the right way to explain things, but has another huge advantage: it means I have a written presentation as well as a visual one. It frustrates me to hear about someone's great presentation, and then to have two options of how to learn from it: either watch a video, or look at slides with no words behind them.

When I write the English, I put it into the .html file also, interleaved with the slides, as <div class="text">. CSS lets me hide those divs during the presentation, but I can work in my HTML file and see the slides near the text.

For publication on my site, I have a Python program that parses the HTML and extracts the text divs into a .px file for insertion into my typically idiosyncratic site publication toolchain.

Producing that .px file also involves producing PNGs from the slides. Slippy comes with a phantomjs program to do this which works well. The px-producing program inserts those PNGs into the page.

As I say, I'm not explaining this to convince you to make slides this way. Most people will vastly prefer a more convenient set of tools. I like the control this gives me, and I like writing the kind of tooling I need to make them this way. To each her own.

Theater cake

Sunday 19 April 2015

My dad and stepmother were here for lunch yesterday. It happened to be their 45th wedding anniversary, so we made them a cake. Not anniversary themed, but theater-themed, because that is a huge passion of theirs. They both worked in the theater, and continue to help run the Barnstomers Theater in New Hampshire.

So we made a theater cake, with stage, house (where the audience sits), proscenium, lights, curtains, and some kind of confused production going on:

Theater cake

The view from backstage:

Theater cake, backstage

You can't see the seats, they are Rolos with Hershey bar backs. Sorry for the poor photo quality...

PyCon 2015

Wednesday 15 April 2015

I am on the plane back to Boston from PyCon 2015 in Montreal. You've probably read over and over again that PyCon is the best conference ever, yadda-yadda. I haven't been to another conference in a long time, so I don't have points of comparison. I can tell you that PyCon feels like a huge family reunion.

I started on Thursday, and was not feeling part of things. I don't know why. I thought perhaps 9 PyCons in a row is too many. I thought maybe I should be spending my energies elsewhere.

But Friday, I started the day by helping with the keynotes, keeping time, tracking down speakers, and so on. I felt involved. I was helping friends with things they needed to do.

PyCon is almost entirely organized and run by volunteers. There is one employee, all the rest is done by people just helping as a side project. I think this gives the event a tone of something you do, rather than something you attend or consume. Anyone can volunteer to make things happen, and it can be a really good way to meet people.

There are 2500 people at PyCon, but we are all in the same group. There isn't a entire cadre of paid staff on one side, and attendees on the other. We're all making the conference happen in our own ways. It an open-source conference in the truest sense of the word.

Adam

My co-worker Adam Palay gave his talk early on Friday. I'd first seen Adam speak in a lightning talk at Boston Python. His girlfriend Anne was there to record him. They seemed supportive and close. I really liked the talk he gave, and told him so. When the call for talks opened for PyCon, he let me know he was submitting a proposal, and I helped him where I could.

His talk was accepted, along with mine and two other speakers from edX. For each talk, we had a rehearsal at work, and at a Boston Python rehearsal night. Each time Adam rehearsed his talk, his girlfriend Anne and his brother Josh were there. I was impressed by their support. It turned out Anne was going to not only come to Montreal, but attend the conference with him.

Friday morning at PyCon, I went to Adam's talk. Sitting in the second row was Anne. Next to her was Josh. Next to him was Adam's sister, and on either side were his mother and father, all with conference badges! I joked about "Team Palay", and that the five of them should have held up cards spelling P-A-L-A-Y.

Clearly, this level of support from a family is unusual, to take the time, buy airfare and hotel, and pay the conference fees, just to see Adam present his 30-minute talk at a technical conference.

I'm explaining all this about Adam's supportive family because when I am at PyCon, I feel a bit like Adam must all the time. I am surrounded by friends who feel like family. We are brought together by an odd esoteric shared interest, but we come together each year, and interact online throughout the year. We are together to talk about technical topics, but it goes beyond that.

I know this must sound like a greeting card or something. Don't get me wrong: like any family, there is friction. I don't like everyone in the Python world. But so many people at PyCon know each other and have built relationships over years, there are plenty of friendly faces all around.

All those friendly faces give rise to an effect my devops guy Feanil coined "Ned latency": the extra time I have to figure in when planning to be at a certain place at a certain time. When traveling over any significant distance at PyCon, there will be people I want to stop and talk to.

This is called the "hallway track": the social or technical activity that happens in the hallways all during the day, regardless of the track talks. I've spoken to people at PyCon who've said, "I haven't seen any talks!"

Jenny

Last year during lunch, I happened to sit next to a woman I didn't know. We introduced ourselves. Her name was Jenny. We chatted a bit, and then headed off to our own activities. Over the next few days, I'd wave to Jenny as we passed each other on the escalators, and so on.

I saw Jenny again this year and miraculously remembered her name, so I waved and said, "Hi Jenny." This happened a few times. Later in the weekend, Jenny came up to me and said, "I want to thank you, you really made me feel welcome."

This made me really happy. I was saying hi to Jenny originally so that I would know more people, but we'd made a tiny connection that helped her in some way, and she felt strongly enough about it to tell me. Ian describes a similar dynamic from the bag-stuffing evening: just learning another person's name gives you a connection to that person that can last a surprisingly long time.

There are people I greet at PyCon purely because I've been chatting with them for five minutes once a year at every PyCon I've been to.

Speaking

One of the highlights of PyCon for me is giving talks. I've spoken at the last 7 PyCons (the talks are on my text page). I put a lot of work into the talks, and am proud that they have some lasting power as things people recommend to other learners. After a talk, people always ask, "how did it go?" My answer is usually, "people seemed to like it," but the other half is, "on the inside, horrible. I know all the things I wish I had done differently!"

On Sunday evening, Shauna Gordon-McKeon and Open Hatch organized an intro to sprinting session for new contributors. I agreed to be a mentor there, thinking it would be a classroom style lecture, with mentors milling around helping people one-on-one. Turned out it was a series of 15-minute lectures at a number of stations around the room, with people shuttling between topics they wanted to hear about. I was the speaker on unit testing.

I was able to start by saying, if you really want to know about this, see my PyCon talk from last year, Getting Started Testing. Then I launched into an impromptu 15-minute overview of unit testing.

During one of the breaks, on my way to the water fountain, I passed a woman in the hallway watching the talk on her headphones. She said it was great, then later on Twitter, we had a typical PyCon love-fest.

To be able to see someone learning from something you've created is very gratifying and rewarding.

Sprinting

I attended one day of sprints. My main project there was Open edX, but I also said I would be sprinting on coverage.py, which I had never done before. I'd always had the feeling that coverage.py was esoteric and thorny, and it would be difficult to get contributors going. I was pleasantly surprised that five people joined me to make some headway against issues in the tracker.

But some of the interesting bugs are about branch coverage, which I had become somewhat frustrated by. I warned people that the problems might require a complete rewrite, but they were game to look into it.

Mickie Betz in particular was digging into an issue involving loops in generators. I was interested to watch her progress, and helped her with debugging techniques, but was not hopeful that there was a practical fix. To my surprise, a day later, she has submitted a pull request with a very simple solution. Mickie has restored my faith in humanity. She persevered in the face of a discouraging maintainer (me), and proved him wrong!

Another sprinter, Jon Chappell, picked up an issue that was easy but annoying to fix. Annoying because it was asking coverage.py to accommodate a stupid limitation in a different tool. It was not glamourous work, but I really appreciated him taking the task so that I didn't have to do it.

Two other sprinters, Conrad Ho and Leonardo Pistone, have each submitted a pull request, and Leonardo is also chasing down other issues. Lastly, Frederick Wagner has expressed interest in adding a warning-suppressing feature.

A very productive time, considering I was only at the sprints for about four hours. PyCon is amazing.

Juggling

One thing I've never seen at PyCon is organized juggling. I considered bringing beanbags with me this time, but thought they would be heavy to carry around. Then Yelp was handing out bouncy balls at their booth, so I got four of those, and used them all weekend. It was a good way to play with people, especially once we did some pair juggling. Next year, I'll bring some serious equipment, and have a real open space (or two!) Who's in?

All in all

I don't know why I felt off the first day. PyCon is an amazing time, and now I again can't imagine missing it. It connects you to people. One afternoon, an attendee pulled me aside to show me a bug in coverage.py. I looked in the issue tracker, and saw that it had been written up four years ago by Christian Heimes, who was attending PyCon this year for the first time, and who I met at the bar on my first night!

PyCon energizes me, and cements my relationship to the entire Python world. Sometimes I wonder about a programming language as the basis for a group of people, but why not? They share my sensibilities and interests. They like what I do, and I like what they do. We move in similar circles. Do you need better reasons for a group of 2500 people to be close friends?

Names and Values at PyCon 2015

Sunday 12 April 2015

I gave my talk yesterday at PyCon 2015: Python Names and Values. PyCon has always been good at getting videos online, but they just keep getting better: the video was online the same day.

People ask me afterwards how the talk went. I got good reactions, but I also know what I would like to have done differently. I think I spoke too fast, and I think I should have had more practical advice about not mutating values if you can avoid it.

At least I didn't swear on stage this time...

« | » Main « | »