Ned Batchelder's blog https://nedbatchelder.com/blog Ned Batchelder's personal blog. en-US Ned Batchelder's blog https://nedbatchelder.com/blog https://nedbatchelder.com/pix/rss-banner.gif SQLite data storage for coverage.py https://nedbatchelder.com//blog/201808/sqlite_data_storage_for_coveragepy.html 2018-08-14T08:41:26-05:00 Ned Batchelder I’m starting to make some progress on Who Tests What. The first task is to change how coverage.py records the data it collects during execution. Currently, all of the data is held in memory, and then written to a JSON file at the end of the process.

But Who Tests What is going to increase the amount of data. If your test suite has N tests, you will have roughly N times as much data to store. Keeping it all in memory will become unwieldy. Also, since the data is more complicated, you’ll want a richer way to access the data.

To solve both these problems, I’m switching over to using SQLite to store the data. This will give us a way to write the data as it is collected, rather than buffering it all to write at the end. BTW, there’s a third side-benefit to this: we would be able to measure processes without having to control their ending.

When running with --parallel, coverage adds the process id and a random number to the name of the data file, so that many processes can be measured independently. With JSON storage, we didn’t need to decide on this filename until the end of the process. With SQLite, we need it at the beginning. This has required a surprising amount of refactoring. (You can follow the carnage on the data-sqlite branch.)

There’s one problem I don’t know how to solve: a process can start coverage measurement, then fork, and continue measurement in both of the child processes, as described in issue 56. With JSON storage, the in-memory data is naturally forked when the processes fork, and then each copy proceeds on its way. When each process ends, it writes its data to a file that includes the (new) process id, and all the data is recorded.

How can I support that use case with SQLite? The file name will be chosen before the fork, and data will be dribbled into the file as it happens. After the fork, both child processes will be trying to write to the same database file, which will not work (SQLite is not good at concurrent access).

Possible solutions:

  1. Even with SQLite, buffer all the data in memory. This imposes a memory penalty on everyone just for the rare case of measuring forking processes, and loses the extra benefit of measuring non-ending processes.
  2. Make buffer-it-all be an option. This adds to the complexity of the code, and will complicate testing. I don’t want to run every test twice, with buffering and not. Does pytest offer tools for conveniently doing this only for a subset of tests?
  3. Keep JSON storage as an option. This doesn’t have an advantage over #2, and has all the complications.
  4. Somehow detect that two processes are now writing to the same SQLite file, and separate them then?
  5. Use a new process just to own the SQLite database, with coverage talking to it over IPC. That sounds complicated.
  6. Monkeypatch os.fork so we can deal with the split? Yuck.
  7. Some other thing I haven’t thought of?

Expect to see an alpha of coverage.py in the next few weeks with SQLite data storage, and please test it. I’m sure there are other use cases that might experience some turbulence...

]]> Fighting spam on freenode https://nedbatchelder.com//blog/201808/fighting_spam_on_freenode.html 2018-08-05T10:43:24-05:00 Ned Batchelder The Freenode IRC network (and I guess others) are under an aggressive spam attack these days. I’ve learned a lot more about Freenode modes in order to help defend against the spam. This is my understanding of the tools and techniques. Corrections welcomed!

IRC is a great way to connect with other people, but it is arcane and poorly documented. How IRC networks are configured and maintained is not standardized. The details here are for Freenode. I have no idea how much of this applies to other networks.

When a user connects to Freenode, they can use a registered nick (nickname), or an unregistered one. Registering is also called “identifying with services,” meaning you have presented your password to the IRC service. Here’s how to register your nick if you haven’t yet: Nickname Registration. The spam bot constantly changes nicks, but doesn’t use registered nicks, so the defense is all based on preventing what unregistered nicks can do.

The main concept for configuring IRC is “modes”. These are single-letter flags that can be applied to users and channels. The modes are different for each: User Modes, and Channel Modes.

The modes you can set depend on whether you are a channel operator, or just a regular user.

For regular users: the spam bot will send you private messages unless you give yourself the +R mode, which means that only registered nicks can send you private messages. To apply this to yourself, use this command in your IRC client (replace “your_nick” with your nick):

# Stop unregistered nicks from PMing me
/mode your_nick +R
# What are my current modes?
/mode your_nick

For channel operators: you have a few options for how to defend your channel. I think the first option is the best.

  • You can allow unregistered nicks to join, but prevent them from speaking with +q. This flag takes an argument which is the pattern of user to quiet. “$~a” means anyone who is unregistered:
    • # Unregistered nicks will be quieted in #your_channel
    /mode #your_channel +q $~a
    # See the list of quieted user patterns
    /mode #your_channel +q
  • If you use +q, but have unregistered bots that you want to be able to talk in your channel, you can exempt them with +e:
    • # Your bot will be able to talk
    /mode #your_channel +e $a:your_bot_nick
    # See the list of exempted user patterns
    /mode #your_channel +e
  • You can completely prevent unregistered nicks from joining your channel with +r:
    • # Only registered nicks can join #your_channel
    /mode #your_channel +r
    # See the list of channel modes
    /mode #your_channel
  • If for whatever reason you cannot stop unregistered nicks from talking in your channel, then you will get spam messages. You can still help defend by inviting freenode’s Sigyn bot into your channel. The bot kills spammers when they send their first message. Invite it like this:
    • # Invite the freenode spam fighting bot
    /invite Sigyn #your_channel
]]> Favorite moths https://nedbatchelder.com//blog/201807/favorite_moths.html 2018-07-30T21:45:24-05:00 Ned Batchelder The Moth is a story-telling podcast. I’ve been listening to it for a long time. I don’t like all the stories, but when I hear one that I do, I really like sharing it with my family. Maybe more than they like me sharing it, but I still try to encourage them to listen to the ones I find for them.

I like stories that are funny but also touching. These are some of my favorites:

  • Empathetic Subway Screaming by Jeff Simmermon: Jeff Simmermon has a run in with a grouchy straphanger on the subway.
  • Father Figure by Steve Zimmer: Hoping to impress his girlfriend’s son with a glassblowing demonstration has disastrous repercussions.
  • Notes on an Exorcism by Andrew Solomon: While studying treatments for depression in rural Africa, Solomon has an overly intimate encounter with a ram.
  • Golf Clubbing by Ed Gavagan: The screwball tale of the child of a tough drill sergeant who is injured on a Texas military base in the early 1970s.
  • Eye Spy by Michaela Murphy: Michaela Murphy endures a family beach vacation.
  • War Games by Micaela Blei: A teacher discovers her playground has turned into a 3rd grade war zone.
]]> Is-a GPL, has-a BSD https://nedbatchelder.com//blog/201807/isa_gpl_hasa_bsd.html 2018-07-19T07:50:02-05:00 Ned Batchelder When talking about classes, and how they relate, there are two classic relationships. Class Y can inherit from class X (inheritance), or class Y can use class X (composition). In the first case we say that an X is-a Y, in the second that an X has-a Y.

Sometimes it isn’t clear which choice is best, and you can imagine using either. One thought that can help decide: is it part of X’s essence that it is a Y? Is it impossible to think of an X that isn’t a Y? Or is Y just a helpful implementation detail of X, and there are other possible ways to build X?

I don’t want to open a can of worms debating licenses, and probably this is an absurd analogy, but it occurred to me that copyleft license (GPL) advocates view their license choice as an is-a, and permissive license (BSD, MIT, Apache) advocates view theirs as a has-a.

]]> Coverage.py is on GitHub https://nedbatchelder.com//blog/201806/coveragepy_is_on_github.html 2018-06-24T16:11:24-05:00 Ned Batchelder After many years on Bitbucket, today I moved coverage.py development to GitHub. I’ve long wanted to use Git instead of Mercurial, but I didn’t want to lose the issues that were on Bitbucket.

I thought GitHub issues were the simplest possible bug tracker. But I realized that Bitbucket’s were even simpler: they have no labels. So I bit the bullet, added some features to the Bitbucket issue migration tool, and switched over.

There had been a GitHub mirror of the code, but I started a new repo so that the issues transferred from Bitbucket could keep their old numbers. The old mirror is still around, because there’s a handful of issues and pull requests there that I didn’t want to lose.

This move is also a kind of way to declare some pull request bankruptcy. I’ve embodied the bad habit of letting pull requests languish, sometimes for years. I shouldn’t let that happen, but I can’t promise that I will do better. I’ll try. At least Git will give me a more familiar environment in which to work with them, which I hope will help.

Another side-effect of making a new repo is that I have hardly any stars yet...

]]> Math factoid of the day: 56 https://nedbatchelder.com//blog/201806/math_factoid_of_the_day_56.html 2018-06-16T09:47:00-05:00 Ned Batchelder 56 is the sum of six consecutive primes (3 + 5 + 7 + 11 + 13 + 17), and is a pronic number, being the product of two consecutive numbers (7 × 8).

]]> Geeky shirts https://nedbatchelder.com//blog/201806/geeky_shirts.html 2018-06-14T18:21:54-05:00 Ned Batchelder I wear t-shirts a lot. A lot of them feature geeky topics. They’re are fun because they start conversations among the people I hang out with. The best times are when someone can take the geeky idea, and one-up it with yet another layer of geekiness.

For example, I have a shirt with this melting Rubik’s cube on it:

Melting Rubik's cube

As geeky as this image is, I add to it by pointing out that even before the melting began, this isn’t a possible configuration of a Rubik’s cube. To avoid spoilers, I’ll explain why at the end of this post.

Here’s another shirt from my closet:

My password is the last 8 digits of pi

In this case, other people added to the geekiness of this shirt. One person suggested that the shirt is simply missing some punctuation, and it should say:

My password is “the last 8 digits of pi”

Of course, there’s still the problem of “pi” vs “π”...

Another person suggested that although no one knows the last 8 digits of pi, my security has been compromised, because it’s really easy to brute-force an 8-digit password, no matter what the digits are.

OK, back to the Rubik’s cube question: why is the melted cube not a real cube? Consider a solved Rubik’s cube:

A solved Rubik's cube

Look at the the two faces colored orange and yellow. They meet at one edge, so there are only two corner cubies (the smaller cubes that make up the complete cube) that have both an orange and a yellow face.

Now think about the two orange-yellow corner cubies on a real cube. If you look at the corner, and think about the faces in clockwise order, one of the cubies must go orange then yellow, and the other must go yellow then orange. They have to have opposite handedness, if you will.

If you look at the melted cube, you can see two corner cubies with both orange and yellow, they are the two closest to the viewer. But they have the same handedness! Going around clockwise, they both go orange and then yellow. So this can’t be a real Rubik’s cube, because these two must be the only two orange-yellow corner cubies, and these two aren’t right.

]]> PyCon 2018 https://nedbatchelder.com//blog/201805/pycon_2018.html 2018-05-16T11:00:23-05:00 Ned Batchelder PyCon 2018 is over, and of course I enjoyed the hell out of it. I tweeted a bit this year, so I’ll tell the story partly with tweets.

I did a talk about Big-O notation: Big-O: How Code Slows as Data Grows. People seemed to like it. I was especially gratified when people told me that the “sitting at the adult table” comment exactly described them, and the talk helped. Nice.

I planned on a juggling open space this year, and brought equipment for it. Getting it through airport security was fun:

And everyone has a wise-crack:

Eric Holscher proposed a rule of thumb: if this is your Nth PyCon, then you have to meet N new people every day. My strategy was at breakfast and lunch, when wandering around looking for a table to sit at, choose one with no one I know, and with people that don’t look like me.

Hillel Wayne did a great talk about advanced testing with Hypothesis and contracts, but he also excelled in a different way:

One goof by the program committee this year: it seemed like often same-topic talks were scheduled in the same time slot. There were four testing talks all scheduled at the same time. One was by Justin Crown (“WHAT IS THIS MESS?” — Writing tests for pre-existing code bases), who I met in the hallway:

The juggling open space was great, a lot (15?) of people showed up, including Christy Heaton who I had met that morning at breakfast. We threw things all over the place:

One of the interesting experiences at PyCon is finally meeting face-to-face with people that you’ve built a strong online-only friendship with, like Mariatta Wijaya.

It’s fun to be captured by Kenneth Reitz, but maybe not outside where it’s windy?

David Beazley showed off how to use the latest meta-programming tools in Python 3.6 to implement a parser generator:

Ying Li’s keynote was a masterful explanation of web security as told through a children’s book. Then Qumisha Goss (who I met at breakfast!) did a great keynote about teaching urban kids programming at a public library:

Another day we had an impromptu juggling session, helped along by penny-whistle accompaniment by Justin Myles Holmes:

Turns out this is conference chair Ernest Durbin’s car:

PyCon did a great job including everyone. At registration, you could choose a ribbon to attach to your badge that announced your preferred pronouns. Also:

I hung around for one day of sprints, which I used mostly to talk with people. Trying to help pylint with an installation issue, I was introduced to check-manifest. Where has it been all my life?

If you are looking for videos to watch, in addition to the ones linked above, take a look at these:

It’s kind of dissatisfying writing about PyCon, because it’s hard to include everything that happened. It doesn’t feel like a technical conference, it feels like a family reunion. I belong to a few different sub-communities and I kept shifting from one to the next: Boston people, ex-Boston people, jugglers, testing geeks, IRC people, meetup organizers, open source developers, and so on.

And I’m sure I’ve forgotten about some amazing piece of the four days. If it was with you, I’m sorry!

It’s always a great place to find your tribe, re-energize, cement old friendships, make new ones, learn, teach, and have fun. Next year in Cleveland!

]]> Money for cotton candy https://nedbatchelder.com//blog/201805/money_for_cotton_candy.html 2018-05-08T18:17:24-05:00 Ned Batchelder My autistic son Nat did something last weekend that got my attention. It’s going to seem like a small thing, and in some ways it is a small thing. One of the things that keeps me thinking about it is why it made such an impression on me.

My father likes circuses, and as a result, I like circuses. It’s been an annual tradition my whole life to see the circus with him. When I was a kid, it was Ringling Brothers at Madison Square Garden. Now it’s the Big Apple Circus in Boston. We take along whatever kids of mine want to go. These days, Nat is the only son in town, so this year, it was me and Nat.

As we were getting ready to leave the house, I was doing the usual things: getting my shoes, telling him to get his shoes, making sure I had the tickets, and so on. Suddenly he looks at me and says something that I didn’t catch. I thought it was about Mom (she was staying behind) and candy. I asked him to repeat it, and it was clear: “Money for cotton candy.”

I was very pleased. He was right: I had to make sure I had money for cotton candy. This was even something I had thought about the night before, but had forgotten now that we were scrambling to get ready to go.

Why did this seem so unusual? Nat is usually focused on snacks. It might be that his entire experience of the circus is: watch inexplicable things for an hour, eat cotton candy, watch some more, go home. As it happens, we hadn’t been to see the circus the last two years, but even a three-year gap is not enough to break the memory of an ingrained tradition like getting cotton candy at the circus.

I wouldn’t have been surprised if he had said, “cotton candy,” meaning, let’s make sure to get some cotton candy. As I say, Nat is very focused on snacks. But he went a step further than that: he thought about what we needed in order to get the cotton candy. Usually he does not handle the money when we are out together. But he understands that money is required, and probably has felt the frustration of wanting a snack and being told, “we can’t, I don’t have money.”

So why was it so pleasing to hear, “Money for cotton candy?” He was thinking about his needs, and thinking about how to really get them met. He advocated for himself without me prompting him. He persisted through the small frustration of not being understood. He helped me by thinking of something I had overlooked. And I guess it made me happy that the circus tradition was strong with him too, even in his own snack-centric way.

Abstract shot of circus tent

]]> Do one thing https://nedbatchelder.com//blog/201804/do_one_thing.html 2018-04-28T08:20:00-05:00 Ned Batchelder A famous and popular set of ideas for object-oriented programs are the SOLID design principles. The name is an acronym of the five principles:

These are all good ideas, but I struggle with the first one. The Single Responsibility Principle says that a class should have only one responsibility. But that’s difficult to be precise about. Doesn’t a class that has three methods do three things? How do you decide if it’s “one responsibility”?

To underscore my uncertainty, it’s easy to find seemingly contradictory examples written by people trying to explain the principle. In SOLID Design Principles Explained – The Single Responsibility Principle, Thorben Janssen says,

The [JPA] specification defines lots of different interfaces for [JPA], specifies a set of entity lifecycle states and the transitions between them, and even provides a query language, called JPQL.

But that is the only responsibility of the JPA specification.

He lists three things, and then says (singular) “that is the only responsibility”? How did we decide that all of that stuff is just one responsibility?

Another example: the SOLID principles were first enumerated by Robert C. Martin. In his book Clean Code, he applies the idea to functions as well as classes. He shows an example of an over-crowded function, and then says,

This bit of code does three things. It loops over all the employees, checks to see whether each employee ought to be paid, and then pays the employee. This code would be better written as:

public void pay() {
    for (Employee e : employees)
        payIfNecessary(e);
}
private void payIfNecessary(Employee e) {
    if (e.isPayday())
        calculateAndDeliverPay(e);
}
private void calculateAndDeliverPay(Employee e) {
    Money pay = e.calculatePay();
    e.deliverPay(pay);
}

Each of these functions does one thing.

That last function does one thing? Then why does it have the word “And” right in the name??

Don’t get me wrong. I’m not saying the Single Responsibility Principle is a bad idea. It’s a great idea. It’s just not quantifiable or precise. It’s a guideline that can be interpreted and applied differently by different people. It requires judgment and interpretation, and can even shift over time as your understanding of the problem changes.

In my Zellij geometric toy I needed to store things keyed by two-dimensional points. But the equality of points had to account for slight inaccuracies that could make “the same” point be not quite equal. I wrote the PointMap class to do that one thing. (Details of the algorithm are in my Finding fuzzy floats blog post.)

As I worked more on the project, I found that I wanted the internals of PointMap in more places. Rather than use that “fuzzy equality” only in PointMap, I needed to also use it to scooch the nearly equal points together so they were exactly equal. I refactored the code to create a Defuzzer class, which only handled the question of, “should this point be considered the same as some previous point we have seen?”

This new Defuzzer class was more useful to me. I could replace the old PointMap class with a Defuzzer and a standard dict, which was simpler.

So: was I wrong when I thought PointMap adhered to the Single Responsibility Principle? It looked good at the time, it was a nice tight class. But Defuzzer is smaller, does less, and is more useful. So it’s definitely a better poster child for SRP. Is one right and the other wrong? I don’t think we can make black-and-white statements like that. As I continue to expand the project, I might find reason to refactor Defuzzer even further. Where would that leave my stark assessment of its orthodoxy?

I’m all in favor of Single Responsibility. But I want people teaching it to help people more with the squishiness of it. Stop giving simplistic examples that clearly seem to violate the idea. Acknowledge the difficulty of knowing the right path.

Programming is hard. Principles and guidelines help, but when we present subjective judgment as binary choices that can be “violated,” we’re doing a disservice to ourselves and to new learners.

]]>