It’s hard to know the exact number, but today Heinz has thousands of products, including at least 20 ketchups.

BTW, you might be interested in other posts I’ve written on this day in the past.

Let’s look at the Why and How model as it applies to corporations funding open source. They don’t do it because the answers to Why and How are really bad right now.

Why should a corporation fund open source? As much as I wish it were different for all sorts of reasons, corporations act only in purely selfish ways. In order to spend money, they need to see some positive benefit to them that wouldn’t happen if they didn’t spend the money.

This frustrates me because a corporation is a collection of people, none of whom would act this way. I could say much more about this, but we aren’t going to be able to change corporations.

Companies only spend money if doing so will bring them a (perceived) benefit. Funding open source would make it stronger and better, but that is a very long effect, and not one that accrues directly to the funder. This is the famous Tragedy of the Commons. It’s a fair question for companies to ask: if they fund open source, what do they get for their money?

That’s the difficulty with Why, but let’s imagine for a moment that we could somehow convince someone to spend their company’s money funding open source: now what? How do they do it? A significant Python project could have a hundred library dependencies. How do they decide how to allocate the funding budget among them? Once that decision is made, how does the money get delivered? Very few open source project are equipped to receive funds. If even 10% of the projects have a clear path for funding, now there are 10 checks to write, or 10 PayPal links to click through or whatever? Some of that money will need to be sent internationally, and it has to be considered at tax time. Does it have to be done again next year, and the year after that? It’s a logistical nightmare!

So when we try to convince companies to fund open source, we don’t have good answers for either Why? or How? It’s no wonder it doesn’t happen.

This is one of the reasons I am optimistic about Tidelift: they have good answers for both of these questions. The Tidelift subscription gives companies information and services around their open source dependencies, which answers the why. And the payment to Tidelift solves the how: Tidelift looks at the list of dependencies, decides an allocation, and distributes the money to the maintainers.

Sure, there are still lots of questions to be answered: is the allocation algorithm right? Will enough companies subscribe to make Tidelift itself sustainable? And even larger questions, like: if an interesting amount of money does flow to open source maintainers, what will be the cultural change in open source?

I don’t know the answers to those questions, but Tidelift seems like the most promising answer to how to support open source. I’m an enthusiastic participant. You should be too.

But let’s start with the syntax for defining functions. It has this really nice property: function definitions look like their corresponding function calls. A function is defined like this:

def func_name(arg1, arg2):


When you call the function, you use similar syntax: the name of the function, and a comma-separated list of arguments in parentheses:

x = func_name(12, 34)


Just by lining up the punctuation in the call with the same bits of the definition, you can see that arg1 will be 12, and arg2 will be 34. Nice.

OK, so now let’s look at how a class with base classes is defined:

class MyClass(BaseClass, AnotherBase):


To create an instance of this class, you use the name of the class, and parens, but now the parallelism is gone. You don’t pass a BaseClass to construct a MyClass:

my_obj = MyClass(...)


Just looking at the class line, you can’t tell what has to go in the parens to make a MyClass object. So “def” and “class” have very similar syntax, and function calls and object creation have very similar syntax, but the mimicry in function calls that can guide you to the right incantation will throw you off completely when creating objects.

This is the sort of thing that experts glide right past without slowing down. They are used to arcane syntax, and similar things having different meanings in subtly different contexts. And a lot of that is inescapable in programming languages: there are only so many symbols, and many many more concepts. There’s bound to be overlaps.

But we could do better. Why use parentheses that look like a function call to indicate base classes? Here’s a better syntax:

class MyClass from BaseClass, AnotherBase:


Not only does this avoid the misleading punctuation parallelism, but it even borrows from the English we use to talk about classes: MyClass derives from BaseClass and AnotherBase. And “from” is already a keyword in Python.

BTW, even experts occasionally make the mistake of typing “def” where they meant “class”, and the similar syntax means the code is valid. The error isn’t discovered until the traceback, which can be baffling.

I’m not seriously proposing to change Python. Not because this wouldn’t be better (it would), but because a change like this is impractical at this late date. I guess it could be added as an alternative syntax, but it would be hard to argue that having two syntaxes for classes would be better for beginners.

But I think it is helpful to try to see our familiar landscape as confused beginners do. It can only help with explaining it to them, and maybe help us make better choices in the future.

There have been attempts to improve this situation, but they usually are some form of internet tip jar that goes nowhere. Businesses won’t put money in tip jars because they don’t know what to contribute to (they have hundreds of open source dependencies), and as infuriating as it is, they wonder what they get for that money (they already got the software!!)

Tidelift is approaching the problem of sustainable open source differently: what help do enterprises need with open source? What services would they be willing to pay for? How can enterprises be connected with open source maintainers to benefit both?

They sell the Tidelift Subscription, a collection of tools, information, and assurances to close some of the gaps businesses typically face when using open source.

The people behind Tidelift have deep experience at the intersection of open source and enterprises, having come from Red Hat, Gnome, and Mozilla. They’ve thought a lot about the problem of open source sustainability from both sides, and know what they are doing.

Coverage.py is part of the Tidelift Subscription, which makes me “a Lifter.” I get a small but not insignificant amount of money each month as a result. I want Tidelift to succeed partly for myself, but more importantly, because it could mean that open source is more sustainable overall.

If you are an open source maintainer, take a look at whether you can make money from Tidelift. What they ask of you is pretty much what well-maintained projects already do (good release notes, accurate metadata, points of contact), and they can help with some things that are difficult, like security reporting and license compliance.

If your company uses open source, consider whether the subscription is something you would use. It could help your business, and it would definitely help open source.

Thanks.

The biggest changes are due to Stephan Richter and Justas Sadzevičius, from Shoobox. They improved the support for recording dynamic contexts, informally known as Who Tests What.

Now third-party code, either as a coverage.py plugin or using the coverage.py API can set the dynamic context.

I’ve added support for this to the pytest-cov plugin, to record the pytest test id as the dynamic context. If you’d like to try it:

pip install coverage==5.0a5
pip install git+https://github.com/nedbat/pytest-cov.git@nedbat/contexts
pytest --cov=. --cov-context


The .coverage data file is now a SQLite database. Coverage.py has no support yet for using the collected context data, but you can examine the raw data in the database:

$ sqlite3 .coverage
SQLite version 3.19.3 2017-06-27 16:48:08
Enter ".help" for usage hints.

sqlite> select * from context;
id          context
----------  --------------------------------------------------
1
2           test_it.py::test_prod1|setup
3           test_it.py::test_prod1|call
4           test_it.py::test_prod1|teardown
5           test_it.py::test_prod2|setup
6           test_it.py::test_prod2|call
7           test_it.py::test_prod2|teardown
8           test_it.py::test_prod3[1-1]|setup
9           test_it.py::test_prod3[1-1]|call
10          test_it.py::test_prod3[1-1]|teardown
11          test_it.py::test_prod3[10-100]|setup
12          test_it.py::test_prod3[10-100]|call
13          test_it.py::test_prod3[10-100]|teardown
14          test_it.py::test_prod3[11-121]|setup
15          test_it.py::test_prod3[11-121]|call
16          test_it.py::test_prod3[11-121]|teardown

sqlite> select * from arc where context_id = 9;
file_id     context_id  fromno      tono
----------  ----------  ----------  ----------
1           9           -14         15
1           9           15          16
1           9           16          17
1           9           17          -14

sqlite> select * from file where id = 1;
id          path
----------  --------------------------------------------------
1           /Users/ned/lab/pytest_func_test/src/product.py


I’m looking for feedback about what kinds of reporting would be useful. Stephan has a pull request to provide some context-based reporting. Does it do what you want? Have you used contexts? What needs to happen before they are ready for everybody?

I use this to import modules I often want to use, define helpers, and configure my command history.

In my .bashrc I have:

export PYTHONSTARTUP=~/.startup.py


Then my .startup.py file is:

# Ned's startup.py file, loaded into interactive python prompts.
# Has to work on both 2.x and 3.x

print("(.startup.py)")

import collections, datetime, itertools, math, os, pprint, re, sys, time
print("(imported collections, datetime, itertools, math, os, pprint, re, sys, time)")

pp = pprint.pprint

# A function for pasting code into the repl.
def paste():
    import textwrap
    exec(textwrap.dedent(sys.stdin.read()), globals())

# Readline and history support
def hook_up_history():
    try:
        # Not sure why this module is missing in some places, but deal with it.
        import readline
    except ImportError:
        print("No readline, use ^H")
    else:
        import atexit
        import os
        import rlcompleter

        history_path = os.path.expanduser(
            "~/.pyhistory{0}".format(sys.version_info[0])
        )

        def save_history(history_path=history_path):
            import readline
            readline.write_history_file(history_path)

        if os.path.exists(history_path):
            readline.read_history_file(history_path)

        atexit.register(save_history)

# Don't do history stuff if we are IPython, it has its own thing.
is_ipython = 'In' in globals()
if not is_ipython:
    hook_up_history()

# Get rid of globals we don't want.
del is_ipython, hook_up_history


A few things could us an explanation. The paste() function lets me paste code into the REPL that has blank lines in it, or is indented. Basically, I can copy code from somewhere, and use paste() to paste it into the prompt without having to fix those things first. Run paste(), then paste the code, then type an EOF indicator (Ctrl-D or Ctrl-Z, depending on your OS). The pasted code will be run as if it had been entered correctly.

The history stuff gives me history that persists across Python invocations, and keeps the Python 2 history separate from the Python 3 history. “pp” is very handy to have as a short alias.

Of course, you can put anything you want in your own .startup.py file. It’s only run for interactive sessions, not when you are running programs, so you don’t have to worry that you will corrupt important programs.

It has sat dormant for a long time. Recently someone asked me if it was maintained, and I huffily answered, “it’s maintained as much as it needs to be.” But they were right to ask: it certainly had the look of an abandoned property.

So I jumped in and did a bunch of work to it. Development moved from Bitbucket to GitHub. I merged a few pull requests. I added Travis and Appveyor CI.

The biggest functional change is that errors during execution now get reasonable tracebacks that don’t require you to reverse-engineer how cog ran your code.

I even used mutmut to add a few more tests.

The result is Cog 3.0, ready for your use! I’ll try to stay on top of it better now, I promise!

Calvin Spealman (aka ironfroggy) chimed in:

[ironfroggy] didn't you write a madlibs python blog post like... forever ago?
    [nedbat] yes :)  14 years ago I think.
    [nedbat] my son was 13, and he just turned 27...
[ironfroggy] nedbat: fun fact: i read that when my wife was pregnant.
[ironfroggy] my son turns 13 in a few weeks.
[ironfroggy] we make games together now
    [nedbat] :) i like the symmetry


My post from 14 years ago is Programming madlibs, written based on a project I did with my then 13-year-old. To think that Calvin read it on the brink of becoming a father, and now has a son the same age that mine was then, is mind-bending.

It’s kind of like a circle of life or something, but I guess it’s just a circle of Mad Libs, which is still good.

Mutation testers modify (mutate) your project code in small ways, then run your test suite. If the tests all pass, then that mutation is considered a problem: a bug that your tests didn’t catch. The theory is that a mutation will change the behavior of your program, so if your test suite is testing closely enough, some test should fail for each mutation. If a mutation doesn’t produce a test failure, then you need to add to your tests.

There are a few problems with this plan. The first is that it is time-consuming. Most people feel like it takes too long to run their entire test suite just once. Mutation testers run the whole suite once for each mutation, and there can be thousands of mutations.

But my larger concern is false positives: not all mutations are bugs, and if the mutation tester reports too many non-bugs as bugs, then its usefulness is diminished or even negated. I wanted to examine this idea more closely.

There are a few mutation testers out there for Python. I thought I would give them a try, starting with mutmut. [Mutmut’s author Anders Hovmöller helped by commenting on a draft of this post. I’ve included some of his commentary.]

I needed a test suite to use, so I created a slightly artificial project. The templite module in coverage.py is almost standalone, and is well-tested. And it’s small enough that its test suite runs in less than a second. I extracted templite, wrote some project scaffolding, and gave it its own repository.

Now I had a project that tested well:

$ coverage run -m pytest
============================= test session starts ==============================
platform darwin -- Python 3.7.1, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /Users/ned/lab/templite, inifile:
collected 26 items

test_templite.py ..........................                              [100%]

========================== 26 passed in 0.09 seconds ===========================

$ coverage report -m
Name              Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------
src/templite.py     144      1     60      1    99%   137, 136->137


(The one line missing coverage is a conditional for Python 2 vs Python 3.)

Running mutmut was easy:

$ pip install mutmut
Collecting mutmut
...
Installing collected packages: mutmut
Successfully installed mutmut-1.3.1

$ mutmut run

- Mutation testing starting -

These are the steps:
1. A full test suite run will be made to make sure we
   can run the tests successfully and we know how long
   it takes (to detect infinite loops for example)
2. Mutants will be generated and checked

Mutants are written to the cache in the .mutmut-cache
directory. Print found mutants with `mutmut results`.

Legend for output:
🎉 Killed mutants. The goal is for everything to end up in this bucket.
⏰ Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁 Survived. This means your tests needs to be expanded.

mutmut cache is out of date, clearing it...
1. Running tests without mutations
⠇ Running... Done

2. Checking mutants
⠧ 154/154  🎉 146  ⏰ 0  🤔 0  🙁 8


This ran 154 different mutations, which took about a minute for my half-second-ish test suite. 146 of them resulted in test suite failures, as they should. But 8 passed the test suite, so they have to be examined as potential test gaps.

One nice touch: if you interrupt mutmut, when you run it again, it picks up where it left off, which is great for a long-running process like this.

I’m not sure how mutmut decides where to find the code to mutate. In this case it found it implicitly. Other projects I tried, I had to add some configuration to setup.cfg, even though I thought the projects were laid out similarly.

[Anders says it looks for “src”, “lib”, or a directory with the same name as the current directory. My other project has a quirk: edx-lint/edx_lint has the code, so the punctuation difference threw it off.]

To look at the mutants, use the results command:

$ mutmut results
To apply a mutant on disk:
    mutmut apply <id>

To show a mutant:
    mutmut show <id>


Survived 🙁 (8)

---- src/templite.py (8) ----

10, 29, 37, 45, 46, 58, 108, 152


This gives me the ids of the mutants that survived, that is, the mutations that didn’t cause a failure in the test suite.

We can see the actual code mutation with the show command:

$ mutmut show 10
--- src/templite.py
+++ src/templite.py
@@ -48,7 +48,7 @@
         self.code.append(section)
         return section

-    INDENT_STEP = 4      # PEP8 says so!
+    INDENT_STEP = 5      # PEP8 says so!

     def indent(self):
         """Increase the current indent for following lines."""


The mutation is shown as a diff. The old line is prefixed with minus, and the new line with plus. Here the INDENT_STEP constant was changed from 4 to 5.

Right off the bat, we have a philosophical decision to make. A bit about how templite works: it converts template files into Python code. Rendering a template is done by executing the generated Python code. This INDENT_STEP constant is the indentation amount used in the generated code.

I have no tests that examine the generated code. That code is an implementation detail. The important thing is that the templates render properly, so that is what’s tested. When mutmut changed the indent level to 5, the generated code was different, but only in white space, so it ran the same, and still produced the right output.

Does this mutation point to a problem in the test suite? I don’t think I should test that the indentation level in the generated code is 4 spaces. Mutmut provides a way to mark the line to exempt it from mutation, but I’m not sure I want to start adding those pragmas. This is one of the things I wanted to understand: what kind of false positives would appear, and how would I deal with them?

Let’s see how the next mutant looks:

$ mutmut show 29
--- src/templite.py
+++ src/templite.py
@@ -134,7 +134,7 @@
         code.add_line("append_result = result.append")
         code.add_line("extend_result = result.extend")
         if sys.version_info.major == 2:
-            code.add_line("to_str = unicode")
+            code.add_line("XXto_str = unicodeXX")
         else:
             code.add_line("to_str = str")



The second mutant has found the one line of code that is not covered by the test suite, because it’s for Python 2, and we are only running under Python 3. Mutmut has a --use-coverage flag, which uses coverage data to skip mutations on lines that are not covered by the test suite. If I had used it to begin with, this mutant wouldn’t have appeared. Nice.

Next:

 mutmut show 37
--- src/templite.py
+++ src/templite.py
@@ -144,7 +144,7 @@
             """Force `buffered` to the code builder."""
             if len(buffered) == 1:
                 code.add_line("append_result(%s)" % buffered[0])
-            elif len(buffered) > 1:
+            elif len(buffered) >= 1:
                 code.add_line("extend_result([%s])" % ", ".join(buffered))
             del buffered[:]



This is a classic false positive. The condition has been changed from greater to greater-or-equal, but it doesn’t change the behavior of the code. This mutation is in an “elif” clause and the equal case was already handled by the previous if clause, so greater-or-equal is the same as greater.

On this point, Anders commented:

Mutmut here does point out that your code is overly complex. Just “elif buffered” can’t be mutated but has the same functionality. I’ve found this to be a weird little side effect to using mutation testing. If I follow this the code gets better and more “just so”. This specific case isn’t a super strong argument, but I’ve had many similar things that build on top of each other in small increments.

I can see Anders’ point here, though I’m not sure I want to change the code that way.

Mutant 45 gives us our first true success:

$ mutmut show 45
--- src/templite.py
+++ src/templite.py
@@ -153,7 +153,7 @@
         # Split the text to form a list of tokens.
         tokens = re.split(r"(?s)({{.*?}}|{%.*?%}|{#.*?#})", text)

-        squash = False
+        squash = True

         for token in tokens:
             if token.startswith('{'):


Templite can squash white space around tokens, and here we are changing the initial value of the “should I squash white space?” flag. How can it not cause a test failure? Because we never tested a template that started with white space! Adding this simple test kills the mutant:

self.try_render("  hello  ", {}, "  hello  ")


I thought that mutmut run again would clear the mutant from the results, but the only way I could find to clear it was to delete the mutmut cache and run all the mutations again. [Anders wrote an issue about this.]

Mutant 46 is another false positive:

$ mutmut show 46
--- src/templite.py
+++ src/templite.py
@@ -153,7 +153,7 @@
         # Split the text to form a list of tokens.
         tokens = re.split(r"(?s)({{.*?}}|{%.*?%}|{#.*?#})", text)

-        squash = False
+        squash = None

         for token in tokens:
             if token.startswith('{'):


Here squash is the same boolean flag we saw in mutant 45. I only ever check it with if squash:, so of course False and None produce the same results. Notice here if I wanted to prevent this mutant by adding a pragma to the line, I would also have prevented the first success we had. Adding that pragma would be counter-productive.

Next:

$ mutmut show 58
--- src/templite.py
+++ src/templite.py
@@ -160,7 +160,7 @@
                 start, end = 2, -2
                 squash = (token[-3] == '-')
                 if squash:
-                    end = -3
+                    end = -4

                 if token.startswith('{#'):
                     # Comment: ignore it and move on.


This is another useful result. Turns out in my tests, I always wrote space-squashing tags with a space, like {{a -}}. This mutated code adjusted the trimming of punctuation to account for the dash. Because I always had a space before the dash, the change to -4 went unnoticed. I killed this mutant by changing some tags in my tests to have no space: {{a-}}, and also added some with many spaces for good measure.

Mutant 108 sure looks like it’s real:

$ mutmut show 108
--- src/templite.py
+++ src/templite.py
@@ -211,7 +211,7 @@
             else:
                 # Literal content.  If it isn't empty, output it.
                 if squash:
-                    token = token.lstrip()
+                    token = None
                 if token:
                     buffered.append(repr(token))


Seems like we have no tests of non-white-space literal content after a squashing tag. Add that test, and that mutant is killed.

Our last mutant is another interesting case:

$ mutmut show 152
--- src/templite.py
+++ src/templite.py
@@ -283,7 +283,7 @@
                     value = value[dot]
                 except (TypeError, KeyError):
                     raise TempliteValueError(
-                        "Couldn't evaluate %r.%s" % (value, dot)
+                        "XXCouldn't evaluate %r.%sXX" % (value, dot)
                     )
             if callable(value):
                 value = value()


Here the error message has been mutated by adding chaff to the beginning and end. We do have a test for this error, including its message:

def test_exception_during_evaluation(self):
    msg = "Couldn't evaluate None.bar"
    with self.assertRaisesRegex(TempliteValueError, msg):
        self.try_render(
            "Hey {{foo.bar.baz}} there", {'foo': None}, "Hey ??? there"
        )


The test still passes because it’s finding the expected error message somewhere in the actual error message. If mutmut had added chaff in the middle of the string as well, it would have failed the test. Is this clever of mutmut? Hard to say!

When I change the test, the mutant is killed:

regex = "^Couldn't evaluate None.bar$"
with self.assertRaisesRegex(TempliteValueError, regex):


BTW, the first time I ran mutmut, it created another nonsensical mutant:

--- src/__init__.py
+++ src/__init__.py
@@ -1,2 +1,2 @@
-from .templite import *
+from .templite import /


This mutant survived because this file was never executed. That in itself was a useful clue to the fact that I had made a useless file. Delete the file, and the mutant is killed. [mutmut has changed so that it won’t create this mutation any more.]

So after all this, how did mutmut do? It gave me seven mutations, four of which resulted in improving the tests. That’s not a bad outcome. But I don’t know how I would use this regularly. I don’t have a good way to silence the three false positives, so if I run mutmut again in the future, I will have to consider them again.

As another data point about the cost of mutation testing, I tried mutmut on another project with a 10-second test suite. It took mutmut 43 minutes to run 513 mutants, of which 165 survived. I haven’t looked through them yet to see what they mean.

All in all, I am pleased with the results. As an occasional (but expensive) way to assess your test suite, mutmut works well.

I waved to him and called out, “I didn’t expect to have to shovel, I’m going to be late for work.”

He came over, and we started chatting. He had a classic thick Boston accent. Was it going to be warm enough later that it would just melt? Or would it just freeze again and cause a real mess? It was the kind of friendly bonding over a shared experience that snow shoveling can bring about, even between two people without much else in common.

I ended with, “Well, I choose to live in Boston, so I can’t complain,” to which he responded, smiling, “And if you did complain, who would listen!?” It was that kind of conversation.

I went back up to the top of my driveway. I was faced away from the street, clearing the car, when I heard something behind me. I turned to see the guy in his pickup truck, zooming toward me. The blade of the snowplow is about three feet from me. He stopped, lowered the blade, and backed down the driveway, scraping away the snow. He came up again, scraped again, and then pushed the resulting snow pile in the street out of the way.

I just stood aside, grinning, pleased at the work he had saved me. As he turned the truck to drive away down the street, I joked to him, “Thanks a lot! But next time give me some warning, you almost gave me a heart attack!”

To which he smiled and responded, “Ah, you wasn’t gonna have no haht attack,” and drove away to the next job.