Decorator shortcuts

Saturday 8 October 2022

When using many decorators in code, there’s a shortcut you can use if you find yourself repeating them. They can be assigned to a variable just like any other Python expression.

Don’t worry if you don’t understand how decorators work under the hood. A decorator is a line like this in your code, usually modifying how a function behaves:

@something(option1, option2)
def my_function(arg1, arg2):
    ... # etc

For this example, it doesn’t really matter what the “something” decorator does. The important thing to know is that everything after the @ sign is a Python expression that is evaluated to get an object that will be applied to the function.

As with other Python expressions, you can give that object a name, and use it later. This produces the same effect:

modifier = something(option1, option2)

@modifier
def my_function(arg1, arg2):
    ... # etc

In this case we haven’t gained much. But let me show you a real example. In the coverage.py test suite, there are unusual conditions that cause tests to fail, and I want to tell pytest that I expect them to fail in those situations. Pytest has a decorator called “pytest.mark.xfail” that can be used to do this.

Here’s a real example:

@pytest.mark.xfail(
    env.PYVERSION[:2] == (3, 8) and env.PYPY and env.PYPYVERSION >= (7, 3, 10),
    reason="Avoid a PyPy bug: https://foss.heptapod.net/pypy/pypy/-/issues/3749",
)
def test_something():
    ...

(Yes, it’s a bit crazy, but a bug in PyPy 3.8 version 7.3.10 or greater causes some of my tests to fail. Coverage.py tries to closely follow small differences between implementations, so it’s not unusual to have to excuse a test that doesn’t work in very specific circumstances.)

The real problem though was that eleven tests failed in this situation. I didn’t want to copy those four lines into three different test files and explicitly decorate eleven tests. So I defined a shortcut in a helper file:

xfail_pypy_3749 = pytest.mark.xfail(
    env.PYVERSION[:2] == (3, 8) and env.PYPY and env.PYPYVERSION >= (7, 3, 10),
    reason="Avoid a PyPy bug: https://foss.heptapod.net/pypy/pypy/-/issues/3749",
)

Then in the test files, I can do this:

from tests.helpers import xfail_pypy_3749

@xfail_pypy_3749
def test_something():
    ...

@xfail_pypy_3749
def test_something_else():
    ...

Now I have a compact notation to apply to affected tests, and I can add as much detail to the definition because it’s only in one place instead of being copied everywhere.

There could be advanced cases where the decorator function needs to be explicitly called for each function, and a shortcut wouldn’t work right, but to be honest I’m not sure what those would be!

Comments

[gravatar]

If something(option1, option2) creates a single-use object that’s used via a closure inside the function it returns, calling that returned function more than once could definitely cause bugs. I suppose that sort of thing should be documented–but it’s kind of obscure and it might not occur to the library author to mention it.

Perhaps it’d be safer to write your own wrapper function for the decorator, rather than caching the result:

def xfail_pypy_3749():
    return pytest.mark.xfail( ... )

@xfail_pypy_3749()
def test_something():
    ...

This gets you the benefit of not having to repeat yourself with the gnarly decorator invocation, combined with calling the library the way it expected to be called.

[gravatar]

I can get Larry’s code to work if I remember to use the decorator as @xfail_pypy_3749().

If I forget the () and just use @xfail_pypy_3749, then I get an error: TypeError: xfail_pypy_3749() takes 0 positional arguments but 1 was given.

Also something to think about.

Regardless. Even though I have no reason to doubt Larry in the general case, in the SPECIFIC case of pytest.mark.xfail or pytest.mark.skip or really any pytest decorator, I can’t think of an example that fits Larry’s concerns.

Therefore, I’m going to stick with recommending Ned’s version for pytest decorators.

[gravatar]

Also, Ned’s version works with or without the parentheses.

So both of these work with Ned’s version:

@xfail_pypy_3749
def test_something():
    ...

@xfail_pypy_3749()
def test_something_else():
    ...
[gravatar]

One more thought, and hopefully I’m done.
For checking which Python version you are running, there are lots of options.

Ned’s code refers to env.PYVERSION. This env is part of coverage.py.

For non “test code for coverage.py”, I can have a similar effect with sys.version_info:

import sys

xfail_py_le_39 = pytest.mark.xfail(
  sys.version_info <= (3, 9),
  reason="fails on legacy Python"
)

@xfail_py_le_39
def test_fail():
  ...

@xfail_py_le_39
def test_fail2():
  ...

[gravatar]

If, instead of :

def xfail_pypy_3749():
    return pytest.mark.xfail( ... )

you did:

def xfail_pypy_3749(func):
    return pytest.mark.xfail( ... )(func)

(note the addition of func twice) You can now use the new decorator without parentheses.

@xfail_pypy_3749
def test_something():
    ...

I did not, however, make it optional.

[gravatar]

Brilliant! What a type-saver!

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.