assertRaisesMsg

Tuesday 5 September 2006

I use the standard unittest module for testing, despite the community’s general protestations about it not being pythonic. It does what I need, except for one thing. It has an assertRaises method that tests that a callable raises an exception of the expected class, but the message that comes with the exception is not checked. So I usually mix this method into my test case base class:

def assertRaisesMsg(self, excClass, msg, callableObj, *args, **kwargs):
    """ Just like unittest.TestCase.assertRaises,
        but checks that the message is right too.
    """
    try:
        callableObj(*args, **kwargs)
    except excClass, exc:
        excMsg = str(exc)
        if not msg:
            # No message provided: any message is fine.
            return
        elif excMsg == msg:
            # Message provided, and we got the right message: it passes.
            return
        else:
            # Message provided, and it didn't match: fail!
            raise self.failureException(
                "Right exception, wrong message: got '%s' expected '%s'" % 
                (excMsg, msg)
                )
    else:
        if hasattr(excClass, '__name__'):
            excName = excClass.__name__
        else:
            excName = str(excClass)
        raise self.failureException(
            "Expected to raise %s, didn't get an exception at all" % 
            excName
            )

I can call it (for example) like this:

self.assertRaisesMsg(MyException, "Exception message", my_function, (arg1, arg2))

This test line will call my_function(arg1, arg2) and succeed if it raises a MyException with the message “Exception message”. It will fail in all other cases.

Comments

[gravatar]
Fuzzyman 7:18 AM on 5 Sep 2006

This looks like a good test case.

We use unittest at work. We've customised it a bit, but it's very easy to add new tests. I think unittest gets more flak than it deserves. :-)

[gravatar]
Michael Chermside 10:01 AM on 5 Sep 2006

While I have no problem with using this class, I think that in 95%+ of the cases I would NOT want to test the message. The exact text of error messages is rarely something that I want to enforce (via tests). What I did once when I was extending a testing framework (in another language, long ago) was to have the test pass if the string provided by the test appeared as a SUBSTRING someplace within the exception message. That way I could do a test that (for instance) verified that the filename appeared someplace within the FileNotFoundException without hard-coding (in the tests) the exact wording of the message.

[gravatar]
Ned Batchelder 11:35 AM on 5 Sep 2006

Michael: you are right, substring is a more useful implementation. If I want to check the whole message, substring works fine, and if I'm only interested in part, it also works fine.

[gravatar]
Trent Mick 11:58 AM on 5 Sep 2006

Ned,

I added an "assertRaisesEx" to the Python Cookbook a while back that adds "(1) the ability to assert the raised exception's args; (2) the ability to test that the stringified exception matches a given regular expression; and (3) much better failure messages."

Cheers

[gravatar]
Trent Mick 11:58 AM on 5 Sep 2006

Ack, I suppose the actual link would help:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307970

[gravatar]
Calvin Spealman 12:09 PM on 5 Sep 2006

I will definately be pulling this! I have lots of cases for using it, and it illustrates a weak point of the python exceptions that no one seems to agree on: they are not descriptive enough!

I want an AttributeError to have obj and name attributes, so I know what name was not found on what object. I want to see exactly what caused the error without looking at a message, so I can handle it properly.

[gravatar]
Ned Batchelder 12:30 PM on 5 Sep 2006

No sooner am I on-board with substrings, when Mick ups the stakes (and the value) by going to regexes!

[gravatar]
Glyph Lefkowitz 3:45 PM on 5 Sep 2006

What about internationalization?

[gravatar]
Eduardo 3:45 PM on 5 Sep 2006

This is ease to do with Py.test
http://codespeak.net/py/current/doc/test.html#how-to-write-assertions-about-exceptions

Something like:
assert py.test.raises(Exception, func, *args, **kwargs).value.args == msg

[gravatar]
Michael Chermside 8:15 PM on 5 Sep 2006

Good point Mick... regexes are indeed better than substrings. As to Glyph's point, I would argue that you should NOT be using internationalization in your exception messages. Internationalization is reserved for user-displayed strings, NOT for program internals. And exception message contents shouldn't be user-visible content.

[gravatar]
Glyph Lefkowitz 12:22 AM on 6 Sep 2006

Michael: I believe there are arguments for both sides in that debate, both internationalizing and englishifying exception messages. However, it's rather academic, as Python has an established convention, which would be difficult to ignore in real-world tests:

>>> import locale
>>> import os
>>> def reallysetlocale(loc):
os.environ['LANG'] = loc
os.environ['LANGUAGE'] = loc
locale.setlocale(locale.LC_ALL, loc)

>>> file("test", "r")
Traceback (most recent call last):
File "", line 1, in ?
IOError: [Errno 2] No such file or directory: 'test'
>>> reallysetlocale('fr_FR.UTF-8')
>>> file("test", "r")
Traceback (most recent call last):
File "", line 1, in ?
IOError: [Errno 2] Aucun fichier ou r������©pertoire de ce type: 'test'

By the way, I don't have an answer to this problem. Personally I've been ignoring internationalization for some time, and I suspect that one day it's going to bite me really hard.

Unfortunately most software ignores more than just that - internationalization and encoding rules. Just look at what this blog software did to my error message! :)

[gravatar]
Ned Batchelder 6:34 AM on 6 Sep 2006

(Sorry, Glyph. This blog code isn't much good at I18N stuff)

Thanks, everyone! This has been a richer discussion than I anticipated, and has moved on to the next blog post...

[gravatar]
Calvin Spealman 10:25 AM on 6 Sep 2006

I just wanted to point out that if exceptions gave more information in a programmably accessable way, as I mentioned previously, the I18N problem goes away pretty quickly.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.