Comments

Two points... first, you spelled my name wrong. ;)

Second, I think it's possible you might be missing a point about unit testing vs. documentation, or perhaps just about "unit" tests vs. "integration" tests. Essentially, the "integration" tests are usually much more useful as developer documentation examples. However, the unit tests are useful as internal documentation. For example, as a specification of what the code is supposed to do.

Does this take some balancing? Yes. For example, my first doctest is a little awkward in spots. However, it's better documentation than I otherwise would have bothered writing for the module, as well as a better test suite than I had with unittest.

So, is it great user documentation? Probably not. Is it useful documentation? Definitely yes, because it has more examples, and it's "safe" to put lots of them in without worrying about them breaking later.

As for the "not all methods can be well documented by showing execution examples" thing, I'm not sure what you mean, so I can't speak to that.

Point 1: I fixed it. That one-L/two-L thing must drive you nuts.

Point 2: It's entirely possible that I'm thinking old school about all of this stuff. And you may well be right that a slew of unit tests is better documentation than I would have had without doctest.

I'll keep an open mind about all this stuff. The sad fact is that I can't really investigate all the alternatives, because beyond reading about (and understanding) all the options, I'd have to actually use them on significant projects. I don't have that many significant projects, and when I do have one, I want to focus on the project, not an objective evaluation of the unit testing methodology.

So it's difficult to evaluate the choices. I value the reviews I get from others, but I also have opinions that I like to share.

I use doctest a fair amount outside of docstrings. You can put them in a dictionary called __test__, and then there's no expectation that it's documentation. It lets you write tests that are very expressions with expected output, which is really simple for some kinds of code, and doesn't require any fixtures or anything else that takes extra time to write.

Just to clarify... I haven't used doctest inside of docstrings at all; I'm writing entirely seperate .txt files containing the tests, and then using doctest.DocFileSuite() to add them to my existing unittest packages. I personally don't care to put them inside of docstrings, because I already write really long docstrings. With a separate file, I can "tell a story" about the code, that wouldn't work if it was embedded inside the code.

Maybe that's the thing you meant about methods not being well-documented by examples... sometimes you need context or narrative, which is easy with a standalone doctest file. And, since I'm using reStructuredText for the separate file, I can also turn it into a nice article on the innards of a particular module.

Anyway, I suppose I am raving a bit about it; it's just so exciting to realize that no, unit tests aren't so hard, it's just unittest that's hard. Will I have to "design for doctest"? Yeah, sure. But I also designed stuff around unittest. That's the point of doing TDD, to let the nature of testing make your code better. I'm just glad it can be easier now, while at the same time helping me make sure I have at least *some* documentation.

It's especially helpful, I think, for documenting the extensibility aspects of a framework, that somebody might want to plug into or subclass or extend. Also, for documenting a project's internals. Some time in the next week, I'll hopefully begin finding out whether it works well for just documenting basic APIs.

Someday, it might be really cool to have an IDE that would run the doctests in a text file as you edit them, and highlight the errors and differences inline. That would be out of this world.

"To me, this is an admission of failure"

To me, that's an amazingly ignorant comment.

When using doctest for unit testing, the idea is to test that the *test program* works as documented. You can write doctest tests many times faster than you can write traditional code-only unit tests (the main advantage is that you don't always have to figure out what's the expected output should be; just write the test, run it, and verify the output; you hardly ever have to debug your tests.).

If that's not how testing is "supposed to be", you have a problem. Not doctest.

Geez, Fredrik, do you always have to be so nasty to people? Didn't you read his very next sentence: "Maybe I'm being too harsh and should give it a try."

Your writing almost always contains useful information, but I'm tired of trying to read around the rudeness and hair-trigger judgments. Perhaps before you hit send, you could go back over your posts and delete the parts where you call people names (either directly or by implication). They don't really add anything, and they discourage people from paying attention to what you actually have to say.

Whoa, why can't we all just get along? :-)

In Fredrik's defense, my comment was an ignorant one. I'm learning something about doctest from these comments: that the focus is not so much on having a function's docstring be its test, as it is to have tests that read like docstrings, regardless of where they live in the source tree.

Of course, Fredrik could have said it more nicely...

You can also look at the nice and light docex:
http://aima.cs.berkeley.edu/python/docex.py
(The useful doctest.ELLIPSIS option can be added)

Using unittest and doctest together is fun. They compliment each other well:

'doctest' tests are the more readable ones, and they belong in the docstring

'unittest' tests are the tests left over, that usually require some non-trivial programmic structure, and these happily live off in their own separate file.

And all these tests are definitely part of the documentation. Before we had to use only English to describe our code. But now we can use Python itself to describe our code. Python is a great way to talk about things when you need precision.

Human_Language + Python == documentation(readable=True, complete=True)

I would suggest trying TestOOB, (http://testoob.sourceforge.net), a Python unit testing framework that extends and integrates seamlessly with unittest suites.

It provides useful features like color output, running tests in parallel (threads), filtering (regexes), reporting (XML/HTML), immediate feedback on failed assertions, firing up pdb on failed tests, and lots more.