Two points... first, you spelled my name wrong. ;)

Second, I think it's possible you might be missing a point about unit testing vs. documentation, or perhaps just about "unit" tests vs. "integration" tests. Essentially, the "integration" tests are usually much more useful as developer documentation examples. However, the unit tests are useful as internal documentation. For example, as a specification of what the code is supposed to do.

Does this take some balancing? Yes. For example, my first doctest is a little awkward in spots. However, it's better documentation than I otherwise would have bothered writing for the module, as well as a better test suite than I had with unittest.

So, is it great user documentation? Probably not. Is it useful documentation? Definitely yes, because it has more examples, and it's "safe" to put lots of them in without worrying about them breaking later.

As for the "not all methods can be well documented by showing execution examples" thing, I'm not sure what you mean, so I can't speak to that.

Point 1: I fixed it. That one-L/two-L thing must drive you nuts.

Point 2: It's entirely possible that I'm thinking old school about all of this stuff. And you may well be right that a slew of unit tests is better documentation than I would have had without doctest.

I'll keep an open mind about all this stuff. The sad fact is that I can't really investigate all the alternatives, because beyond reading about (and understanding) all the options, I'd have to actually use them on significant projects. I don't have that many significant projects, and when I do have one, I want to focus on the project, not an objective evaluation of the unit testing methodology.

So it's difficult to evaluate the choices. I value the reviews I get from others, but I also have opinions that I like to share.

I use doctest a fair amount outside of docstrings. You can put them in a dictionary called __test__, and then there's no expectation that it's documentation. It lets you write tests that are very expressions with expected output, which is really simple for some kinds of code, and doesn't require any fixtures or anything else that takes extra time to write.

Just to clarify... I haven't used doctest inside of docstrings at all; I'm writing entirely seperate .txt files containing the tests, and then using doctest.DocFileSuite() to add them to my existing unittest packages. I personally don't care to put them inside of docstrings, because I already write really long docstrings. With a separate file, I can "tell a story" about the code, that wouldn't work if it was embedded inside the code.

Maybe that's the thing you meant about methods not being well-documented by examples... sometimes you need context or narrative, which is easy with a standalone doctest file. And, since I'm using reStructuredText for the separate file, I can also turn it into a nice article on the innards of a particular module.

Anyway, I suppose I am raving a bit about it; it's just so exciting to realize that no, unit tests aren't so hard, it's just unittest that's hard. Will I have to "design for doctest"? Yeah, sure. But I also designed stuff around unittest. That's the point of doing TDD, to let the nature of testing make your code better. I'm just glad it can be easier now, while at the same time helping me make sure I have at least *some* documentation.

It's especially helpful, I think, for documenting the extensibility aspects of a framework, that somebody might want to plug into or subclass or extend. Also, for documenting a project's internals. Some time in the next week, I'll hopefully begin finding out whether it works well for just documenting basic APIs.

Someday, it might be really cool to have an IDE that would run the doctests in a text file as you edit them, and highlight the errors and differences inline. That would be out of this world.

"To me, this is an admission of failure"

To me, that's an amazingly ignorant comment.

When using doctest for unit testing, the idea is to test that the *test program* works as documented. You can write doctest tests many times faster than you can write traditional code-only unit tests (the main advantage is that you don't always have to figure out what's the expected output should be; just write the test, run it, and verify the output; you hardly ever have to debug your tests.).

If that's not how testing is "supposed to be", you have a problem. Not doctest.

Geez, Fredrik, do you always have to be so nasty to people? Didn't you read his very next sentence: "Maybe I'm being too harsh and should give it a try."

Your writing almost always contains useful information, but I'm tired of trying to read around the rudeness and hair-trigger judgments. Perhaps before you hit send, you could go back over your posts and delete the parts where you call people names (either directly or by implication). They don't really add anything, and they discourage people from paying attention to what you actually have to say.

Whoa, why can't we all just get along? :-)

In Fredrik's defense, my comment was an ignorant one. I'm learning something about doctest from these comments: that the focus is not so much on having a function's docstring be its test, as it is to have tests that read like docstrings, regardless of where they live in the source tree.

Of course, Fredrik could have said it more nicely...

You can also look at the nice and light docex:
http://aima.cs.berkeley.edu/python/docex.py
(The useful doctest.ELLIPSIS option can be added)

Using unittest and doctest together is fun. They compliment each other well:

'doctest' tests are the more readable ones, and they belong in the docstring

'unittest' tests are the tests left over, that usually require some non-trivial programmic structure, and these happily live off in their own separate file.

And all these tests are definitely part of the documentation. Before we had to use only English to describe our code. But now we can use Python itself to describe our code. Python is a great way to talk about things when you need precision.

Human_Language + Python == documentation(readable=True, complete=True)

I would suggest trying TestOOB, (http://testoob.sourceforge.net), a Python unit testing framework that extends and integrates seamlessly with unittest suites.

It provides useful features like color output, running tests in parallel (threads), filtering (regexes), reporting (XML/HTML), immediate feedback on failed assertions, firing up pdb on failed tests, and lots more.

Having used py.test, doctest and unittest, I'm not sold on unittest at all, even though we have to use it at work, which is why I ended up here. I was fairly impressed by py.test.

My main problem with it is it means every test must inherit from unittest.TestCase. This brings all the fragility of inheritance hierarchies to your tests. It's a typical 90's C++/Java way of thinking about things. A testing framework based on delegation would be more flexible, in particular it would allow easy parameterisation of tests, which would reduce the number of test needing to be written. unittest.TestCase sub-classes expect to be initialised in a certain way, and that way is not useful, transparent or explicit.

Because of this, it doesn't completely solve the problem of making test-discovery fully automatic, like with py.test. I always have to inherit from unittest.TestCase. I may well have to import those test cases from elsewhere. Seeing as I'm going to the trouble, I might as well be registering tests. In fact, registering tests is exactly what unittest.TestCase is doing in its abuse of inheritance. The Zen of python rightly states "Explicit is better than implicit". I'd much rather explicitly register tests for running e.g. using a decorator, and this would avoid the need for inheritance, as the test case class could just implement some common interface.

The other problem that few test frameworks really address is explicitly tracking what is being tested (or more importantly, what isn't). Coverage testing is useful, but doesn't show you in your code what is tested until after running the tests, and does so in some other document. When you have a big system, and big developer teams, and someone adds a class, once the tests are divorced from the code they are testing, (which arguably they should be), then you can have trouble finding tests, or worse, end up duplicating testing effort, as there is nothing that points from the code to the test case that verifies it.

For me the nice thing about doctests is that they solve this problem by living with the code, so they are easy to find. They can get cluttered though, so I do think of them as 'verified documentation'/"Here's how to use this" rather than 'tests in the documentation'/"This comes with these guarantees.". However, it is nice to have tests sitting right in the code. So, perhaps a 'best-of-both-worlds' solution would be a decorator e.g. @tested_by(test_case) before a class, which could both show what is being tested and perform explicit test registration at the same time. It would also allow quick discovery of untested code (just search a model for anything not behind one of these). In production, the decorator could be replaced with the identity function, so it doesn't slow things down much. This scheme would also allow tests to be easily reused against different classes.

I guess this last point underlines the TDD idea that tests are just programmatically enforced interfaces.

Ho hum.