« | » Main « | »

Coverage.py 3.6b3

Saturday 29 December 2012

So that happened... I had just posted 3.6b2 earlier this week, and was quickly informed that it was terribly broken. The process works!

The very popular nose coverage plugin uses the coverage.py API in a way that I wasn't testing, shame on me! Now I have a test of that style, to prevent this in the future.

The good news is that coverage.py 3.6b3 fixes the problem. I'm not planning any more changes for v3.6. Try it and let me know if you find any problems.

Thanks, and sorry for the version-churn.

Coverage.py 3.6b2

Sunday 23 December 2012

The second beta version of coverage.py 3.6 is ready. There are only a few changes:

  • Coverage.py runs on Python 2.3 and 2.4 again. It was broken in 3.6b1.
  • The C extension is optionally compiled using a different more widely-used technique, taking another stab at fixing issue 80 once and for all.
  • Combining data files would create entries for phantom files if used with source and path aliases. It no longer does.
  • debug sys now shows the configuration file path that was read.
  • If an oddly-behaved package claims that code came from an empty-string filename, coverage.py no longer associates it with the directory name, fixing issue 221.

I'm not planning any more changes for v3.6. Try this and let me know if you find any problems.

NotesPeek sprites

Saturday 22 December 2012

Warning: what follows is me reminiscing about stuff I found in a box in my attic (not really, metaphorically). You don't have to read it. I enjoyed going through it!

Almost 20 years ago, when I worked at Lotus, I wrote a tool called NotesPeek, for peering into Notes databases. It ran on Windows, and was modelled on the then-new Windows Explorer: a tree control on the left, and a pane on the right for details. More description and screenshots are at the DeveloperWorks page about NotesPeek.

One of the appealing aspects of writing NotesPeek was making the pixel art. The tree control had a different icon for each different kind of thing in the tree. Notes databases consisted of many different kinds of records, and each one got its own icon. I had to design (or steal) each of them.

NotesPeek ran on Windows, and at the time, 16-color displays were typical, so most artwork used the Windows 16-color palette as a lowest common denominator. This meant using hand-dithering to get interesting effects. Light and dark pixels would have to suggest features that couldn't be actually drawn in such tight spaces.

Showing these icons now is a tricky matter, because pixels are much smaller than they were then, and LCD screens are crisper than the CRTs then. To see them as they were then, try magnifying your browser window a bit, which will give you both larger pixels and a bit of smearing.

The Notes icon then was three "bowling pin" people on a yellow plinth, and the NotesPeek icon was a riff on that: an x-ray machine showing the insides of the people:

Notes icons, original size

Notes icons, 8× magnified

The x-ray screen worked really well, and I liked the subtle tie-in of the skeletons with the tree control at the core of the UI. The bowling-pin person would show up in a few of my icons to represent "Notes."

The tree-control icons were smaller, only 16×16 pixels, which meant there weren't many pixels to play with. It was fun to try to come up with a graphical representation of a concept, and then to try to make it look good with only 16×16 pixels and 16 colors. I didn't always succeed, and luckily this was just a free side-project developer's tool, so if the icons were clunky, tough luck. But a few of them I liked.

original size

4× magnified

Here the last two icons in the top row are a database, and a template for a database. The last icon in the second row represented "blob," and has some nice shading effects. In icons with overlapping objects, there's a gray pixel in the black outline to keep them from merging together.

original size

4× magnified

These represent the various access levels. I lifted them from the product itself, so they look better than most. I drew the drama mask at the end to represent "Role."

original size

4× magnified

Various kinds of documents. Notice the second-to-last icon has a bowling-pin guy rendered only 3 pixels wide.

original size

4× magnified

These are types of fields. The second and third icons in the third row please me still: Signature and Seal, and the next one, Binary, I also like.

original size

4× magnified

About now is when the icons start getting out of hand. Rich text in a Notes database is stored as Compound Document Records, and there are many kinds of CD Record, each with their own icon. In this menagerie you can find OS icons (Windows, OS/2, and Mac), a palette, a red-to-blue link used in a few ways and plenty of other stuff, I didn't even know at the time what they all meant.

original size

4× magnified

original size

4× magnified

Forbidden documents? Random stuff.

original size

4× magnified

Some developer things, including two different representations of "Java."

original size

4× magnified

Another burst of CD Record icons, including international things, frameset pieces, a color wheel, and another blobby thing.

original size

4× magnified

original size

4× magnified

The last few miscellaneous icons. Whew!

There's one last bit of art in NotesPeek, an icon of my face for the easter egg:

My face, original size

I didn't draw this, I made it from a photo, of course. I include it only because I am amazed at how little it looks like my, or any other, face when magnified:

My face, 8× magnified

There are no features on there! But when viewed actual size, I am clearly recognizable. Pixel art indeed. Spooky.

Implementing rpartition

Thursday 20 December 2012

One of my crazy decisions about coverage.py is to keep it running on Python 2.3 and up. One of the reasons this is difficult is that I've switched to using tox to test, and tox only supports >=2.5, so actually testing on 2.3 has fallen behind.

It's also really easy to forget what's not allowed in 2.3. During the bug-fixing frenzy in the latest beta, I accidentally broke 2.3 support in a few ways: combined try/except/finally, imports with parentheses, and rpartition.

For the first two, I just had to change the statements to use the old style, but I didn't want to give up rpartition, it's too useful. So I implemented it myself.

My first thought was to implement it in terms of partition, by reversing everything, using partition, then reversing everything back. It's one of the oddest-looking functions I've written:

def rpartition(s, sep):
    """rpartition in terms of partition!"""
    a, b, c = s[::-1].partition(sep[::-1])
    return c[::-1], b[::-1], a[::-1]

Too bad I couldn't have gotten one more [::-1] in there to reverse the return tuple, but that would just be weird for the sake of it...

Unfortunately, after I got this working, I discovered that partition and rpartition appeared at the same time, so implementing one in terms of the other wouldn't help me with backward compatibility. So I had to use a much less interesting implementation:

def rpartition(s, sep):
    """Implement `s.rpartition(sep)` for old Pythons."""
    i = s.rfind(sep)
    if i == -1:
        return ('', '', s)
    else:
        return (s[:i], sep, s[i+len(sep):])

Now 2.3 is back in the house. One of these days, I'll drop it, but I like the idea of being able to provide the latest coverage.py to anyone who wants it.

Skipping C extensions

Monday 17 December 2012

Coverage.py has a C extension: the trace function is implemented in C for speed, since it's invoked for every line of your program. This works great, and makes things much faster, but not every user has a C compiler properly installed on their machine. There's also a Python implementation of the trace function, so we can make do without a compiler, but this make the installation a little tricky.

It turns out that distutils is great at attempting what you instruct it to, but not that great at letting you know what happened. In particular, building an extension can fail in a variety of ways. Separating actual build failures from other pathologies is not simple.

I had a way I was doing it, but still got bug reports from people who could not install the kit without getting tangled. Two suggestions were to do it the way SQLAlchemy did it, or the way simplejson did it. It turns out they do it the same way!

For example, compare SQLAlchemy's setup.py:

ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError)
if sys.platform == 'win32' and sys.version_info > (2, 6):
    # 2.6's distutils.msvc9compiler can raise an IOError when failing to
    # find the compiler
    ext_errors += (IOError,)

class BuildFailed(Exception):

    def __init__(self):
        self.cause = sys.exc_info()[1]  # work around py 2/3 different syntax

class ve_build_ext(build_ext):
    # This class allows C extension building to fail.

    def run(self):
        try:
            build_ext.run(self)
        except DistutilsPlatformError:
            raise BuildFailed()

    def build_extension(self, ext):
        try:
            build_ext.build_extension(self, ext)
        except ext_errors:
            raise BuildFailed()
        except ValueError:
            # this can happen on Windows 64 bit, see Python issue 7511
            if "'path'" in str(sys.exc_info()[1]):  # works with both py 2/3
                raise BuildFailed()
            raise

with simplejson's setup.py:

if sys.platform == 'win32' and sys.version_info > (2, 6):
   # 2.6's distutils.msvc9compiler can raise an IOError when failing to
   # find the compiler
   ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError,
                 IOError)
else:
   ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError)

class BuildFailed(Exception):
    pass

class ve_build_ext(build_ext):
    # This class allows C extension building to fail.

    def run(self):
        try:
            build_ext.run(self)
        except DistutilsPlatformError, x:
            raise BuildFailed()

    def build_extension(self, ext):
        try:
            build_ext.build_extension(self, ext)
        except ext_errors, x:
            raise BuildFailed()

The code is clearly copied from the same source, even the name ve_build_ext is the same, as is its comment. SQLAlchemy has a new twist based on Python bug 7511, but it's clear we're all copying and pasting the same solution.

In fact, a Google search for that comment reveals a dozen or so packages using the same technique, with varying degrees of automatic fallback.

Does anyone know where this code first came from? What does the "ve" stand for in "ve_build_ext"? Is any of the work on better packaging tackling this issue? And how do people test this code? One suggestion was mocking out distutils, but the central issue here is properly catching the diverse exceptions that might occur if the compile fails. Help!

Kim's Python lesson

Saturday 1 December 2012

I spent the afternoon on Thanksgiving teaching my 14-year-old niece Kim about programming in Python. She hadn't done any programming before, but was very interested, so I figured we could just give it a shot. I didn't have a text to follow, and hadn't known she was interested, so I hadn't prepared anything.

But the path we took seemed to work, so here's what I remember of it, in rough outline form. As you look at the code samples, remember the goal here isn't to use the best or most Pythonic way to write the code, but to use simple tools to teach beginning programming.

The text below is not for the teacher to read to the student, or for the student to read on their own. This is just of list of topics we discussed, and programs we wrote.

•    •    •

What is a computer program? Instructions for a computer to follow. Computers are really dumb, so you have to spell out everything precisely.

Install Python (IDLE doesn't work well on a Mac, we ended up with plain-old textedit and a terminal.)

Try some arithmetic in the interactive prompt. Assign values to names:

>>> 2 + 2
4
>>> 10 * 3 + 5 * 3
45
>>> x = 10
>>> 2 * x
20
>>> x = x+1
>>> x
11
>>> 2 * x
22

"x = 1" in Python is not like "x = 1" in algebra. You can't say, "x + 1 = 2" in Python, and you can't say "x = x + 1" in algebra.

Create and run a simple program:

print "Hello, world!"

You can ask for input from the user:

name = raw_input("What's your name? ")
print "Hi", name

If statements let you make decisions:

name = raw_input("What's your name? ")
print "Hi", name
age = int(raw_input("How old are you? "))
if age > 25:
    print "Wow, you are old"

An else clause lets you go either way:

name = raw_input("What's your name? ")
print "Hi", name
age = int(raw_input("How old are you? "))
if age > 25:
    print "Wow, you are old"
else:
    print "Don't worry, you're still cool"

An elif clause lets you keep trying different conditions:

name = raw_input("What's your name? ")
print "Hi", name
age = int(raw_input("How old are you? "))
if age > 25:
    print "Wow, you are old"
elif age < 6:
    print "You're just a baby!"
else:
    print "Don't worry, you're still cool"

We've used two kinds of values: numbers and text, called strings. You have to be clear which kind you have. 12 != "12".

Loops: when you want to do something over and over, you can use a while loop. It checks a condition and if the condition is true, runs the statements, then checks the condition again etc, over and over, until the condition is false.

count = 10
while count > 0:
    print "Count:", count
    count = count - 1
print "Blastoff!"

Also, we can write "count = count - 1" as "count -= 1"

Play around with that first number. Computers can do a lot of boring stuff really quickly.

Let's write a program to add up all the numbers up to 1000. You have to think like a computer, which means you can't just say, "Add them all up," you need to break it down into smaller steps.

total = 0
num = 0
while num <= 1000:
    total += num
    num += 1
print "total is", total

Now let's add a twist: let's only include the number in the sum if the number is divisible by 3 or by 5. To see if a number is divisible by 3, you can test if N % 3 == 0. "%" gives you the remainder after dividing, and "==" is how you test for equality.

This program is where we start to get opportunities for real logic errors. If they happen, make sure you talk about them and how they happened before moving on.

total = 0
num = 0
while num <= 1000:
    if num % 3 == 0:
        total += num
    elif num % 5 == 0:
        total += num
    num += 1
print "total is", total

Try a new kind of value: lists. In the interactive prompt, try this:

>>> nums = [1, 2, 3]
>>> print nums
[1, 2, 3]
>>> print len(nums)
3
>>> nums.append(17)
>>> print nums
[1, 2, 3, 17]

Lists can hold any kind of value, like text strings:

>>> cats = ["penny", "kiki", "naomi"]
>>> print cats
['penny', 'kiki', 'naomi']
>>> print len(cats)
3
>>> cats.append('arwen')
>>> print cats
['penny', 'kiki', 'naomi', 'arwen']

Let's write a program to catalog cats (Kim has 5 cats). It will ask the user for the name of a cat, and keep asking until the user types nothing, then will show how many cats the user has.

If you want to loop forever, or test a condition that you can't test immediately, you can use "while True". Remember while tests its condition and runs the loop if the condition is true. True is always true. Later, if you want to end the loop, you can use the "break" statement.

cats = []
while True:
    name = raw_input("Tell me the name of a cat (just plain enter to end): ")
    if name == "":
        break
    cats.append(name)
print "You have", len(cats), "cats"

Hmm, it looks kind of funny if you only have one cat, because it says, "You have 1 cats". Fix that.

cats = []
while True:
    name = raw_input("Tell me the name of a cat (just plain enter to end): ")
    if name == "":
        break
    cats.append(name)
if len(cats) == 1:
    print "You have 1 cat"
else:
    print "You have", len(cats), "cats"

or:

# ...
if len(cats) == 1:
    group = "cat"
else:
    group = "cats"
print "You have", len(cats), group

We'll add one more feature to our cat cataloguer: print the names of the cats. There's another way to loop in Python beside "while". A "for" loop will go around the loop once for each element in a list. We can use it to print the names of the cats:

#.. tack this on the end of the cat cataloguer from above..
print "Here are their names:"
for cat in cats:
    print cat

Now we make a mighty leap to madlibs, which isn't that much more complicated, but needs a few more new concepts.

Play in the interactive prompt with building a string by adding strings together:

>>> s = ""
>>> s = s + "Hello"
>>> s += " world"
>>> print s
'Hello world'
>>> s += " I have something"
>>> s += " to say."
>>> print s
'Hello world I have something to say.'

If we have a list of things, we can examine individual elements in the list:

>>> cats = ["penny", "kiki", "naomi"]
>>> print cats[0]
'penny'
>>> print cats[1]
'kiki'
>>> print cats[2]
'naomi'

To do madlibs, the story will be represented as a list of pieces, where each piece can be a literal piece of text in the story, or a slot to be filled in. A piece will be a 2-element list: for text, the first element will be 'text', and the second element will be the actual text to go in the story. For a slot, the first element will be 'slot', and the second element will be the prompt for the user. The madlib will be a list of these little lists:

madlib = [
    ['text', "Once upon a time there was a "],
    ['slot', "an adjective"],
    ['text', ", "],
    ['slot', "another adjective"],
    ['text', " "],
    ['slot', "an animal"],
    ['text', ". He liked to "],
    ['slot', "a verb in the present tense"],
    ['text', " all day. One day, he went to "],
    ['slot', "an adjective"],
    ['text', " "],
    ['slot', "a place"],
    ['text', " to meet "],
    ['slot', "a person"],
    ['text', "."],
    ]

When writing a madlib this way, it's really hard to get the spacing and punctuation right...

The madlib program will scan the madlib. Each 'text' piece will be added to the story we're building. For each 'slot' piece, we'll prompt the user to give us someething to go in the slot, and we'll add their answer to the story. When it's all done, we'll print the story.

madlib = [
    #.. from above ..
    ]

story = ""

for piece in madlib:
    if piece[0] == 'text':
        story += piece[1]
    elif piece[0] == 'slot':
        prompt = "Give me " + piece[1] + ": "
        answer = raw_input(prompt)
        story += answer

print story

•    •    •

At this point, the student has a program they can actually enjoy. If all has gone well, they've probably wanted to show off each program to nearby admirers (typically parents), but the madlibs program is the crowning acheivement.

By the way, the madlibs exercise is one I first did with my son Max when he was 13, and wrote about then, with more code and other ideas about how to expand it.

Working with Kim on this was really interesting. There were some parts where I was astonished that she understood so quickly, and then others where something she had understood in the previous program needed to be re-explained. Programming is a really foreign environment, and it's hard for those of us who are fully steeped in it all day to realize just how foreign.

« | » Main « | »