Finding stale pyc files

Thursday 3 October 2013

Recently I was debugging one of those “it can’t happen” kinds of problems, and wanted to make sure I didn’t have any stale .pyc files lying around. I figured the “find” command could find pairs of files whose dates compared incorrectly, but I didn’t know how to do it.

I asked in the #bash IRC channel, and they gave me this:

find . -name '*.pyc' -exec bash -c 'test "$1" -ot "${1%c}"' -- {} \; -print  #stalepyc

It’s one of those Unix-isms I wont be able to remember (yet?), so I’ll leave it here to find again when I need it later.

Notice I’ve added a bashtag to it so I can search for it in my command history. (I wish I had come up that name!).

I’m sure there are other ways to find stale files, maybe even better ones?

Comments

[gravatar]

Those pyc bugs occasionally bit me when switching from one git branch to another.

I normally just do a "find . -name '*.pyc' | xargs rm" in the root of my project since it's easy to remember & I don't care about deleting valid pyc files along with stale ones.

[gravatar]

@Randy, yes, in this case I wanted to not just fix the problem, but understand why it was happening, so if there were stale pyc files, I wanted to see them, not just clobber them.

BTW: find has -delete, so you can use: find . -name '*.pyc' -delete

[gravatar]

It took me a while to parse the syntax and understand what was going on... Pretty neat!
To save others the time:
bash -c : from the man page
-c string If the -c option is present, then commands are read from
string. If there are arguments after the string, they are
assigned to the positional parameters, starting with $0.

Therefore the command going to bash is:
test "$1" -ot "${15c}

the -- is used to signal the end of command line options and the {} \; is the results of the find command and the exec terminator.

${1} is the first command line arg passed (the found file name + path == path to a *.pyc file)
$(1%c} is the first arg with a "c" stripped off the end ( == path to a *.py file)

the test -ot asks bash to determine if the pyc file is "older than" the py file

I don't want to guess how much more bash I need to do every day before I would have thought that up on my own ;-)

[gravatar]

IIRC the .pyc file records the timestamp of the corresponding .py, and Python won't use it if the timestamp doesn't match.

Stale pyc files cause trouble when the original .py is gone.

[gravatar]

@Marius: you are right. I may have been in a "grasping at straws" mood! It would be interesting to see the find command that finds .pyc files with no corresponding .py file.

[gravatar]

> It would be interesting to see the find command that finds .pyc files with no corresponding .py file.

find . -name '*.pyc' -exec bash -c 'test ! -e "${1%c}"' -- {} \; -print

[gravatar]

nooooo you should use 'sh' not 'bash'. Remember that many systems may not have bash on them at all!

[gravatar]

I have been using this to find orphan .pyc or .pyo files:

import os
for pat, d, fns in os.walk('.'):
    for fn in fns:
        if fn[-4:] in ('.pyc', '.pyo'):
            pyfn = os.path.join(pat, fn[:-1])
            if pat.endswith('__pycache__'):
                pyfn = os.path.join(os.path.split(pat)[0], fn.split('.', 1)[0] + '.py')
            if not os.path.exists(pyfn):
                print 'Orphan: %s/%s' % (pat, fn)

[gravatar]

A good trick for development (esp. when switching between branches often) is to set the PYTHONDONTWRITEBYTECODE environment var and all .pyc related problems are magically gone :)

Useful mostly for development systems.

[gravatar]

I've set a small bash function in my .bashrc to quickly nuke 'em all from orbit (it's the only way to be sure...)

pyc() {
    find . \( -iname \*\.pyc -o -iname \*\.pyo \) -print0 | xargs -0 rm -f
    find . -name __pycache__ -print0 | xargs --no-run-if-empty -0 rmdir
}

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.