Reminds me of a native code profiler that I looked at for the Notes server. The literature claimed that "it worked great on notepad.exe"

Swell.

A better starting page is this http://www.lambdacs.com/debugger/debugger.html
It includes a link to the one page description that Ned pointed to, but also some links to other stuff.

Clearly you're not going to be using it to record "everything" during a weather simulation run, but it sounds like an efficient way to debug certain classes of problems.

This actually could work on large programs. I used to work at a company where we had our own similar debugging system. It recorded everything, including tester's screen preferences. You could even debug multiuser apps! It allowed us to find 90% of those hard-to-reproduce bugs that only come out something like once a year, after a 4-hour session. With 10MB of C source code, accumulated over something like 10 years, that was a major help.

Then a new guy arrived at the company who found many of those "Oh! That must be that one that happened four years ago!" bugs. With code printouts and a pencil...

I will take a shot at trying to build an Omniscient Debugger in Python, just for fun.
It will be hosted at https://github.com/rodsenra/pode