![]() | Ned Batchelder : Blog | Code | Text | Site Cog » Home : Code |
Created 10 February 2004, last updated 12 April 2011 This document is also available in Russian. Cog is a code generation tool. It lets you use pieces of Python code as generators in your source files to generate whatever code you need. The sections below are:
What does it do?Cog transforms files in a very simple way: it finds chunks of Python code embedded in them, executes the Python code, and inserts its output back into the original file. The file can contain whatever text you like around the Python code. It will usually be source code. For example, if you run this file through cog: // This is my C++ file. it will come out like this: // This is my C++ file. Lines with triple square brackets are delimiter lines. The lines between [[[cog and ]]] are the generator Python code. The lines between ]]] and [[[end]]] are the output from the generator. When cog runs, it discards the last generated Python output, executes the generator Python code, and writes its generated output into the file. All text lines outside of the special markers are passed through unchanged. The cog marker lines can contain any text in addition to the triple square bracket tokens. This makes it possible to hide the generator Python code from the source file. In the sample above, the entire chunk of Python code is a C++ comment, so the Python code can be left in place while the file is treated as C++ code. DesignCog is designed to be easy to run. It writes its results back into the original file while retaining the code it executed. This means cog can be run any number of times on the same file. Rather than have a source generator file, and a separate output file, typically cog is run with one file serving as both generator and output. Because the marker lines accommodate any language syntax, the markers can hide the cog Python code from the source file. This means cog files can be checked into source control without worrying about keeping the source files separate from the output files, without modifying build procedures, and so on. I experimented with using a templating engine for generating code, and found myself constantly struggling with white space in the generated output, and mentally converting from the Python code I could imagine, into its templating equivalent. The advantages of a templating system (that most of the code could be entered literally) were lost as the code generation tasks became more complex, and the generation process needed more logic. Cog lets you use the full power of Python for code generation, without a templating system dumbing down your tools for you. InstallationCog requires Python 2.3 or later, or Jython 2.5. Python 3.x is not supported. Depending on the extent of your use, you may also need these packages:
Cog is installed with a standard Python distutils script:
You should now have cog.py in your Python scripts directory. LicenseCog is distributed under the MIT license. Use it to spread goodness through the world. Writing the source filesSource files to be run through cog are mostly just plain text that will be passed through untouched. The Python code in your source file is standard Python code. Any way you want to use Python to generate text to go into your file is fine. Each chunk of Python code (between the [[[cog and ]]] lines) is called a generator and is executed in sequence. The output area for each generator (between the ]]] and [[[end]]] lines) is deleted, and the output of running the Python code is inserted in its place. To accommodate all source file types, the format of the marker lines is irrelevant. If the line contains the special character sequence, the whole line is taken as a marker. Any of these lines mark the beginning of executable Python code: //[[[cog Cog can also be used in languages without multi-line comments. If the marker lines all have the same text before the triple brackets, and all the lines in the generator code also have this text as a prefix, then the prefixes are removed from all the generator lines before execution. For example, in a SQL file, this: --[[[cog will produce this: --[[[cog Finally, a compact form can be used for single-line generators. The begin-code marker and the end-code marker can appear on the same line, and all the text between them will be taken as a single Python line: // blah blah You can also use this form to simply import a module. The top-level statements in the module can generate the code. If there are multiple generators in the same file, they are executed with the same globals dictionary, so it is as if they were all one Python module. Cog tries to do the right thing with white space. Your Python code can be block-indented to match the surrounding text in the source file, and cog will re-indent the output to fit as well. All of the output for a generator is collected as a block of text, a common whitespace prefix is removed, and then the block is indented to match the indentation of the cog generator. This means the left-most non-whitespace character in your output will have the same indentation as the begin-code marker line. Other lines in your output keep their relative indentation. The cog moduleA module called cog provides the functions you call to produce output into your file. The functions are:
cog.out(""" Running cogCog is a command-line utility which takes arguments in standard form. cog - generate code with inlined Python code. Files on the command line are processed as input files. Files can also be listed in a text file named on the command line with an @: $ cog @files_to_cog.txt These @-files can be nested, and each line can contain switches as well as a file to process. For example, you can create a file cogfiles.txt:
then invoke cog like this: cog -s " //**cogged**" @cogfiles.txt Now cog will process four files, using C++ syntax for markers on all the C++ files, SQL syntax for the .sql file, and no markers at all on the readme.txt file. As another example, cogfiles2.txt could be:
with cog invoked like this: cog -D version=3.4.1 @cogfiles2.txt Cog will process template.h twice, creating both data1.h and data2.h. Both executions would define the variable version as "3.4.1", but the first run would have thefile equal to "data1.xml" and the second run would have thefile equal to "data2.xml". Overwriting filesThe -r flag tells cog to write the output back to the input file. If the input file is not writable (for example, because it has not been checked out of a source control system), a command to make the file writable can be provided with -w: $ cog -r -w "p4 edit %s" @files_to_cog.txt Setting globalsGlobal values can be set from the command line with the -D flag. For example, invoking Cog like this: cog -D thefile=fooey.xml mycode.txt will run Cog over mycode.txt, but first define a global variable called thefile with a value of "fooey.xml". This variable can then be referenced in your generator code. You can provide multiple -D arguments on the command line, and all will be defined and available. The value is always interpreted as a Python string, to simplify the problem of quoting. This means that: cog -D NUM_TO_DO=12 will define NUM_TO_DO not as the integer 12, but as the string "12", which are different and not equal values in Python. Use int(NUM_TO_DO) to get the numeric value. Checksummed outputIf cog is run with the -c flag, then generated output is accompanied by a checksum: --[[[cog If the generated code is edited by a misguided developer, the next time cog is run, the checksum won't match, and cog will stop to avoid overwriting the edited code. Output line suffixesTo make it easier to identify generated lines when grepping your source files, the -s switch provides a suffix which is appended to every non-blank text line generated by Cog. For example, with this input file (mycode.txt):
invoking cog like this: cog -s " //(generated)" mycode.txt will produce this output: [[[cog MiscellaneousThe -x flag tells cog to delete the old generated output without running the generators. This lets you remove all the generated output from a source file. The -d flag tells cog to delete the generators from the output file. This lets you generate code in a public file but not have to show the generator to your customers. The -U flag causes the output file to use pure Unix newlines rather than the platform's native line endings. You can use this on Windows to produce Unix-style output files. The -I flag adds a directory to the path used to find Python modules. The -z flag lets you omit the [[[end]]] marker line, and it will be assumed at the end of the file. HistoryCog's change log is on a separate change page. FeedbackI'd love to hear about your successes or difficulties using cog. Comment here, or send me a note. See AlsoThere are a handful of other implementations of the ideas in Cog:
At PyCon 2011, I did a lightning talk explaining Cog (I start at about 8:00 minutes in). You might like to read:
| |
Comments
I'm using Cog! I use it to do code generation for a library that's implemented in three different languages (C#, c++, Java). Linked from my blog. Thanks for the distribution fix!
This is really nice - normally I don't like the idea of having files which are hand-edited AND machine generated, but this is simple enough to change my mind about this. I really like this.
Some minor issues - the 'cog.py' file in scripts has a cr-lf at the end of the first line, which makes it fail to run, and makes distutils fail to edit in the proper python bin location. Easily fixed.
Running the script with no command line parameters should cause it to print a help message.
How about an option which will cause
cog to fail with an error if the file's
actual autogenerated code doesn't match the live python output. This might make more sense than '-r' for use in a makefile, to ensure that you don't edit the wrong code and have your changes discarded.
I notice that the 'import cog' is not actually required, this is good. It would be very nice if each python snippet were run in the same global dictionary, so you could define a variable (or import a module) at one point and use it throughout.
Finally -- certain languages (VHDL,
Makefiles, Python, for instance) do
not have multi-line comments, so
you can't use it with these, unless
I am missing something. May I suggest:
if the [[[cog is preceded by some text on its line, then all lines between that
and the ]]] are checked to see if they
start with the same text, and if so, that is discarded before further processing.
Thanks, Greg. I added the non-multi-line comment support, and a few other minor things. Thanks for the suggestions!
It looks neat, but I generally like to have no codegen code in the generated code.
But a slightly different approach would fix that:
1. External Python or cog-CodeBehind file next to the source file.
2. You add naming to your cog-codegen slots.
This allows the CodeBehind codegen code to find the output slot for its generated code.
EXAMPLE:
// file:MyCxxFile.hpp
// SLOT-CONTE
/*[[[cog-slot:EnumerateMethods]]] */
void DoSomething();
void DoAnotherThing();
void DoLastThing();
//[[[end]]]
// file:MyCxxFile.hpp.cogCodeBehind or MyCxxFile.hpp.cog or ...
// NOTE: Comments etc. are optional now.
/*[[[cog-codeBehind:EnumerateMethods
import cog
fnames = ['DoSomething', 'DoAnotherThing', 'DoLastThing']
for fn in fnames:
cog.outl("void %s();" % fn)
]]]*/
You can come close with Cog as it is. Remember that you can import any Python module you want into the cog code. By moving all of your Python code into another module, you can reduce the Cog code to a single import statement:
// file: MyCxxFile.hpp
/*[[[cog
import MyCxxFileGen
]]]*/
/*[[[end]]]*/
# file: MyCxxFileGen.py
import cog
fnames = ['DoSomething', 'DoAnotherThing', 'DoLastThing']
for fn in fnames:
cog.outl("void %s();" % fn)
Cog is great, but what I'd really like to be able to do is the following:
// [[[cog import mycodgen as m]]]
// [[[end]]]
... lots of regular code ...
// [[[cog m.somestuff()]]]
// [[[end]]]
... other code ...
// [[[cog m.otherstuff()]]]
// [[[end]]]
As it is, each cog slot seems to "forget" the globals() dict for previous cog slots (i.e. line 86 in cogapp.py passes an empty globals() dict to eval).
You got it: 1.3 does what you want.
Just FYI, think I've finally settled on COG as a tool for PHP code generation. Had looked at empy (some notes here: http://www.sitepoint.com/blog-post-view.php?id=222590) but what's clinched it is the way you've implemented the output area.
The mission is to eliminate any work PHP might do that relates to application configuration or the environment it's running in - stuff that won't change once an app is deployed so eliminate the runtime overhead.
One particular thing I want to keep is the ability to execute the PHP scripts, while hacking, before they have been run through COG, e.g.
/**
[[[cog
import cog
cog.outl("require_once '/full/path/to/someClass.php';")
]]]*/
// While hacking, use this
require_once DEV_PATH 'someClass.php';
//[[[end]]]
May even ditch the require_once completely and have COG embed the class code directly into the script.
Anyway - thanks.
Just what I'm looking for, thanks! However, am using it to generate multiple source files from each template file (ie. feed a different xml config file in to the template to generate a unique file). Is there any way to pass an argument (e.g. a file name) through the command line invocation?
That way I could provide the xml file to the template dynamically.
Thanks again!
Theo
Theo, it would be straightforward to add a -D name=value syntax to the command line. Each would create a variable in the global context. I think this would cover your needs.
I think this is a good idea. I'll add it into Cog soon.
Excellent, looking forwards to it,
Theo
You know what COG really needs? A "COG-recipes" site. I'm sure people have developed some interesting scripts.
One script I'm interested in (that I'm sure others would be too) is a script that will generate C++ functions that will translate enum values to and from string representations. Having a recipes site would allow me to share my code as well as allow others to give feedback about my script.
can you use COG to make codes out of videos andpictures and soundfile? for instance on myspace you can go to websites and get html codes for videos, wich resemble COG in a way, would you be able to e-mail me backon this subject?
I like the idea expressed by Kevin above about a "COG recipes" site. Because of that I started a COG Wiki site (http://www.bluwiki.org/go/COG_Code_Generator) to allow for discussions and code snippets for COG.
Anyone who has snippets to share, please post them. The Wiki will become better as more people share their code.
Hey man,
this is a really handy tool you wrote.
Keep up the great work.
*And why not directing the stdout instead of cog.outl ?
Thanks
You should note that intalling COG also requires the PATH module.
Hi, this tool certainly looks very promising, I'm going to give it a go to clean up some of the internals used in aqsis (www.aqsis.org). The other devs favour xsltproc at the moment - I'll see if I can convince them and myself...
One thing I noticed after just installing cog-2.0 from the tarball is that cog.out no longer seems to recognise the trimnewlines command. After grepping the source I see that it's apparently been changed to trimblanklines?
is anyone willing to try and teach me this stuff ? i find it really interesting.. but i don't expect anyone to take me up on this, it seems really complicated. thats why im so interested.
Bug report.
/*[[[cog
import cog
cog.outl(" extern void simple1();")
cog.outl(" extern void simple2();")
]]]*/
//[[[end]]]
I think the result should be indented, since the dedent parameter isn't given and the default value is False.
However, the result is "dedented". (reindentBlock is called twice)
Jay: this code is behaving as intended. I've edited the description of indentation to try to make it clearer. The dedent parameter doesn't affect the indentation of the line in the output, just the interpretation of a multi-line string parameter. Cog collects output as a block of text, then indents the block to match the generator. The dedent parameter affects how cog adds the lines to the collected output, but now how that output is finally written.
To see what the dedent parameter does, try putting five spaces at the left of one of your lines, and try it with dedent=True and dedent=False to see the difference.
Sorry for the confusion. If you need the output block indented differently, indent your entire cog block to where you want the output to go.
Hi Ned,
This is some neat python app :-)
I would like to try it for some Java apps around here. Has anyone tried whether
it works with jython? If so I could easily use it from ant and would not
require people to install Python.
... I just checked, it seems that jython won't work since cog requires the compiler package (which isn't available on jython).
I've never used Jython, but yes, Cog depends on being able to execute the chunks of Python code it finds.
nice tool
thanks
I happily used cog 3 or so years ago on a C# project. Now I need it again! Thanks...
FYI, I have code that users must run, and the code has strings with passwords in them. I don't want users to be able to find passwords by looking at the code they run (strings show up in binary code). I'll use cog to generate encrypted strings, and when the program runs the strings will be decrypted. Users won't be able to see clear text passwords no matter what they do, except if they figure out the decryption code. This is not NSA type security, it is for ordinary users.
Of course I could manually encrypt the password strings; then I'd not need cog. But that is a lot more work as there are several programs that will use this technique, and passwords change from time to time. The python code will be very short.
g.
Great tool
Thanks
Ned, I love cog. I use emacs too, and run cog from within emacs with a single keystroke, as you noted here http://nedbatchelder.com/blog/200602/pcg_cog_in_perl.html
thank you very much
haroldo
Brilliant.
It saved me a lot of work.
Thx
I prefer Enhanced CWEB and yesweb to do some of these things. (Enhanced CWEB will generate #line directives in the output file, and you can use metamacros and named chunks to do all sorts of things. This program can be used with C and C++.) (yesweb is written entirely in TeX, so you can use any TeX commands to extend its features.)
Here are a bunch of examples of Enhanced CWEB metamacros:
@m Repeat@T ``yg_Repeat @) @m _Repeat@T J@ A-g_Repeat @) @m Repeat@W q/@tRepeat Bq``: zyqnB" times:\ @> qA"@# @) @mp@- ``u{YJ"@<Predeclaration of procedures@>= qJA"; J"@ "@<Procedure codes@>= B" { @) @ms@- ``u{YJ"@<Static declarations@>= static qJA"; J"@ "@<Procedure codes@>= static B" { @) @mpacked@- q``Y/__attribute__((packed)) j/@t$\flat\ $@> AqA @) @mkind@T ``k"@<Data of kinds@>= "#define kind (kind_data[_kindnum]) "#undef _kindnum :#z:dz:ez:fz:iz:nz:ez: z:_z:kz:iz:nz:dz:nz:uz:mz: z :0z:xzu;qA"strcpy(kind.name+1, :"zqu;qBz");kind.character= :0z:xzqu;qA";kind.color= :0z:xzqu;qA";kind.menu= qu;qA";kind.selection_key= qu;q:'zBz";kind.cycle= qu;qB"; qgkind_ @) @mkind_@T xJ@ z"_kind_flag; gkind_ @) @mkind@W "\addkind% B";;% @) @mname@- ``YJ"@ q/@d : zBqA @)Hi,
I have code that has Windows line endings and got syntax errors on Linux because of them. Other developers use only Windows so changing the line endings of the file is a bit too tedious. To fix this, I had to modify the evaluate method to convert \r\n line endings to \n before running the code:
intext = "import cog\n" + intext + "\n"
intext = intext.replace("\r\n", "\n"); # Replace Windows line endings
I like it. I use it.
I would like the ability to add comments in my C code (like "do not edit" or "generated from somefile.txt"), which is a bit awkward since the closing comment mark terminates the cog code.
I added this to my local version, which would be nice in a future release:
Index: cogapp.py =================================================================== --- cogapp.py (revision 2416) +++ cogapp.py (revision 2417) @@ -131,6 +131,7 @@ cog.cogmodule.msg = self.msg cog.cogmodule.out = self.out cog.cogmodule.outl = self.outl + cog.cogmodule.c_comment = self.c_comment cog.cogmodule.error = self.error self.outstring = '' @@ -167,6 +168,13 @@ self.out(sOut, **kw) self.out('\n') + def c_comment(self, sOut='', **kw): + """ Add a C-style comment to the output + """ + self.out('/* ') + self.out(sOut, **kw) + self.out(' */\n') + def error(self, msg='Error raised by cog generator.'): """ The cog.error function. Instead of raising standard python errors, cog generators can use@Brad: I would rather not add a language-specific method like c_comment. Couldn't you output the comment closer as two strings, or with a backslash: cog.outl("/* comment *\/")
@Ned: Sure. I'm ok with keeping it as a local change - just thought it might be an interesting addition.
Splitting the comment isn't very "attractive", which is why I went with a wrapper (not dissimilar to how you don't _really_ need outl() - you could just use out() and add a "\\n" each time.)
Thanks again for providing cog!
When calling cog.py, it seems that it passes in sys.argv and modifies it. We like to do things locally that rely on sys.argv (mainly passing in the name of a program to databases so we know what/who's logged in).
Would you consider either having cog.py pass in list(sys.argv) to pass in a copy of sys.argv, or in cogapp.py set argv0=argv[0];argv_rest = argv[1:]; and then use argv_rest for parsing instead of doing argv.pop(0)?
Sorry for asking stupid question. Does this example still work?
import cog
cog.outl(" extern void simple1();")
I am having trouble running this in Eclipse/Pydev. Thanks for your help.
A useful feature would be to have an option that would allow the output of running cog to accumulate between ]]] and [[[end]]] rather than have it deleted. An example use case for this would be a file whose purpose is to accumulate data that is produced periodically, such as from a weather data generation service. This would be a great application for cog is there were someway to have it not deleted what was generated from previous runs. Further, it would be useful to specify that this behavior should per block rather than per file. This perhaps could be implemented as a meta-variable on the opening delimiter, say something line "p[[[cog{retain}", or something to that effect.
What do you think?
Hi Ned,
A very useful tool.
I would like to know whether, if I import one of my module in one file, then can I use the functions present in that module or any python variable defined in that cog file in other file ?
Hi Ned,
I'm using cog for automating some code generation from a text file, and I love it. It works great for removing some of the repetition from my day. Unfortunately, I've come across a bug. I read my input text file, and store my code lines in a dictionary of lists.
When I come to a specific section I loop over each line in each list, in the appropriate dictionary entry, and print them with cog.outl(). All the lines start with \t to indent them appropriately, but cog strips this out, unless I also print a line with no indent.
I have found a bug (Might not be after all).
Currently with the latest COG available for download, I am unable to indent the code with '\t'(tab) in the beginning.
I am using cog.out and cog.outl for creating the code statements.
Currently I have changed the "reindentBlock" function in "whiteutils.py".
Changed statement to "l = l.replace(oldIndent, '\t', 1)".
Can you please check if it is a bug or am I missing something ?
Add a comment: