Shell = Maybe
Created 11 April 2017, last updated 13 January 2018
Lots of people use Python to run other programs. Sometimes this is because they are using Python to coordinate other processes. Sometimes, it’s because they are coming from a shell scripting world, and running other programs to get work done feels most natural.
If you are trying to run other programs (spawn subprocesses) in Python, the first thing to do is make sure you need to. Lots of things that are done with programs in a shell script are done more naturally with Python libraries. As an example, there’s no need to use “ls” to list the files in a directory when you have os.listdir().
Once you decide you are going to create subprocesses, there’s a common question to work through: whether to use a shell or not. To answer that, let’s talk about shells.
What is a shell, and what does it do?
When you type a command in your terminal, you are not typing to “the computer.” You are typing to a program, called a shell. The shell’s job is to interpret the command line you type, and actually do what it says. A shell is a program that is very good at running other programs.
When you type a command line to the shell, it splits it into a list of words, or arguments. The first word is the command itself. The shell uses that to find the program to run. The rest of the words are bundled up as a list of strings, and given to the program.
This process of splitting the command line into words and passing them as a list of strings isn’t a Python thing, this is the way Unix works, and the way Windows generally mimics Unix. It’s the shell’s job to turn the command line you type into a list of strings.
A note about Windows: at its deepest native level, Windows is different than this. Programs get a single string, the original command line. But because of the C language’s close cultural ties to Unix, C programs on Windows get a list of strings, and other languages do the same. There are some differences between Windows and Unix still, but the big picture is the same: command lines get turned into a list of strings.
How does the shell turn a command line into a list of strings? For the simplest cases, it just splits the line on spaces. So this command:
grep apple foo.txt
is turned into:
['grep', 'apple', 'foo.txt']
If you want to experiment with this conversion of command lines into lists of strings, put this short Python program into echo.py:
#!/usr/bin/env python
import sys
print(sys.argv)
Now you can try it yourself:
$ chmod +x echo.py
$ ./echo.py grep apple foo.txt
['./echo.py', 'grep', 'apple', 'foo.txt']
But what if you need an argument to have a space? If you want to search a file for “red apple”, you need a command like:
grep "red apple" foo.txt
Just splitting this on spaces would give four strings, which isn’t right. The shell sees the quotes and understands that the quoted string should be kept together as a single argument. The resulting list is:
['grep', 'red apple', 'foo.txt']
Notice that the double-quotes themselves are not in the argument. They were there so the shell would understand that red apple should be kept together as one argument. But they aren’t part of the argument themselves.
There are other ways to protect spaces. This command could have been typed any of these ways:
grep "red apple" foo.txt
grep 'red apple' foo.txt
grep red\ apple foo.txt
The grep program literally can’t tell the difference between these three lines, because the shell produces the exact same argument list for all of them.
Shells do much more than just split the line into an argument list. As we’ve just seen, they also deal with quoting and escaping special characters. But there’s much more. When you use a wildcard pattern to do something with many files, it’s the shell that expands that pattern into a list of actual files. This command:
grep apple *.txt
could be turned into this argument list:
['grep', 'apple', 'bar.txt', 'baz.txt', 'foo.txt']
There are other more-advanced features of command line programs that are actually features of the shell:
- Variable expansion:
- Tilde expansion:
- Redirection:
- Piping:
- Sequencing:
- Sub-commands:
grep $WORD *.txt
cp foo.txt ~
grep apple *.txt > apple_lines.txt
grep apple *.txt | wc -l
cp old.txt new.txt && rm old.txt
mv now.txt $(date +%Y%m%d).txt
Understanding the role of the shell is critical to getting your subprocesses to run correctly. Because you will be deciding whether to even use a shell or not.
Using subprocess
The Python subprocess module has a few different functions and classes you can use to run a subprocess. One thing they all have in common: you have to tell it what program to run and what arguments to give the program. There are two ways to do this, and it all comes down to shells.
The more familiar way is to run the program with a shell:
output = subprocess.check_output("ls -al", shell=True)
(Note: subprocess has a number of functions. I’ll use check_output because it is conceptually simple, but the shell considerations I’m discussing apply equally well to run, call, check_call, Popen, and so on.)
(Also note: this is one of those commands you shouldn’t use a sub-process for. Listing files is easy to do in other ways. But it’s nice and short for examples. We’ll get to more realistic examples in a bit.)
When you specify shell=True, the program and arguments are provided in a single string. The shell is started, and given that string as the command line to execute. This gives a very familiar interface to running programs: it’s exactly what we are used to from the command line. The shell parses the command line it’s given, and invokes the program.
The other way to run the program is with no shell, which is the default:
output = subprocess.check_output(["ls", "-al"])
Here we’re running the program without the help of a shell, so we provide the program arguments as an explicit list of strings.
To shell, or not to shell?
If you’re wondering whether to use a shell with subprocess, the answer is simple: only use one if you have to. You should use a shell if you need some of its behavior, and otherwise avoid using a shell. Most of the time, you don’t need a shell.
There are good reasons to avoid using a shell:
- Using a shell takes more resources and time, since you are starting a program (the shell), which will then run the program you really want. There’s less to do if you run the program directly yourself.
- Using a shell introduces a layer of interpretation that can be hard to reason about. You have Python strings, with their escaping, producing shell command lines, with their escaping, to finally produce the arguments for your program. It’s simpler to just make your own list of arguments.
- Using a shell can be dangerous. Shells do a lot of things, maybe some things you didn’t anticipate. This is especially true if you are including untrusted input as part of the subprocess command line.
Shell injection
Here’s an example of that last point. Suppose you want to split a video into a series of images. Ffmpeg is a powerful video tool that can do that, with a command like this:
ffmpeg -i video.flv image%d.jpg
But you want to get the video file name from the user. You might do this to insert the user’s filename into the command, and then run it:
cmd = "ffmpeg -i {} image%d.jpg".format(user_filename)
subprocess.run(cmd, shell=True)
This works fine, but suppose the user gave you this file name: “; rm -rf * ; ” Now the constructed command line would be:
ffmpeg -i ; rm -rf * ; image%d.jpg
Running this would delete a lot of files, which is definitely not what you wanted. The user has maliciously injected shell content where you didn’t want it.
This is the risk of using the shell: it can do much much more than you intended it to.
How to avoid the shell
If you have a command line in mind, and you want to turn it into some Python code that runs the program the same way, you have to think like the shell. When the shell runs your command line, what list of string arguments does it produce? If you have a tricky case, it can help to use the echo.py program above to experiment with your command line.
Once you understand how the shell works, and what it is doing for you, you can decide whether you want to keep the shell in the mix (carefully), or skip the shell, and do that work yourself. Often, all the shell does is split your command into words, something you can do easily yourself.
If you are using shell features like wildcards or pipes, it becomes trickier to replace the shell with your own code. But Python provides all the tools you need:
- Wildcards are expanded into lists of files with glob.glob.
- Expansion of environment variables is done with os.environ or os.path.expandvars.
- Tilde expansion is done with os.expanduser.
- Redirection is done by setting the stdout and stderr arguments of your subprocess function.
- Subcommands are emulated by collecting the output of one command and using it to build the arguments of the next command.
- Piping is done by running a few subprocess functions and linking their stdout and stdin arguments together. The subprocess docs have an example.
Keep in mind that many simple commands can be avoided altogether in favor of Python libraries. For example, there’s no reason to run “date +%Y%m%d” to get the current date. You can get it from datetime.now.
There are a number of libraries to help with complex scenarios, though I have no experience with any of them, so I don’t know which to recommend! If you are running complex pipelines of commands, it will be easier to use a shell to do it. Just be very very careful.
Comments
The one piece that still seems awkward to me is dealing with pipes. It is so simple in shell scripts to pipe from one program to another or redirect to files. All of this is possible with the subprocess library, but it seems a lot more cumbersome than the shell script | < and > syntax. If there is a good syntactic sugar to replicate this functionality in one of the Python libraries, I'd love to know more about it.
For now, I need to go back to some previous projects and rip out more os.system() calls.
-D
Moreover, cmd.exe built-ins won’t work unless shell=True — while that also applies to *nix, cmd.exe has many built-ins that could be useful for subprocesses that do not have standalone .exe equivalents. (But if you have to do subprocesses on Windows, you’re already in for a fun ride.)
https://sarge.readthedocs.io/en/latest/overview.html#why-not-just-use-subprocess
Add a comment: