Python Subprocess is an excellent module for running external commands within a Python script. When this module is used, process(es) related to the executing command is started. This article will cover ways to terminate these processes created by subprocess (parent and child processes).
Running Commands on the subprocess
There are two common ways to run commands using subprocess
- Using subprocess.run() function – This method not only executes commands but also captures output, checks the command’s return code, and allows the user to set time-out, among other things.
- Using subprocess.Popen() – This function works the same way as run() with most arguments missing. One of the missing arguments in Popen()- which is essential in this article- is the time-out feature.
Here is an example of code executed in python to run the “ls -la” command to get a long listing of all files and directories in the current working directory in the UNIX system (In Windows, you can display a long listing with “dir /a” command).
|
1 |
subprocess.Popen(["ls", "-la"]) |
If we want to pass a single string, we either set the shell argument to True (as shown below), or else the string must issue the command to be executed without specifying any arguments.
|
1 |
subprocess.Popen(["ls -la"], shell=True) |
Parent and Child Processes
A process in most systems is identified by the process name or the process id (PID or simply ID). A child process is a process created by the parent process, and each of them will have a distinct PID. In UNIX Sytems, system processes can be found on the GUI app (like System Monitor) or terminal-based htop tool (run “htop” on the terminal). On Windows, Task Manager should do the job.
The following Figure shows a Tilix GTK3 tiling terminal with three open sessions.
Let’s inspect the processes started when these three sessions are opened on the terminal.
The system monitor shows that opening three bash sessions on the Tilix terminal created three child processes (bash) on the parent process (tilix). By default, if we end the tilix process, the 3 child processes attached to it will also be killed. That is not always the case with subprocesses.
The correct way of Killing Processes in subprocess
We will discuss three methods for this. The first method works for UNIX Systems only, but the second and third methods are platform-independent.
Method 1: Killing processes using the Group ID
As an example, we will execute the following Python script named write_to_file.py from another script (on the same directory) running a subprocess.
The script writes numbers 0 through n=2 into a file; in each case, the execution is paused for 10 seconds. This means the script should take a minimum of 20 seconds to execute.
File: write_to_file.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
import time n = 2 # Open a text file on write mode (w) with open("out.txt", "w+") as f: for i in range(n): # loop n times and write the # loop index to the file # each number in a new line f.write(str(i)+"\n") # sleep for 10 seconds time.sleep(10) # execute another Python script in subprocess import subprocess pro = subprocess.Popen("python3 write_to_file.py", stdout=subprocess.PIPE, shell=True, start_new_session=True) print("Process ID:", pro.pid) pro.kill() |
Output:
Process ID: 6627
When the subprocess executes write_to_file.py using Python 3 two processes are started – python3.11 and sh (shell process) because shell=True.
When we issue the kill command on pro, it will kill the parent process (sh) but not the child (python3). In that case, the child process will continue running in the background.
To kill both the parent and child processes, we need to use the group ID, which identifies the parent process and all the resulting child processes. That can be achieved as follows:
|
1 2 3 4 5 6 7 8 9 |
import os import signal import subprocess # execute another Python script in subprocess pro = subprocess.Popen("python3 write_to_file.py", stdout=subprocess.PIPE, shell=True, start_new_session=True) print("Process ID:", pro.pid) os.killpg(os.getpgid(pro.pid), signal.SIGTERM) |
The os.getpgid(<pid>) function, as the name suggests, gets the process group ID linked to the parent identified by <pid>, which is then killed by os.killpg() – kill process group. The signal-SIGTERM sends the termination signal. Often, processes meant to run in the background will catch this signal and start a shutdown process, resulting in a clean exit.
Note: This method only works on UNIX-based operating systems. (Read: it won’t work on Windows. For Windows users, go to method 2).
Method 2: Using the psutil package
This package will allow us to identify parent and child processes and terminate them recursively.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
import psutil import subprocess import os def kill_processes(pid): '''Kills parent and children processess''' parent = psutil.Process(pid) # kill all the child processes for child in parent.children(recursive=True): print(child) child.kill() # kill the parent process print(parent) parent.kill() # remember to assign subprocess to a variable pro = subprocess.Popen("python3 write_to_file.py", stdout=subprocess.PIPE, shell=True, start_new_session=True) # get the process id print("Process ID:", pro.pid) # call function to kill all processes in a group kill_processes(pro.pid) |
Output:
Process ID: 7792 psutil.Process(pid=7793, name='python3', status='running', started='23:45:37') psutil.Process(pid=7792, name='sh', status='sleeping', started='23:45:37')
There is one child process (7793) in this case and one parent process (7792).
Method 3: Killing processes using time-out
For both subprocess.Popen() and subprocess.run() functions, we can issue a time-out after which a process is stopped. Once the running process is timed out, the TimeoutExpired exception is raised, and thus the process is stopped.
Here is how we can do this for each of the two functions.
|
1 2 3 4 5 6 |
import subprocess try: pro1 = subprocess.run("python3 write_to_file.py",stdout=subprocess.PIPE, shell=True, timeout=10) except: print("Process timed out") |
Output:
Process timed out
As said earlier, it takes at least 20 seconds to finish executing write_to_file.py. Therefore, when we set time_out to 10 seconds, we will land into TimeoutExpired.
Using try…except, we then catch TimeoutExpired when it is raised so that we can continue executing the rest of the script (if there was any code below that).
In subprocess.Popen() we issue time-out using the wait() function as shown below.
|
1 2 3 4 5 6 7 8 9 10 11 |
import subprocess import os try: pro1 = subprocess.Popen("python3 write_to_file.py",stdout=subprocess.PIPE, shell=True) # Wait 10 seconds for the process to finish execution pro1 = pro1.wait(timeout=10) except: print("Process timed out. Cleaning up background processes...") # just to make sure that all processes in the group are terminated. os.killpg(os.getpgid(p.pid), signal.SIGTERM) |
After the code is timed-out, we can use the os.killpg() to kill the group processes running in the background, if there are any, just like we did on Method 1.
Conclusion
When running external commands using the subprocess module, some processes may continue running in the background even after closing the subprocess.
This happens when we open the commands on the shell or when we explicitly create detached processes using arguments available on the subprocess.Popen() and subprocess.run().
In this article, we covered three methods to efficiently kill processes started by subprocess – ensuring that there’s no background process running after that.