Python, Popen and select - waiting for a process to terminate or a timeout
I run a subprocess using:
p = subprocess.Popen("subprocess", stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
This subprocess could either exit immediately with an error on stderr, or keep running. I want to detect either of these conditions - the latter by waiting for several seconds.
I tried this:
SECONDS_TO_WAIT = 10 select.select(, [p.stdout, p.stderr], [p.stdout, p.stderr], SECONDS_TO_WAIT)
but it just returns:
on either condition. What can I do?
Have you tried using the Popen.Poll() method. You could just do this:
p = subprocess.Popen("subprocess", stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE) time.sleep(SECONDS_TO_WAIT) retcode = p.poll() if retcode is not None: # process has terminated
This will cause you to always wait 10 seconds, but if the failure case is rare this would be amortized over all the success cases.
t_nought = time.time() seconds_passed = 0 while(p.poll() is not None and seconds_passed < 10): seconds_passed = time.time() - t_nought if seconds_passed >= 10: #TIMED OUT
This has the ugliness of being a busy wait, but I think it accomplishes what you want.
Additionally looking at the select call documentation again I think you may want to change it as follows:
SECONDS_TO_WAIT = 10 select.select([p.stderr], , [p.stdout, p.stderr], SECONDS_TO_WAIT)
Since you would typically want to read from stderr, you want to know when it has something available to read (ie the failure case).
I hope this helps.
This is what i came up with. Works when you need and don't need to timeout on thep process, but with a semi-busy loop.
def runCmd(cmd, timeout=None): ''' Will execute a command, read the output and return it back. @param cmd: command to execute @param timeout: process timeout in seconds @return: a tuple of three: first stdout, then stderr, then exit code @raise OSError: on missing command or if a timeout was reached ''' ph_out = None # process output ph_err = None # stderr ph_ret = None # return code p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # if timeout is not set wait for process to complete if not timeout: ph_ret = p.wait() else: fin_time = time.time() + timeout while p.poll() == None and fin_time > time.time(): time.sleep(1) # if timeout reached, raise an exception if fin_time < time.time(): # starting 2.6 subprocess has a kill() method which is preferable # p.kill() os.kill(p.pid, signal.SIGKILL) raise OSError("Process timeout has been reached") ph_ret = p.returncode ph_out, ph_err = p.communicate() return (ph_out, ph_err, ph_ret)
Using select and sleeping doesn't really make much sense. select (or any kernel polling mechanism) is inherently useful for asynchronous programming, but your example is synchronous. So either rewrite your code to use the normal blocking fashion or consider using Twisted:
from twisted.internet.utils import getProcessOutputAndValue from twisted.internet import reactor def stop(r): reactor.stop() def eb(reason): reason.printTraceback() def cb(result): stdout, stderr, exitcode = result # do something getProcessOutputAndValue('/bin/someproc',  ).addCallback(cb).addErrback(eb).addBoth(stop) reactor.run()
Incidentally, there is a safer way of doing this with Twisted by writing your own ProcessProtocol:
Here is a nice example:
from threading import Timer from subprocess import Popen, PIPE def kill_proc(): proc.kill() proc = Popen("ping 127.0.0.1", shell=True) t = Timer(60, kill_proc) t.start() proc.wait()
If, as you said in the comments above, you're just tweaking the output each time and re-running the command, would something like the following work?
from threading import Timer import subprocess WAIT_TIME = 10.0 def check_cmd(cmd): p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) def _check(): if p.poll()!=0: print cmd+" did not quit within the given time period." # check whether the given process has exited WAIT_TIME # seconds from now Timer(WAIT_TIME, _check).start() check_cmd('echo') check_cmd('python')
The code above, when run, outputs:
python did not quit within the given time period.
The only downside of the above code that I can think of is the potentially overlapping processes as you keep running check_cmd.
import subprocess as sp try: sp.check_call(["/subprocess"], timeout=10, stdin=sp.DEVNULL, stdout=sp.DEVNULL, stderr=sp.DEVNULL) except sp.TimeoutError: # timeout (the subprocess is killed at this point) except sp.CalledProcessError: # subprocess failed before timeout else: # subprocess ended successfully before timeout
See TimeoutExpired docs.
This is a paraphrase on Evan's answer, but it takes into account the following :
- Explicitly canceling the Timer object : if the Timer interval would be long and the process will exit by its "own will" , this could hang your script :(
There is an intrinsic race in the Timer approach (the timer attempt killing the process just after the process has died and this on Windows will raise an exception).
DEVNULL = open(os.devnull, "wb") process = Popen("c:/myExe.exe", stdout=DEVNULL) # no need for stdout def kill_process(): """ Kill process helper""" try: process.kill() except OSError: pass # Swallow the error timer = Timer(timeout_in_sec, kill_process) timer.start() process.wait() timer.cancel()