Running a subprocess and waiting for it to return.

[Valmian]

Hello,

First of all, I'm sorry if my question seems silly, but I am very unfamiliar with Python.

I am using a particular python code base which, in a Linux system, solves the problem I am working on (I am using Windows). A major part of the python code's task is to generate and run certain matlab scripts. The code starts matlab subprocesses by an os.system(command). Under Mac/Linux, it seems that python code waits until the matlab script terminates; however, under windows, the code simply proceeds before the matlab script terminates which crashes the application. I am wondering how I could rewrite the code to ensure that the python script waits for matlab subprocess to terminate.

The function I am using is below:

def runMatlab(self,dirname,scriptname):
        """ run matlab as a unix command
        commands: the list of command lines for matlab
        Returns the output from the matlab commands
        """
        
        logfile = scriptname+".log"
        scriptname = scriptname.rstrip(".m")
        scriptname = os.path.basename(scriptname)
        
        cwd=os.getcwd()
        os.chdir(dirname)               # matlab is not happy with running a script not in the current directory
        command=self.runInfo.matlab+' -logfile '+logfile+' -nojvm -nosplash -nodisplay -r '+scriptname
        print "matlab command=|"+command+"|\n"
        print "current directory is:"+os.getcwd()+"\n"
        self.matlabStatus = os.system(command) 
        os.chdir(cwd)                   # switch back to original working directory
        if self.matlabStatus != 0:
            print "Matlab terminated with error: check logfile "+ dirname+logfile
            sys.exit(1)

I tried to modify os.system(command) as

filename=self.runInfo.matlab
argin=(' -logfile ',logfile,' -nojvm',' -nosplash',' -nodisplay', '-r', scriptname)
os.spawnv(os.P_WAIT,filename,(filename,)+argin)

but I guess this is invalid syntax...

Any help would be greatly appreciated,
Thank you very much in advance,

Ilya

[Ene Uran]

Module subprocess, specifically subprocess.Popen(), has a wait() method. Here is a typical example:

# using module subprocess to pass arguments to and from an
# external program:
import subprocess

# put in the corresponding strings for
# "mycmd" --> external program name
# "myarg" --> arg for external program
# several args can be in the list ["mycmd", "myarg1", "myarg2"]
# might need to change bufsize
p = subprocess.Popen(["mycmd", "myarg"], bufsize=2048, shell=True,
    stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True)

# allow external program to work
p.wait()

# read the result to a string
result_str = p.stdout.read()

[Valmian]

I tried to substitute os.system for subprocess.call but that did not help.

Ilya

P.S: I posted this before Ene Uran's post, I'll try your suggestion now.

[Valmian]

Hello,

I modified the code in the following way:

command=' -logfile "'+logfile+'" -nojvm -nosplash -nodisplay -r '+scriptname
        print "matlab command=|"+self.runInfo.matlab+command+"|\n"
        print "current directory is:"+os.getcwd()+"\n"
        p = subprocess.Popen([self.runInfo.matlab,command], bufsize=2048, shell=True,stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True)
        p.wait();
        
        self.matlabStatus=p.stdout.read;

I am getting the following error :

Traceback (most recent call last):
  File "neuronLearningManager.py", line 49, in <module>
    main()
  File "neuronLearningManager.py", line 18, in main
    run_edge(imageFile,"test")
  File "neuronLearningManager.py", line 33, in run_edge
    manager.prepareDataForJBoost()
  File "C:\Users\Ilya\Desktop\neuron boosting\scripts\NeuronLearning\src\learnin
gManager.py", line 149, in prepareDataForJBoost
    matlabOutput = self.runMatlab(self.runInfo.matlabScriptDir,scriptName)
  File "C:\Users\Ilya\Desktop\neuron boosting\scripts\NeuronLearning\src\learnin
gManager.py", line 352, in runMatlab
    p = subprocess.Popen([self.runInfo.matlab,command], bufsize=2048, shell=True
,stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True)
  File "C:\Program Files\Python26\lib\subprocess.py", line 553, in __init__
    raise ValueError("close_fds is not supported on Windows "
ValueError: close_fds is not supported on Windows platforms if you redirect stdi
n/stdout/stderr

Removing close_fds=True argument did not help, as the code now crashes with

The filename, directory name, or volume label syntax is incorrect.

(again, os.system and subprocess.call managed to run the process, they simply didn't wait).

Any suggestions?

Thank you,

Ilya

Module subprocess, specifically subprocess.Popen(), has a wait() method. Here is a typical example:

# using module subprocess to pass arguments to and from an
# external program:
import subprocess

# put in the corresponding strings for
# "mycmd" --> external program name
# "myarg" --> arg for external program
# several args can be in the list ["mycmd", "myarg1", "myarg2"]
# might need to change bufsize
p = subprocess.Popen(["mycmd", "myarg"], bufsize=2048, shell=True,
    stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True)

# allow external program to work
p.wait()

# read the result to a string
result_str = p.stdout.read()

[Ene Uran]

May be because your code block indentations are screwed up.

[Valmian]

May be because your code block indentations are screwed up.

just checked, it seems that the indentation is correct....

[Valmian]

Hello,

This code doesn't crash but it doesn't wait either, I'm starting to wonder if the problem lies with matlab2007b....

command=self.runInfo.matlab+' -logfile '+logfile+' -nojvm -nosplash -nodisplay -r '+scriptname
        print "matlab command=|"+command+"|\n"
        print "current directory is:"+os.getcwd()+"\n"
        
        p = subprocess.Popen(command, bufsize=2048, shell=True,stdin=subprocess.PIPE, stdout=subprocess.PIPE)
        p.wait()
        self.matlabStatus=p.stdout.read()

Ilya

[Valmian]

I am currently using time.sleep() function, I guess I'll make some work around with matlab generating a temp file marking operations "complete" which python will wait and check periodically. I just wish something less ugly could be done.

-Ilya

[zeterp]

The problem apparently has to do with the way the command line matlab launcher operates in Windows. The launcher spawns a separate process and then immediately exits unless you use the -wait flag.

[bjerke]

Warning concerning Popen.wait()

This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

http://docs.python.org/library/subprocess.html

[Jim_9]

I had a similar problem when I was trying to run Excel. If would work if Excel wasn't running, but would fail when other instances of Excel were already in memory. The code shown below worked for me. I think it might work with programs like Matlab that spawn subprocesses, because child processes inherit stdin and stdout from their parent.

from subprocess import Popen, PIPE, STDOUT
P=Popen('EXCEL.EXE',shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
Excel_output=P.stdout.read()
Status=P.wait()