How to run an external program and report back the browser asyn in Cherrypy

I am trying to setup a web app with a Cherrypy backend. The web app is just a GUI for a program that will run in the server parsing some files. What I want to be able to do is when it files are uploaded, and the program is executed at the back end, the output of the process to be reported back to the browser.

I have setup CORS and Server sent events with my Cherrypy server. I am unsure whether this is a case of Server sent events or WebSockets. I am really looking for a way to implement this functionality (reporting back) efficiently and be able to serve multiple clients (which cherrypy already supports).

What I have tried:

  • Setup SSE events which sends a variable (public) holding the output from the subprocess. This works for single client. Since the object attributes are persistent in cherrypy. (Covered in this question).
  • So then I tried to do sessions. Here is where I am stuck, the session elements set disappear in the SSE function. I tried many things, but not been successful. Even if they were, I am not sure how I will go about avoiding the shared data between requests with SSE.

SSE method:

@cherrypy.expose
def getUpdate(self, _=None):

    print "Horay! %s" % cherrypy.session.get('pid') # <-- this is None, even though its set in the main method
    try:
        cherrypy.session['youknow'] = "iknow"
        cherrypy.session.save()
    except Exception as e:
        raise e
    if _:
        data = 'data: meh meh...\n\n'
        return data
    else:
        def content(data=None):

            if self.dataReady and self.prog_output is not None:
                response = ""
                for output in self.prog_output:
                    response += 'data: %s \n\n' % str(output)
                data = 'retry: 2000\n%s \n\n' % response
                return data
            elif self.dataFinished:
                self.dataFinished = False
                return 'data: + Done\n\n'

        return content()

getUpdate._cp_config = {'response.stream': True, 'tools.encode.encoding': 'utf-8'}

Main method:

@cherrypy.expose
@cherrypy.tools.noBodyProcess()
def uploadFile(self, file=None):

    theFile = file

    # Save the pid in the user's session (a thread-safe place)
    cherrypy.session['pid'] = "pid"
    cherrypy.session.save()

    // calls the external program and parses here in a different thread, which updates the prog_output variable

Cherrypy Config:

if __name__ == '__main__':
    cherrypy.tools.CORS = cherrypy._cptools.HandlerTool(CORS)
    cherrypy.config.update({
    "tools.CORS.on": True,
    "server.socket_port": 8090,
    "server.thread_pool": 8,
    "tools.sessions.on": True,
    # 'tools.sessions.storage_type': "File",
    # 'tools.sessions.storage_path': 'sessions',
    # 'tools.sessions.locking': 'explicit',

Is this just a case of fixing the session issue, or should I be using WebSockets in this particular usecase.

Answers


I was able to solve this issue (a while ago now) by breaking the issue down into two parts.

  • First, I uploaded the files which needed to be processed. The server then returned these file paths and the name for the new file, as the response.
  • Once I get the response from the server (filepaths, newfilename), I call the SSE event function on the server (i.e. www.server.com/runProc/filepath/filepath2/newfilepath), which runs the process and streams the output back to the browser.

My issue was that I was trying to do this in one go, which in turn created an unnecessary problem. Also, with this there is no need for sessions or saving data across multiple requests.


Need Your Help

iPhone Sqlite insert running on nsthread

iphone ios5 sqlite xcode4.3

Im developing an application in which i need to insert a huge amount of data into sqlite database. The data gets inserted properly but the issue is some times i get an exception that

Erlang/OTP R13B ARM binary download

erlang arm

Is there a site where Erlang/OTP binaries for ARM-Linux can be downloaded? www.erlang.org only has source release (and I haven't managed to cross-compile it on the initial attempts), while http://c...