Multithreaded FTP upload. Is it possible?
I need to upload multiple files from directory to the server via FTP and SFTP. I've solved this task for SFTP with python, paramiko and threading. But I have problem with doing it for FTP. I tried to use ftplib for python, but it seems that it doesn't support threading and I upload all files one by one, which is very slow.
I'm wondering is it even possible to do multithreading uploads with FTP protocol without creating separate connections/authorizations (it takes too long)?
Solution can be on Python or PHP. Maybe CURL? Would be grateful for any ideas.
There is an effor of creating an asynchronous FTP client but basically the FTP protocol allows synchronous commands. To allow parallel downloads/uploads you need to open multiple FTP connections. This can be done by ftplib and threads.
Another approach to concurrency besides through is asynchronous io. For Python the standard toolkit for asynchronous network is Twisted.
Take a look at this FTP client example in Twisted.
It looks like something might be possible with the pycurl module.
There are two examples that if you put them together would work.
and look into the retriever-multi.py and file_upload.py for examples on how to possibly put them together.
You can run the script in multiple command prompts / shells (just make sure each file is only handled once by all the different scripts). I am not sure if this quick and dirty trick will improve transfer speed though..