AsyncIO Python Pitfalls

In an earlier post I mentioned switching my code to use Python3’s Async/Await and not seeing any noticeable improvements. In an article on LWN this week the answer popped out:

Much of the Python standard library is written in blocking fashion, however, so the socket, subprocess, and threading modules (and other modules that use them) and even simple things like time.sleep() cannot be used in async programs. All of the asynchronous frameworks provide their own non-blocking replacements for those modules, but that means “you have to relearn how to do these things that you already know how to do”, Grinberg said.

(From An introduction to asynchronous Python )

That’s an annoyance that I can’t easily fix if I use other libraries that internally use these blocking modules. So I’ll need to see if I can break down the logic into smaller chunks and write asnyc versions for some parts at least (e.g. the fetching of a webpage if not for the pdf conversion, or by calling a subprocess asynchronously instead of using the existing blocking library).

The docs have sections for async transports, protocols, subprocesses and so on, so I’ll need to spend some time looking at it before I proceed.

More to come when I make some progress.