One of the best features included in the Python 3.X is the asynchronous programming. In 3.4 was added the asyncio module. With the "@asyncio.coroutine" decorator you can define co-routines. A co-routine is a generalization of the subrutine. It allows to stop and continue the execution at some point inside the function.
In later versions was added a sugar syntax to simplify the calls: await y async.
Without the details, the steps are, firstly execute a event loop. This is the main engine that controls the execution of the coroutines. It manages what coroutine is executed depending the events of the system (reception of a socket, database reading, etc.) Inside a coroutine can be launched another asynchronous tasks and so on. The execution is interleaved between them through the event loop.
For example, the "hello world", that prints a message and wait a second:
import asyncio
async def hello_world():
print("Hello World!")
await asyncio.sleep(1)
loop = asyncio.get_event_loop()
loop.run_until_complete(hello_world())
loop.close()
Actually asyncio is very mature and used, there are many libraries to improve the use and to help with the most common tasks. Examples of this libraries are:
More examples can be found in the Github repository: aio-libs.
One of the most useful task that can be made asynchronously is to make many HTTP requests. Every requests takes some milliseconds, so it would be useful, while the program is waiting the first request, to lanunch the other URLS. Finally collect the request results.
To test it I have created a little benchmarch. It launches many requests. For all the tests, except for aiohttp, i have used the library requests. I know that there is the library asks. It can be integrated with curio y trio and replaces the requests library. But finally I've decided to not use it because it gave me errors with my Python version.
As a baseline case, I also created a synchronous example:
def run(urls):
for url in urls:
r = requests.get(url)
After that, I have implemented a native asynchronous version, with the asyncio library.
async def main(loop, urls):
futures = []
for url in urls:
future = loop.run_in_executor(None, requests.get, url)
futures.append(future)
for future in futures:
r = await future
def run(urls):
loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop, urls))
The aiohttp example is pretty simple. A similar implementation can be found in the official documentation:
async def fetch(session, url):
with async_timeout.timeout(50):
async with session.get(url) as response:
return await response.text(encoding="iso8859-1")
async def main(urls):
async with aiohttp.ClientSession() as session:
for url in urls:
await fetch(session, url)
def run(urls):
loop = asyncio.get_event_loop()
loop.run_until_complete(main(urls))
In the curio implementation I have used the spawn function. It allows to launch many tasks. After that you have to call the join method until the task finishes.
async def fetch(url):
requests.get(url)
async def main(url_list):
tasks = []
for url in url_list:
task = await curio.spawn(fetch(url))
tasks.append(task)
for task in tasks:
await task.join()
def run(urls):
curio.run(main(urls))
Finally in the trio implementation I have created a pool of coroutines through a context. This context waits all the tasks before finish.
async def fetch(url):
requests.get(url)
async def main(urls):
async with trio.open_nursery() as nursery:
for url in urls:
nursery.start_soon(fetch, url)
def run(urls):
trio.run(main, urls)
The tests requests 50 urls, calculating the average time in batches of 10 iterations.
| Implemetation | Time (seconds) |
|---|---|
| Synchronous | 26.95 s. |
| Native Async | 2.87 s. |
| AioHttp Client | 21.14 s. |
| Curio | 22.45 s. |
| Trio | 23.28 s. |
As you can see, the asynchronous versions are fast. The native is the best. However the difference is very high. There is probably an implementation error.
The complete sourcecode and the launcher of the tests, can be viewed/downloaded in Github
I hope you find it useful :D