The perfect combination of collaborative process and multi process

This is a 「 Attacking Coder」 Of the 607  Technology sharing

author ：kingname

source ： Unheard of Code

“

It is necessary to read this article 4 minute .

”

We know , A coroutine is essentially a single thread, single process , By making the most of IO Wait time to achieve high concurrency . stay IO Code outside the waiting time , Or serial operation . therefore , If there are many processes , How many serial codes in each collaboration run longer than IO Waiting time for request , There will be an upper limit on concurrency .

for instance , Cook in an electric cooker , Washing clothes with a washing machine , Boil water in a hot kettle , They all start the equipment and then they can run it by themselves , We can use their own running time , Make these three things seem to be going on almost at the same time . But if apart from these three things , And turn on the TV , turn the air conditioner on , Send wechat …… Wait, dozens of things . It really only needs to be a beginning when everything is taken out alone , The rest is to wait , But because it takes time to start , So it will take a lot of time to start them all , Your efficiency is still stuck .

Now? , If there are two people doing these things together , That's not the case . Cook and boil water alone , The other man turns on the washing machine , Turn on the TV and air conditioning . Efficiency has been further improved .

This is the combination of cooperative process and multi process , Multiple coprocesses in each process run at the same time , make the best of CPU Every core of , Make the most of it again IO Waiting time , hold CPU Run full , Run the network bandwidth full . Strong alliance , Faster .

There is a third-party library aiomultiprocess, It allows you to realize the combination of multi process and collaborative process in a few lines of code .

use first pip install ：

python3 -m pip install aiomultiprocess

Its grammar is very simple ：

from aiomultiprocess import Pool
async with Pool() as pool:
    results = await pool.map( coroutines ,  parameter list )

It only needs 3 Line code , It'll be there for you CPU Start a process on each core , Start the process continuously in each process .

Let's write a piece of actual code ：

import asyncio
import httpx
from aiomultiprocess import Pool

async def get(url):
    async with httpx.AsyncClient() as client:
        resp = await client.get(url)
        return resp.text


async def main():
    urls = [url1, url2, url3]
    async with Pool() as pool:
        async for result in pool.map(get, urls):
            print(result)  #  every last URL What's coming back 

if __name__ == '__main__':
    asyncio.run(main())

When I was writing asynchronous collaboration articles before , Some of my classmates will ask me , Is the speed of a reptile really that important ？ Isn't it the most important thing to break through anti reptiles ？

My answer is , Don't see the use of aiohttp Asking for a website feels like being a crawler . In microservices , Ask for your own HTTP Interface , Also need to use httpx perhaps aiohttp. In such a scene , Speed is very important , Sometimes it just needs to be done as soon as possible .

About aiomultiprocess More use of , You can refer to its official documentation [1].

reference

[1]  Official documents : https://aiomultiprocess.omnilib.dev/en/latest/guide.html

End

Cui Qingcai's new book 《Python3 Web crawler development practice （ The second edition ）》 It's officially on the market ！ The book details the use of zero basis Python Develop all aspects of reptile knowledge , At the same time, compared with the first edition, it has added JavaScript reverse 、Android reverse 、 Asynchronous crawler 、 Deep learning 、Kubernetes Related content ,‍ At the same time, this book has obtained Python The father of Guido The recommendation of , At present, this book is on sale at a 20% discount ！

Content introduction ：《Python3 Web crawler development practice （ The second edition ）》 Content introduction

Scan purchase

Haowen and his friends watch ~