Notes on concurrent programming of vegetables (IX) asynchronous IO to realize concurrent crawler acceleration

Series index ： Concurrent programming notes of vegetables | Python Concurrent programming details （ Continuous updating ~）

List of articles

• • One 、 Mind mapping
  • Two 、 What is a journey ？
  • 3、 ... and 、Python asynchronous IO Library Introduction ：asyncio
  • Four 、 The power of asynchronous programming
  • 5、 ... and 、 The core principle of asynchronous programming
  • 6、 ... and 、 Asynchronous programming code example
  • 7、 ... and 、 Application of semaphore mechanism

One 、 Mind mapping

Two 、 What is a journey ？

coroutines ： My personal understanding is that when there is a need IO In operation ,CPU To perform other procedures , etc. IO Restore after completion CPU Use . When using a collaborative process, you must use loop = asyncio.get_event_loop() Create a super loop .

3、 ... and 、Python asynchronous IO Library Introduction ：asyncio

There are several important keywords , One is async, The function used to declare is An asynchronous function , also await Is to meet IO Block pending to execute await Functions inside ,tasks You can also use... In the list ensure_future To create . The last line is to execute the crawler until tasks Complete .

Four 、 The power of asynchronous programming

• Nginx As Web The server ：
  Beat the synchronization blocking server Apache, Support more concurrent connections with fewer resources , Higher efficiency , Be able to support up to 50,000 Responses for the number of concurrent connections , Use epoll and kqueue As a development model
• Redis Why so soon? ：
  Single thread is used to process network requests + Using multiple channels I/O Reuse model , Non blocking IO , Unnecessary context switches and race conditions are avoided , There is also no switching consumption caused by multiple processes or multiple threads CPU, You don't have to worry about locks , There is no lock release operation , There is no performance penalty due to possible deadlocks ;
• Node.js The advantages of ：
  Use event driven 、 Asynchronous programming , Designed for Web Services . Actually Javascript The anonymous function and closure feature of is very suitable for event driven 、 Asynchronous programming .Node.js Non blocking mode IO Processing brings high performance and outstanding load capacity under relatively low system resource consumption , It's perfect for relying on other IO The middle layer services of resources
• Go One advantage of language ：
  Go Use Goroutine and channel It provides a lightweight syntax for generating co procedures and using channels , It makes it very easy to write high concurrent server software , In many cases, we don't need to consider the lock mechanism and all kinds of problems , comparison Python Single Go Applications can also effectively use multiple CPU nucleus , The performance of parallel execution is good

5、 ... and 、 The core principle of asynchronous programming

The core principle 1： Super cycle

 Achieve concurrency in a single thread , With a super loop （ In fact, that is while true） loop , Inside, each poll handles all task

The core principle 2：IO Multiplexing

 It's a kind of synchronization IO Model , A thread can monitor multiple file handles ;
 Once a file handle is ready , The application can be informed of the corresponding read and write operations ;
 No file handle is ready to block the application , hand over cpu
 Multichannel means network connection , Reuse refers to the same thread 

3 Type of implementation , Namely select,poll,epool

• select

   data structure ：bitmap
   maximum connection ：1024
  fd Copy ： Every time you call select Copy 
   The work efficiency ： polling O(N)
  

• poll

   data structure ： Array 
   maximum connection ： There is no upper limit 
  fd Copy ： Every time you call poll Copy 
   The work efficiency ： polling O(N)
  

• epool

   data structure ： Red and black trees 
   maximum connection ： There is no upper limit 
  fd Copy ：fd Call for the first time epool_ctl Copy , Every time you call epoll_wait No copy 
   The work efficiency ： Callback O(1)
  

6、 ... and 、 Asynchronous programming code example

Because the process does not support requests, So we use aiohttp Instead of .
In the coplanar function , Can pass await Syntax to suspend its own coroutine , And wait for another collaboration process to complete until the result is returned . It should be noted that ,await Grammar can only appear through async In the modified function , Otherwise it will be reported SyntaxError error .

import asyncio
import aiohttp
import blog_spider

async def async_craw(url):
    print("craw url: ", url)
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            result = await resp.text()
            print(f"craw url: {url}, {len(result)}")

loop = asyncio.get_event_loop()

tasks = [
    loop.create_task(async_craw(url))
    for url in blog_spider.urls]

import time

start = time.time()
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()
print("use time seconds: ", end - start)

7、 ... and 、 Application of semaphore mechanism

We mainly use semaphore mechanism to control the number of processes , The next three add # The line is the difference from the previous code , You can learn by comparison .

import asyncio
import aiohttp
import blog_spider

semaphore = asyncio.Semaphore(10)#

async def async_craw(url):
    async with semaphore:#
        print("craw url: ", url)
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as resp:
                result = await resp.text()
                await asyncio.sleep(5)#
                print(f"craw url: {url}, {len(result)}")

loop = asyncio.get_event_loop()

tasks = [
    loop.create_task(async_craw(url))
    for url in blog_spider.urls]

import time

start = time.time()
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()
print("use time seconds: ", end - start)

In the next article, we will analyze Gevent and asyncio The difference between , And explain how to use Gevent Transform an Asynchronous Server .

Python Advanced concurrent programming is continuously updated , welcome Like collection ＋ Focus on

Last one ： Concurrent programming notes of vegetables |（ 8、 ... and ） Using multiple processes multiprocessing Project development
Next ： Concurrent programming notes of vegetables |（ Ten ） Asynchronous programming library Asyncio and Gevent Comparison of 、 Use Gevent Transform asynchronous server

My level is limited , Please comment and correct the deficiencies in the article in the comment area below ~

If feelings help you , Point a praise Give me a hand ~

Share... From time to time Interesting 、 Have a material 、 Nutritious content , welcome Subscribe to follow My blog , Looking forward to meeting you here ~