coroutines (Coroutine), Also called tasklet , fibers .( Coroutine is a lightweight thread in user mode )
effect ： In execution A Function , Can interrupt at any time , To carry out B function , Then interrupt B function , Carry on A function （ It can switch automatically ）, But this is not a function call （ No call statement ）, The process is much like multithreading , However, the coroutine has only one thread executing
Popular understanding ： Some function in a thread , Information such as temporary variables of the current function can be saved anywhere , Then switch to another function to execute , Note that this is not done by calling a function , And how often to switch and when to switch back to the original function is up to the developer
When implementing multitasking , Thread switching goes far beyond save and restore at the system level CPU The context is so simple . The operating system has its own cache for each thread to run efficiently Cache Data, etc. , The operating system also does the data recovery for you . So switching between threads is very expensive . However, the switching of collaborative process is only simple operation CPU The context of , So you switch it a million times a second and the system works .
Concurrency must be implemented in only one single thread
greenlet, It's a third-party module , Used to implement coroutine code （Gevent Xiecheng is based on greenlet Realization ）yield, generator , With the help of the characteristics of the generator, you can also implement the co program code .asyncio, stay Python3.4 The module introduced in is used to write the coroutine code .async & awiat, stay Python3.5 Two keywords introduced in , combination asyncio Module can be more convenient to write code （ recommend ）. There are many ways to implement coprocessing , Now the most popular way is async&await, Other ways to understand , This article introduces the most popular way
Using coroutines requires understanding 2 individual , Event loops and define coprogramming functions
Event loop is an effective way to deal with multi concurrency , It can be understood as a dead cycle , Loop to detect and execute some code , Let's look at the following pseudo code
Task list = [ Mission 1, Mission 2, Mission 3............]
while True:
List of executable tasks , List of completed tasks = Go to the task list and check all the tasks , take ' Executable ' and ' Completed ' My task is to return to
for Ready for mission in List of executable tasks ：
Perform the tasks that are ready
for Completed tasks in List of completed tasks ：
Remove... From the task list Completed tasks
If all the tasks in the task list have been completed , Then stop the cycle
The pseudo code above means ： Get events in the loop , And then keep listening to the task list , If you have a task, just carry it out , Remove the completed task , Until all tasks in the task list are completed , End cycle
The benefits of using event loops ： So that programmers do not have to control the addition of tasks 、 Delete and event control
The code is written as follows ：
import asyncio
# Get the event loop
loop = asyncio.get_event_loop()
# Put the task on ` Task list `, Listen to the event loop
loop.run_until_complete( Mission )
# Closing event
loop.close()
You want to define a coprogram function , Format ：async def Function name
Coroutine object ： Execute the coprogram function () Get the coroutine object
# Define the coprogram function
async def func():
pass
# Create a collaboration object
result = func()
Be careful ： Execute the coprogram function , Create a collaboration object , Function code doesn't run , If you want to run the internal code of the coroutine function , You have to give the coroutine object to the event loop to handle , Look at the code below
import asyncio
async def func():
print(" Hello ")
result = func()
# The way 1
loop = asyncio.get_event_loop()
loop.run_until_complete(result)
# The way 2
asyncio.run(result) # python3.7 How to write it
await Is a keyword that can only be used in coprogramming functions , Used to meet IO Suspend during operation The current schedule （ Mission ）, The current schedule （ Mission ） Pending process The event loop can be used to execute other coroutines （ Mission ）, The current schedule IO When processing is complete , You can switch back again to execute await Later code .
give an example ： We created 2 A mission , A download picture , A download video , Let's do the picture download task first , And then I met io operation , Normally, it will wait for the picture to download , but await You can suspend the task of downloading pictures first , Then automatically switch to download video task
Usage method ：await + Can wait for the object （ Coroutine object 、Future object 、Task object ）
Case study 1
import asyncio
async def func():
print(" Execute the internal code of the coroutine function ")
# encounter IO Operation suspends the current orchestration （ Mission ）, etc. IO After the operation is completed, continue to execute .
# When the current schedule is suspended , The event loop can be used to execute other coroutines （ Mission ）.
response = await asyncio.sleep(2)
print("IO End of request , The result is ：", response)
result = func()
asyncio.run(result)
Case list 2
import asyncio
async def others():
print("start") # ④ Print start
await asyncio.sleep(2) # ⑤ Waiting time 2 second , In this process, you can switch to other coroutines
print("end") # ⑥ Print end
return ' Return value '
async def func():
print(" Execute the internal code of the coroutine function ") # ② Execute the coprogram function , Print print Code
response = await others() # ③ Wait for the coprogram function others
print(f"io End of request , The result is {response}") # ⑦ wait for others Print it when you're done print sentence
if __name__ == '__main__':
asyncio.run(func()) # ① The coprogram function runs in an event loop
All of the above examples just create a task , namely ： There is only one task in the task list of the event loop , So in IO You can't demonstrate the effect of switching to another task while waiting . When the program wants to create multiple task objects , Need to use Task Object to implement .
Tasks For concurrent scheduling coroutines , adopt asyncio.create_task( Coroutine object ) The way to create Task object , This allows the coroutine to join the event loop and wait for the scheduled execution . Besides using asyncio.create_task() Function , You can also use lower level loop.create_task() or ensure_future() function . Manual instantiation is not recommended Task object .
In essence, it encapsulates the coroutine object into task object , And join the coroutine into the event loop immediately , At the same time, track the state of the coroutine .
Be careful ：asyncio.create_task() Function in Python 3.7 Is added to . stay Python 3.7 Before , You can switch to a lower level asyncio.ensure_future() function .
Case study 1
import asyncio
async def func():
print(1)
await asyncio.sleep(2)
print(2)
return " Return value "
async def main():
print("main Start ")
# Create coroutines , Encapsulate the coroutine into a Task Object and immediately added to the task list of the event loop , Wait for the event loop to execute （ The default is ready state ）.
task1 = asyncio.create_task(func())
# Create coroutines , Encapsulate the coroutine into a Task Object and immediately added to the task list of the event loop , Wait for the event loop to execute （ The default is ready state ）.
task2 = asyncio.create_task(func())
print("main end ")
# When executing a coroutine, you encounter IO In operation , Will automatically switch to perform other tasks .
# Here await It is to wait for all the corresponding coroutines to be executed and get the results
ret1 = await task1
ret2 = await task2
print(ret1, ret2)
asyncio.run(main())
Case study await+ Task list （ Most used ）
import asyncio
async def func():
print(1)
await asyncio.sleep(2)
print(2)
return " Return value "
async def main():
print("main Start ")
# Create coroutines , Encapsulate the coroutine into Task Object and added to the task list of the event loop , Wait for the event loop to execute （ The default is ready state ）.
# Calling
task_list = [asyncio.create_task(func()), asyncio.create_task(func())]
print("main end ")
# When executing a coroutine, you encounter IO In operation , Will automatically switch to perform other tasks .
# Here await It's waiting for all coroutines to be executed , And save the return values of all coroutines to done
# If set timeout value , It means the maximum number of seconds to wait here , The return value of the completed coroutine is written to done in , If not, write pending in .
done, pending = await asyncio.wait(task_list)
print(done)
asyncio.run(main())
Be careful ： asyncio.wait Inside the source code, each of the coroutines in the list is executed ensure_future So it can be encapsulated as Task object , So I'm talking to wait When used in combination task_list The value of is [func(),func()] It's OK, too .
asyncio Medium Future Object is a relatively lower level object , Usually we don't use this object directly , But directly Task Object to complete the task and track the status .（ Task yes Futrue Subclasses of ）
Future Provides us with... In asynchronous programming final result To deal with （Task Class also has the function of state processing ）
Case study 1
async def main():
# Get the current event loop
loop = asyncio.get_running_loop()
# Create a task （Future object ）, It's a task of doing nothing .
fut = loop.create_future()
# Waiting for the final result of the mission （Future object ）, If there is no result, we will wait forever .
await fut
asyncio.run(main())
The result is that the program has been waiting , Can't end
Case study 2
import asyncio
async def set_after(fut):
await asyncio.sleep(2)
fut.set_result("666")
async def main():
# Get the current event loop
loop = asyncio.get_running_loop()
# Create a task （Future object ）, No behavior , Then the task will never know when to end .
fut = loop.create_future()
# Create a task （Task object ）, The binding set_after function , The function is inside 2s after , Will give fut assignment .
# I.e. manual setting future The end result of the mission , that fut And that's it .
await loop.create_task(set_after(fut))
# wait for Future Object acquisition final result , Otherwise, just wait
data = await fut
print(data)
asyncio.run(main())
Future The object itself is bound to the function , So you want to loop events to get Future Result , You need to set it manually . and Task Object inheritance Future object , That's right Future Expand , It can be implemented after the execution of the corresponding binding function is completed , Automatic execution set_result, In order to achieve automatic end .
although , I usually use Task object , But the nature of dealing with results is based on Future Object .
stay Python Of concurrent.futures There is also one in the module Future object , This object is used for asynchronous operation based on thread pool and process pool .
import time
from concurrent.futures import Future
from concurrent.futures.thread import ThreadPoolExecutor
from concurrent.futures.process import ProcessPoolExecutor
def func(value):
time.sleep(1)
print(value)
pool = ThreadPoolExecutor(max_workers=5)
# or pool = ProcessPoolExecutor(max_workers=5)
for i in range(10):
fut = pool.submit(func, i)
print(fut)
Two Future The object is different , They are designed for different application scenarios , for example ：concurrent.futures.Future I won't support it await grammar etc. .
stay Python Provided a will to futures.Future Object packed as asyncio.Future Object function asynic.wrap_future.
Next, you must ask ： Why? python Will provide this function ？
Actually , Generally, in program development, we either use it uniformly asycio Implementation of asynchronous operation 、 Or both use the process pool and thread pool for asynchronous operations . But if Asynchrony and The process of pool / Asynchrony of thread pool Mix and match , Then you'll use this function .
import time
import asyncio
import concurrent.futures
def func1():
# Some time-consuming operation
time.sleep(2)
return "OK"
async def main():
loop = asyncio.get_running_loop()
# The way 1. Run in the default loop's executor ( Default ThreadPoolExecutor )
# First step ： Internal will call first ThreadPoolExecutor Of submit Method to apply for a thread to execute in the thread pool func1 function , And return a concurrent.futures.Future object
# The second step ： call asyncio.wrap_future take concurrent.futures.Future The object is packaged as asycio.Future object .
# because concurrent.futures.Future Object does not support await grammar , So it needs to be packaged as asycio.Future object Can be used .
fut = loop.run_in_executor(None, func1)
result = await fut
print('default thread pool', result)
# The way 2. Run in a custom thread pool:
# with concurrent.futures.ThreadPoolExecutor() as pool:
# result = await loop.run_in_executor(
# pool, func1)
# print('custom thread pool', result)
# The way 3. Run in a custom process pool:
# with concurrent.futures.ProcessPoolExecutor() as pool:
# result = await loop.run_in_executor(
# pool, func1)
# print('custom process pool', result)
asyncio.run(main())
Application scenarios ： When a project is developed with asynchronous programming of a co program , If you want to use a third-party module , When the third-party module does not support asynchronous programming in CO programming mode , You need to use this function , for example requests modular ：
import asyncio
import requests
async def download_image(url):
# Send network request , Download the pictures （ Meet the network to download pictures IO request , Automation switches to other tasks ）
print(" Start the download :", url)
loop = asyncio.get_event_loop()
# requests Module does not support asynchronous operation by default , So we use thread pool to implement .
future = loop.run_in_executor(None, requests.get, url)
response = await future
print(' Download complete ')
# Save the image to a local file
file_name = url.rsplit('_')[-1]
with open(file_name, mode='wb') as file_object:
file_object.write(response.content)
if __name__ == '__main__':
url_list = [
'https://www3.autoimg.cn/newsdfs/g26/M02/35/A9/120x90_0_autohomecar__ChsEe12AXQ6AOOH_AAFocMs8nzU621.jpg',
'https://www2.autoimg.cn/newsdfs/g30/M01/3C/E2/120x90_0_autohomecar__ChcCSV2BBICAUntfAADjJFd6800429.jpg',
'https://www3.autoimg.cn/newsdfs/g26/M0B/3C/65/120x90_0_autohomecar__ChcCP12BFCmAIO83AAGq7vK0sGY193.jpg'
]
tasks = [download_image(url) for url in url_list]
loop = asyncio.get_event_loop()
loop.run_until_complete( asyncio.wait(tasks) )
Realized __aiter__() and __anext__() Object of method .__anext__ Must return a awaitable object .async for Can handle asynchronous iterators __anext__() Method returns the waiting object , Until it triggers a StopAsyncIteration abnormal .
Can be found in async for The object used in the statement . Must pass through it __aiter__() Method returns a asynchronous iterator.
import asyncio
class Reader(object):
""" Custom asynchronous iterators （ It's also an asynchronous iterative object ） """
def __init__(self):
self.count = 0
async def readline(self):
# await asyncio.sleep(1)
self.count += 1
if self.count == 100:
return None
return self.count
def __aiter__(self):
return self
async def __anext__(self):
val = await self.readline()
if val == None:
raise StopAsyncIteration
return val
async def func():
# Create asynchronous iteratable objects
async_iter = Reader()
# async for It must be placed in async def Within the function , Otherwise syntax error .
async for item in async_iter:
print(item)
asyncio.run(func())
Asynchronous iterators don't really help much , Just supported async for It's just grammar .
Such objects are defined by __aenter__() and __aexit__() Method async with Control the environment in the statement
import asyncio
class AsyncContextManager:
def __init__(self):
self.conn = None
async def do_something(self):
# Asynchronous operation database
return 666
async def __aenter__(self):
# Asynchronously linked database
self.conn = await asyncio.sleep(1)
return self
async def __aexit__(self, exc_type, exc, tb):
# Closing database links asynchronously
await asyncio.sleep(1)
async def func():
async with AsyncContextManager() as f:
result = await f.do_something()
print(result)
asyncio.run(func())
This asynchronous context manager is useful , Usually in the development process open 、 Handle 、 close In operation , It can be dealt with in this way .
uvloop yes asyncio An alternative to the event loop in , The replacement can make asyncio Performance improvement . in fact ,uvloop than nodejs、gevent Others, such as python Asynchronous frameworks should be at least fast 2 times , The performance is comparable Go Language .
install uvloop
pip3 install uvloop
Want to use... In a project uvloop Replace asyncio The loop of events is also very simple , Just do it in the code .
import asyncio
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
# To write asyncio Code for , Consistent with the previous code .
# The automation of the internal event loop becomes uvloop
asyncio.run(...)
Be careful ： Well-known asgi uvicorn It's used internally uvloop The event loop of .
When passed python To operate redis when , link 、 Set the value 、 Get value It's all about the Internet IO request , Use asycio Asynchronous mode can be used in IO Do some other tasks while waiting , To improve performance .
install Python Asynchronous operations redis modular
pip3 install aioredis
Case study : Connect multiple redis Do something （ encounter IO Will switch to other tasks , Provides performance ）
import asyncio
import aioredis
async def execute(address, password):
print(" Start execution ", address)
# The Internet IO operation ： First, connect 77.95.4.197:6379, encounter IO The task is automatically switched , De link 77.95.4.198:6379
redis = await aioredis.create_redis_pool(address, password=password)
# The Internet IO operation ： encounter IO Will automatically switch tasks
await redis.hmset_dict('car', key1=1, key2=2, key3=3)
# The Internet IO operation ： encounter IO Will automatically switch tasks
result = await redis.hgetall('car', encoding='utf-8')
print(result)
redis.close()
# The Internet IO operation ： encounter IO Will automatically switch tasks
await redis.wait_closed()
print(" end ", address)
task_list = [
execute('redis://77.95.4.197:6379', "123456"),
execute('redis://77.95.4.198:6379', "123456")
]
asyncio.run(asyncio.wait(task_list))
When passed python To operate MySQL when , Connect 、 perform SQL、 Shutting down is all about the Internet IO request , Use asycio Asynchronous mode can be used in IO Do some other tasks while waiting , To improve performance .
install Python Asynchronous operations redis modular
pip3 install aiomysql
Case study
import asyncio
import aiomysql
async def execute(host, password):
print(" Start ", host)
# The Internet IO operation ： First, connect 77.95.40.197, encounter IO The task is automatically switched , De link 77.95.40.198:6379
conn = await aiomysql.connect(host=host, port=3306, user='root', password=password, db='mysql')
# The Internet IO operation ： encounter IO Will automatically switch tasks
cur = await conn.cursor()
# The Internet IO operation ： encounter IO Will automatically switch tasks
await cur.execute("SELECT Host,User FROM user")
# The Internet IO operation ： encounter IO Will automatically switch tasks
result = await cur.fetchall()
print(result)
# The Internet IO operation ： encounter IO Will automatically switch tasks
await cur.close()
conn.close()
print(" end ", host)
task_list = [
execute('77.95.40.197', "123456"),
execute('77.95.40.198', "123456")
]
asyncio.run(asyncio.wait(task_list))
When writing crawler applications , Need to go through the Internet IO To request target data , This situation is suitable for using asynchronous programming to improve performance , Next, we'll use aiohttp Module to achieve .
install aiohttp modular
pip3 install aiohttp
Case study
import aiohttp
import asyncio
async def fetch(session, url):
print(f" Send a request ：{url}")
async with session.get(url, verify_ssl=False) as response:
text = await response.text()
print(" Get the results ：", url, len(text))
async def main():
async with aiohttp.ClientSession() as session:
url_list = ["http://www.baidu.com", "http://www.taobao.com", "http://www.jd.com"]
tasks = [asyncio.create_task(fetch(session, url)) for url in url_list]
await asyncio.wait(tasks)
if __name__ == '__main__':
asyncio.run(main())
Concept level description of the collaboration process ( Compared with threads ): Turn to know link Threads have two issues that must be addressed : One is blocking I\O Will cause the whole process to hang : Second, due to the lack of clock blocking , Processes need to have their own ability to schedule threads . If an implementation makes each thread ...
Content of this section : Multi process coroutines Event driven and Select\Poll\Epoll asynchronous IO 1. Multi process Start multiple processes Start process in process Parent and child processes Interprocess communication Memory is not shared between different processes , In order to realize the inter process ...
Start with a reptile Python 2 The era of using generator process ,Python 3.7 Provides a new basis for asyncio and async / await Methods . First look at a simple crawler code , Reptile scrawl_pa ...
Catalog : Concurrent multithreading coroutines I/O Multiplexing ( Hang in the air , To be continued ) One . Concurrent multithreading 1. Thread description : The execution of a pipeline is a thread , A production line must belong to a workshop , The running process of a workshop is a process ( At least one in a process ...
In my last blog post, I introduced how to transform a crawler into a multi process crawler , However, this method does not significantly improve the efficiency of reptiles , And take up the computer cpu Higher , Not very good for reptiles . In this blog , I will introduce multithreading, which is widely used in crawlers + The solution of collaborative process , Personal test ...
Python Process thread coroutines GIL Closure And higher order functions ( 5、 ... and ) 1 GIL Thread global lock Thread global lock (Global Interpreter Lock), namely Python In order to ensure thread safety, an independent thread running method is adopted ...
Chapter 11 :Python Advanced programming - Coroutines and asynchronies IO Python3 Advanced core technology 97 speak note Catalog Chapter 11 :Python Advanced programming - Coroutines and asynchronies IO 11.1 Concurrent . parallel . Sync . asynchronous . Blocking . Non blocking 11.2 ...
What is a journey A coroutine can be seen as a thread in user space . Operating system alignment exists , Users need to schedule themselves . For example, the process , Threaded operating systems know they exist . If a coroutine is a thread in user space , The operating system is unknown . Why ...
http://blog.rainy.im/2016/03/10/how-the-heck-does-async-await-work-in-python-3-5/ [ translate ] Python 3.5 A study of Xiecheng ...
from :http://blog.rainy.im/2016/03/10/how-the-heck-does-async-await-work-in-python-3-5/ [ translate ] Python 3.5 ...
Seduced by teammates , I went to listen to Li Qiang C# Public elective course , I finished my homework the next day . Job requirements : 1. Pupils' addition procedure : 2. Can set difficulty freely : 3. Count the results . Write for the first time C# Program , There are many difficulties , Discuss with your teammates , Baidu and Google Go together ...
I've met... Before . But I don't know how to solve it . It took a lot of work today to find the best solution . For some complex cell It's always done in a custom way , But if it's complicated cell There's a lot of content in it . Especially image loading , There's bound to be overlap ...
There are preorder traversal and postorder traversal of binary trees , Construct a binary tree /** * Definition for binary tree * public class TreeNode { * int val; * TreeNod ...
Lunch with colleagues , During the meeting, we discussed the problem of array de duplication I immediately shared one of my common weight removal methods , Then the boss pointed out that this method is not efficient When I got home, I tested myself , I found that the method was really slow So there is this high-performance array de duplication study One . ...
1 Control Panel\Network and Internet\Network Connections right click Local Area Connection<proper ...
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/ ...
Data is available only after initialization , Otherwise, you can't get String tag = "android:switcher:"+viewPager.getId()+":"+viewPa ...
thymeleaf Grammar explanation 1. Variable output : th:text : Output a value in the page th:value : Put a value in input In the tag value in .2. Determines if the string is empty ①: Call built-in object must use # ② ...
Mac office ppt Solution to the problem that text cannot be input normally Mac Next time you start up office ppt after , When inputting text, the input method text box will flash back quickly, and the text cannot be entered normally , stay PowerPoint This kind of ...
Mina series ( Four ) And KeepAliveFilter -- The heartbeat detection Abstract : The heartbeat protocol , Yes, based on CS Pattern is a common and effective way of connection detection for system development , Recently in use MINA frame , Originally I wrote a heartbeat protocol ...
