Part 1 ： It's from senior brother Ali 170 Avenue Python Interview questions , I have landed successfully 【Python The basic chapter 】
Comprehensive part ： Network programming
101. sketch OSI Seven tier protocol
102. Three handshakes 、 The process of four waves
103. What is? C/S and B/S framework
104. TCP and UDP The difference between
105. LAN and WAN
106. arp agreement
107. What is? socket？ The brief is based on TCP Protocol socket communication process
108. sketch process 、 Threads 、 The differences of coroutines and application scenarios
109. How to use thread pool and process pool
110. How processes communicate
111. Process lock and thread lock
112. What is concurrency and parallelism
113. threading.local The role of
114. What is domain name resolution
115. LVS What is it and its function
116. Nginx The role of
117. keepalived And HAProxy
118. What is? rpc
119. The process of entering a web address from the browser to displaying the web address page
120. What is? cdn
It's network transport protocol , Artificially divide different stages of network transmission into different levels
The seven layers are divided into ： application layer 、 The presentation layer 、 The session layer 、 Transport layer 、 The network layer 、 Data link layer 、 The physical layer
The five layers are divided into ： application layer 、 Transport layer 、 The network layer 、 Data link layer 、 The physical layer
The physical layer ： Ethernet cable , Cables and other physical equipment
Data link layer ：Mac Address
The network layer ：IP Address
Transport layer ：TCP,UDP agreement
application layer ：FTP agreement ,Email,WWW etc.
All happen at the transport layer
Three handshakes ：
TCP Protocol is the transmission control protocol from host to host layer , Provide reliable connection service , Use three handshakes to establish a connection .TCP Sign a （ Bit code ）, Yes 6 Species mark :SYN(synchronous Establish online ) ACK(acknowledgement confirm ) PSH(push delivery ) FIN(finish end ) RST(reset Reset ) URG(urgent emergency ) Sequence number( Sequence number ) Acknowledge number( Confirmation number ) The first handshake ： host A The sending bit code is syn＝1, Randomly generated seq number=1234567 Of packets to the server , And enter SYN_SEND state , host B from SYN=1 know ,A Online required
The second handshake ： host B Confirm the online information after receiving the request , towards A send out ack number=( host A Of seq+1),syn=1,ack=1, Randomly generated seq=7654321 My bag , And enter SYN_RECV state
The third handshake ： host A Check after receiving ack number Whether it is right , First sent seq number+1, Bit code ack Is it 1, If correct , host A It will be sent again. ack number=( host B Of seq+1),ack=1, host B Confirmation upon receipt seq Value and ack=1 Connection established successfully , Both hosts enter ESTABLISHED state
Complete the above three handshakes , host A With the host B Start sending data
Four waves ：
because TCP Connection is full duplex , Therefore, each direction must be closed separately . This principle is that when a party completes its data transmission task, it can send a FIN To terminate the connection in this direction . Receive a FIN It just means there's no data flow in this direction , One TCP Connect to receive a FIN Can still send data after . Active shutdown will be performed by the first party to close , And the other party performs a passive shutdown
The server A Send a FIN, Used to close A To the server B Data transfer of . The server B Receiving this FIN, It sends back a ACK, The confirmation serial number is the received serial number plus 1. and SYN equally , One FIN Will occupy a sequence number
The server B Shut down the connection with the server A The connection of , Send a FIN To server A
The server A remand ACK Message confirmation , And set the confirmation serial number to the received serial number plus 1
B/S Also known as browser / Server mode . For example, various websites ,jupyter notebook etc. . advantage ： Zero installation , Simple maintenance , Good sharing . shortcoming ： Poor safety , Lack of personalization
C/S Also known as the client / Server mode . For example, wechat client ,Oracle Client, etc . advantage ： Good safety , Fast data transmission , Stable . shortcoming ： Yes PC Machine operating system, etc , When there are many clients , The server side is heavily loaded
TCP and UDP All are OSI The protocol of transport layer in the model .TCP Provide reliable communication transmission , and UDP It is often used for the communication transmission of broadcasting and detail control to the application .UDP No complex control mechanism , utilize IP Provide connectionless communication services .TCP It fully realizes various control functions during data transmission , Retransmission control of packet loss can be carried out , It can also control the order of the disordered subcontracting
TCP application ：FTP transmission , Point to point text messages, etc
UDP application ： Media streaming, etc
Wide area network （WAN,Wide Area Network） Also known as remote network （long haul network ）. Usually across a large physical range , It covers tens to thousands of kilometers , It can connect multiple cities or countries , Or across several continents and providing long-distance communication , Forming an international remote network
Domain network （Local Area Network,LAN） It refers to a group of computers interconnected by multiple computers in a certain area . It's usually within a few thousand meters . LAN can realize file management 、 Application software sharing 、 Printer sharing 、 Schedule within the working group 、 E-mail and fax communication services . LAN is closed , It can be made up of two computers in the office , It can also be made up of thousands of computers in a company
ARP（Address Resolution Protocol） Address resolution protocol , Used to implement from IP Address to MAC Address mapping , Ask the target IP Corresponding MAC Address
socket It's right TCP/IP Encapsulation of protocol , It just makes it easier for programmers to use TCP/IP The protocol stack .socket It's not an agreement in itself , It's the application layer and TCP/IP Intermediate software abstraction layer for protocol family communication , Is a set of calling interfaces （TCP/IP Online API function ）
“TCP/IP It's just a protocol stack , Just like the operating system's operating mechanism , It must be realized concretely , At the same time, it also provides external operation interface . It's like the operating system will provide a standard programming interface , such as win32 Programming interface is the same .TCP/IP It is also necessary to provide an interface for programmers to do network development , This is it. Socket Programming interface (API) .”
Server:
import socket
import threading
def tcplink(sock, addr):
print('Accept new connection from %s:%s...' % addr)
sock.send(b'Welcome!')
while True:
data = sock.recv(1024)
time.sleep(1)
if not data or data.decode('utf-8') == 'exit':
break
sock.send(('Hello, %s!' % data.decode('utf-8')).encode('utf-8'))
sock.close()
print('Connection from %s:%s closed.' % addr)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Listening port :
s.bind(('127.0.0.1', 9999))
s.listen(5)
print('Waiting for connection...')
while True:
# Accept a new connection :
sock, addr = s.accept()
# Create a new thread to handle TCP Connect :
t = threading.Thread(target=tcplink, args=(sock, addr))
t.start()
Client:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Establishing a connection :
s.connect(('127.0.0.1', 9999))
# Receive a welcome message :
print(s.recv(1024).decode('utf-8'))
for data in [b'Michael', b'Tracy', b'Sarah']:
# send data :
s.send(data)
print(s.recv(1024).decode('utf-8'))
s.send(b'exit')
s.close()
The example comes from Liao Xuefeng's official website
A process is a program with some independent functions. It is a running activity on a data set , A process is an independent unit of the system for resource allocation and scheduling . Each process has its own independent memory space , Different processes communicate through inter process communication
A thread is an entity of a process , yes CPU Basic unit of dispatch and dispatch , It's a smaller, independent, basic unit than a process . The thread itself basically does not have system resources , Have only a few essential resources in operation ( Such as program counter , A set of registers and stacks ), But it can share all the resources owned by a process with other threads belonging to the same process
Coroutine is a lightweight thread in user mode , The scheduling of the process is completely controlled by the user . It has its own register context and stack
Multi process ： Concentrated CPU Mission , We need to make full use of multicore CPU resources （ The server , A lot of parallel computing ） When , With multiple processes . defects ： The cost of communication between multiple processes is high , Switching costs a lot
Multithreading ： Concentrated I/O Mission （ The Internet I/O, disk I/O, database I/O） It's appropriate to use multithreading . defects ： Only one thread can run in the same time slice , It can't be high parallel , But it can achieve high concurrency
coroutines ： Also called tasklet , Performing multiple tasks on a single thread , Switch with functions , The cost is minimal . Not scheduled through the operating system , No process 、 Thread switching overhead . defects ： Single thread execution , Processing intensive CPU And local disks IO When , Low performance . Deal with the network I/O The performance is still quite high
Multithreaded request returns are unordered , Any thread that has data to return will be processed , The data returned by the coroutine is ordered
The function of the pool is to limit the number of processes or threads started . When the number of concurrent tasks far exceeds the capacity of the computer , That is, when too many processes or threads cannot be started at one time , We should use the concept of pool to limit the number of open processes or threads to the range that the computer can bear
Multi process
from multiprocessing import Pool
import os
import time
import random
def long_time_task(name):
print('Run task %s (%s)...' % (name, os.getpid()))
start = time.time()
time.sleep(random.random() * 3)
end = time.time()
print('Task %s runs %0.2f seconds.' % (name, (end - start)))
def test_pool():
print('Parent process %s.' % os.getpid())
p = Pool(4)
for i in range(5):
p.apply_async(long_time_task, args=(i,))
print('Waiting for all subprocesses done...')
p.close()
p.join()
print('All subprocesses done.')
if __name__ == '__main__':
test_pool()
output
Parent process 32432.
Waiting for all subprocesses done...
Run task 0 (15588)...
Run task 1 (32372)...
Run task 2 (12440)...
Run task 3 (18956)...
Task 2 runs 0.72 seconds.
Run task 4 (12440)...
Task 3 runs 0.82 seconds.
Task 1 runs 1.21 seconds.
Task 0 runs 3.00 seconds.
Task 4 runs 2.95 seconds.
All subprocesses done.
apply_async(func[, args[, kwds]]) ： Call in non blocking mode func（ Parallel execution , Blocking mode must wait for the previous process to exit to execute the next process ）,args For transmission to func Parameter list for ,kwds For transmission to func Keyword parameter list for ;close()： close Pool, Make it no longer accept new tasks ;terminate()： Whether the task is completed or not , Immediately terminate ;join()： Main process blocking , Waiting for child process to exit , Must be in close or terminate Then use
You can also use concurrent.futures Module provides functions to realize
def test_future_process():
print('Parent process %s.' % os.getpid())
p = ProcessPoolExecutor(4)
for i in range(5):
p.submit(long_time_task, i)
p.shutdown(wait=True)
print('Finish')
if __name__ == '__main__':
# test_pool()
test_future_process()
output
Parent process 29368.
Run task 0 (32148)...
Run task 1 (31552)...
Run task 2 (24012)...
Run task 3 (29408)...
Task 2 runs 0.52 seconds.
Run task 4 (24012)...
Task 3 runs 0.86 seconds.
Task 1 runs 1.81 seconds.
Task 0 runs 1.83 seconds.
Task 4 runs 1.69 seconds.
Finish
Multithreading
def sayhello(a):
print("hello: " + a)
start = time.time()
time.sleep(random.random() * 3)
end = time.time()
print('Task %s runs %0.2f seconds.' % (a, (end - start)))
def test_future_thread():
seed = ["a", "b", "c", "d"]
start = time.time()
with ThreadPoolExecutor(3) as executor:
for i in seed:
executor.submit(sayhello, i)
end = time.time()
print("Thread Run Time: " + str(end - start))
output
hello: a
hello: b
hello: c
Task a runs 0.40 seconds.
hello: d
Task b runs 0.56 seconds.
Task d runs 1.70 seconds.
Task c runs 2.92 seconds.
Thread Run Time: 2.9195945262908936
It can be seen that , Because it is created, it is limited to 3 Thread pool of , So only three tasks are performed at the same time
def write(q):
print("write(%s), Parent process is (%s)" % (os.getpid(), os.getppid()))
for i in "Python":
print("Put %s to Queue" % i)
q.put(i)
def read(q):
print("read(%s), Parent process is (%s)" % (os.getpid(), os.getppid()))
for i in range(q.qsize()):
print("read from Queue Get message ： %s" % q.get(True))
def test_commun():
print("(%s) start" % os.getpid())
q = Manager().Queue()
pw = Process(target=write, args=(q, ))
pr = Process(target=read, args=(q, ))
pw.start()
pr.start()
pw.join()
pr.terminate()
output
(23544) start
write(29856), Parent process is (23544)
Put P to Queue
Put y to Queue
Put t to Queue
Put h to Queue
Put o to Queue
Put n to Queue
read(25016), Parent process is (23544)
read from Queue Get message ：P
read from Queue Get message ：y
read from Queue Get message ：t
read from Queue Get message ：h
read from Queue Get message ：o
read from Queue Get message ：n
Python Of multiprocessing Modules wrap the underlying mechanism , Provides Queue、Pipes And so on
Process lock ： Is to control multiple processes in the same operating system to access a shared resource , Just because of the independence of the program , Each process cannot control the access of other processes to resources , But you can use the semaphore control of the local system . Semaphore (Semaphore), Sometimes it's called a signal light , It is a kind of facility used in multithreading environment , Can be used to ensure that two or more critical code segments are not called concurrently
Thread lock ： When multiple threads modify a shared data almost simultaneously , Synchronous control required , Thread synchronization can ensure that multiple threads can safely access competing resources ( Global content ), The simplest synchronization mechanism is to use mutex . When a thread wants to change shared data , Lock it first , At this time, the status of the resource is locked , Other threads can change , Until the thread changes the resource state to an unlocked state , That is to release resources , Other threads can lock resources again . Mutexes ensure that only one thread enters the write operation at a time . So as to ensure the security of data under multithreading
parallel ： Multiple CPU The core , Different programs are assigned to different CPU To run the . Multiple programs can be executed at the same time
Concurrent ： Single CPU The core , You can only run one program at a time in a time slice , If you need to run multiple programs , Serial execution
ThreadLocal It's called thread local variable ,ThreadLocal A copy is created in each variable , Each thread can access its own internal copy variables , Invisible to other threads , The modification will not affect other threads
Domain name resolution refers to resolving a domain name to IP Address . There are also reverse “ Inverse analysis ”, take IP adopt DNS The server finds the corresponding domain name address
DNS It's the domain name system (Domain Name System), The domain name system assigns domain names and addresses to hosts on the Internet IP Address . User's domain name address , The system will automatically transfer the domain name address to IP Address
LVS yes Linux Virtual Server Abbreviation , meaning Linux Virtual server , Is a virtual server cluster system , Load balancing server
LVS The working mode is divided into NAT Pattern 、TUN Pattern 、 as well as DR Pattern
Nginx The main function ：1、 Reverse proxy 2、 Load balancing 3、HTTP The server （ Including the separation of static and dynamic ） 4、 Forward agency
Forward agency ： In some cases , Proxy users to access the server , You need to manually set the proxy server IP And port number
Reverse proxy ： It's for proxy server , The target server that the proxy wants to access . The proxy server accepts the request , Then forward the request to the server on the internal network ( clustering ), And return the result from the server to the client , At this time, the proxy server is represented as a server
A load balancing server is similar to LVS HTTP The server is similar to Tomcat etc.
HAProxy Provide high availability 、 Load balancing , And based on TCP and HTTP Application proxy for .keepalived It is a service software to ensure high availability of cluster in cluster management , Its function is similar to heartbeat, To prevent a single point of failure
RPC Remote procedure call , That is to say, two servers A,B, An application is deployed in A Server , Want to call B Functions provided by the application on the server / Method , Because it's not in a memory space , Can't call directly , We need to express the semantics and data of the call through the network
Browser pass DNS The server found the corresponding domain name IP Address
The browser gives IP Corresponding web Server send HTTP request
web The server receives HTTP After the request , Return the response to the browser
After the browser receives the response, render the page
CDN The full name is Content Delivery Network, The content distribution network .CDN It's a content distribution network built on the Internet , Rely on edge servers deployed everywhere , Load balancing through the central platform 、 content distribution 、 Scheduling and other functional modules , Let users get the content they need nearby , Reduce network congestion , Improve user access response speed and hit rate .CDN The key technologies are content storage and distribution technology
