requests Module function : Impersonate a browser to send a request .
Use :(requests Module coding process )
- Guide pack import requests
- Appoint url
- Initiate request (get,post)
- Get the data of the response ( page )
- Persistent storage ( Store response data )
Case study : Crawl the page data of Sogou homepage :
# Crawling search dog home page
import requests
url = "http://www.sogou.com"
#step2: Initiate request
response=requests.get(url=url)
#step3: Get response data .text Return response data of string type
page_text = response.text
print(page_text)
#step4: Persistent storage
with open('./sogou.html','w',encoding='utf-8') as fp:
fp.write(page_text)
print(" End of crawling data ")