From simple to deep: learning journey of Xiaobai reptile (1)

 

Catalog

 

1. Google browser developer mode

2.requests Library to get static web page data

3.requests Library to get dynamic web page information

4.urllib Get web data

---

1. Google browser developer mode

How to open :F12/ Right click to check ( Or shortcut keys CTRL+Shift+i) Call out

The four most commonly used function modules of developer tools : Elements , Console , Source code , The Internet .

Elements (Elements): For viewing HTML Elements

The Internet (network): Web pages are mainly used to view header And other information related to network connection .

2.requests Library to get static web page data

requests library

Generate request :

In the use of get After obtaining the network request, you will get a response object

response Object's 5 Attributes :

Get Method to obtain online resources, there will be the following two states :

• Request header Settings

When visiting the website, if the website reports 400 Error of , It shows that the website has anti crawler strategy , Usually by setting headers Medium USER-Agent solve (header It's in the form of a dictionary , There are Key, And corresponding Value value ):

• Timeout Set up

Purpose : Avoid permanent loss of response due to waiting for a response from the server .( The setting should be reasonable , Reduce waiting time )

  

• Generate complete HTTP request

complete HTTP request : link , Request header , Timeout time , Status code , Correct coding format

3.requests Library to get dynamic web page information

• Static web page : Web content and HTML The source code is the same

• Dynamic web pages : Web content and HTML The source code is inconsistent

How to judge the type of web page :

• Right click , View page source code
• F12, Developer tools to view the source code

if 1,2 The same is a static web page , if 1,2 The difference is dynamic web pages        

Determine the type of web page before obtaining web page data , Dynamic web page data acquisition needs to find the correct web page address .             

4.urllib Get web data

• urllib library

urllib Kuo is python Built in HTTP Request Library

urllib Library in python2 and python3 It's different

stay python2 in :import urllib2

stay python3 in :import urllib.request

• urllib Library crawls a web page

• urlopen() Method parameters

use urlopen() Method to crawl web pages

• urlopen() Use of parameters

1. data Parameters :

Must be a bytes object

Must be in standard format , Use urllib.parse.urlencode() Change the format

The default value is None

send out post When asked ,data Parameters must have

1. timeout Parameters :

Timeout time

The unit is in seconds

When using agents , Check the status of the agent