web crawler error

When using the request method in urllib, the system returns HTTPerror, but no error code is given

Re-wrote a piece of code to display specific errors:

# exception handlingfrom urllib import request,errortry:response = urllib.request.urlopen('https://movie.douban.com/top250')except error.HTTPError as e:print(e.reason,e.code,e.headers,sep='\n') # Use httperror to judge 

I visited Douban here, and the result returned error 418. I checked that it was anti-crawling.

Processing method: Instead of requesting the entire webpage at one time, add the header option and only read the header, as follows:

The second question is, how to read the information of multiple pages. At this time, through observation, we know that the page link of douban contains page number information, and the for loop can be used to match the page number: