What is a proxy IP pool and how to build it?

What is agency ip pool ？

Let's make a comparison , It's a pool , There are a lot of agents in it ip. It has the following behavioral characteristics ：

1. In the pool ip There is a life cycle , They will be verified on a regular basis , Those that fail will be removed from the pool .
2. In the pool ip There are supplementary channels , There will be new agents ip Constantly being added to the pool .
3. Agents in the pool ip It can be taken out at random .
such , There are always multiple changing agents in the agent pool 、 Valid proxy ip, And we can randomly take agents out of the pool ip, Then let the crawler use the proxy ip Visit the target website , You can avoid reptiles being ban The situation of .

today , Let's talk about how to build our own agent ip pool . and , We need to make a more flexible agent pool , It provides two proxy methods ：

1. Pass every time http Interface to extract a random proxy ip, Then use this agent in the crawler ip（ Most agents ip Pool services are in this form ）
2. Use squid3 The agent does request forwarding , Crawler set squid3 The address of the agent , Each request will be made by squid3 Automatically forward to agents in the agent pool