Hello,

I was tinkering around with send(self, request, ...) in ip_rotator.py and noticed that print statements in the method will not show up. But, print statements in the module's other methods like start() will show in console.

I assume send(self, request, ...) is being used under the hood? If this is the case, how come the print statements don't show up? Is there any other way way to view details about the request parameter?

Thanks

There are 3 main changes in this PR. Perhaps I should have separated them out...

1. Removed creation of usage plan. Isn't required for functioning are we aren't cleaning up. 300 limit on usage policy

2. added modify_request_dict which allows passing in a dict that defines a request to be modified for use with the API Gateway. This change allowed me to easily implement an async scraper.

3. Project was previously ignoring query string paramaters in request if path was empty. Params are now explicitly extracted and appended to params to request object.

Just discovered this package and it has huge promise, however I'm not sure if something on AWS has changed since the last release.

Using the below code i always get a 403 response.

import requests
from requests_ip_rotator import ApiGateway

gateway = ApiGateway(
    "https://www.google.com",
    regions=['eu-west-1', 'eu-west-2', 'eu-west-3'],
    access_key_id="MY_KEY",
    access_key_secret="MY_SECRET"
    )
gateway.start()

session = requests.Session()
session.mount("https://www.google.com", gateway)
response = session.get("https://www.google.com/search?q=random+search")
print(response.status_code)

import requests
from requests_ip_rotator import ApiGateway, EXTRA_REGIONS

gateway = ApiGateway("https://spare.avspart.com/", access_key_id = 'my key', access_key_secret = 'my secret key')
gateway.start()

session = requests.Session()
session.mount("https://spare.avspart.com/", gateway)

response = session.get("https://spare.avspart.com/catalog/case/64848/4534337/677993/")
print(response.status_code)

gateway.shutdown()

When I try and send a request to the page I get this error. Not sure what's happening?

Starting API gateways in 10 regions.
Using 10 endpoints with name 'https://spare.avspart.com/ - IP Rotate API' (10 new).
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    response = session.get("https://spare.avspart.com/catalog/case/64848/4534337/677993/")
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 677, in send
    history = [resp for resp in gen]
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 677, in <listcomp>
    history = [resp for resp in gen]
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 166, in resolve_redirects
    raise TooManyRedirects('Exceeded {} redirects.'.format(self.max_redirects), response=resp)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

I am experiencing a strange situation:

• when making requests here: https://www.zillow.com/homes/Miami.dash.Dade-County,-FL_rb/ everything goes correctly
• when i try to make a request here instead : https://www.zillow.com/homedetails/14851-SW-150th-St-Miami-FL-33196/44327576_zpid/ I get this: {"message":"Forbidden"}

I know it has to be something related to headers because if i dont use any header I always get 403, but even using headers in the second link i cant get a 200 response. I dont know if something like this has happened to any of you. If someone knows the issue i would appreciate a lot if they let me know. Thank you!

Hello sir, very nice module :)

I am trying to make it nicer by speeding up closing and opening of gateways. Current implementation uses:

apis = awsclient.get_rest_apis()["items"] // to get the ids to close and, lists max 20 awsclient.delete_rest_api(restApiId=api["id"]) // to actually close it

Which works fine, unless you have to close 600 gateways which easily takes half an hour and you have to recursively call get_rest_apis(), because it lists only 20

The same problem happens with opening gateways. Opening gateways for 10 domains takes around 30s. I am attaching the relevant code from the project:

    `# Create simple rest API resource
    create_api_response = awsclient.create_rest_api(
        name=self.api_name,
        endpointConfiguration={
            "types": [
                "REGIONAL",
            ]
        }
    )

    # Get ID for new resource
    get_resource_response = awsclient.get_resources(
        restApiId=create_api_response["id"]
    )
    rest_api_id = create_api_response["id"]

    # Create "Resource" (wildcard proxy path)
    create_resource_response = awsclient.create_resource(
        restApiId=create_api_response["id"],
        parentId=get_resource_response["items"][0]["id"],
        pathPart="{proxy+}"
    )

    # Allow all methods to new resource
    awsclient.put_method(
        restApiId=create_api_response["id"],
        resourceId=get_resource_response["items"][0]["id"],
        httpMethod="ANY",
        authorizationType="NONE",
        requestParameters={
            "method.request.path.proxy": True,
            "method.request.header.X-My-X-Forwarded-For": True
        }
    )

    # Make new resource route traffic to new host
    awsclient.put_integration(
        restApiId=create_api_response["id"],
        resourceId=get_resource_response["items"][0]["id"],
        type="HTTP_PROXY",
        httpMethod="ANY",
        integrationHttpMethod="ANY",
        uri=self.site,
        connectionType="INTERNET",
        requestParameters={
            "integration.request.path.proxy": "method.request.path.proxy",
            "integration.request.header.X-Forwarded-For": "method.request.header.X-My-X-Forwarded-For"
        }
    )

    awsclient.put_method(
        restApiId=create_api_response["id"],
        resourceId=create_resource_response["id"],
        httpMethod="ANY",
        authorizationType="NONE",
        requestParameters={
            "method.request.path.proxy": True,
            "method.request.header.X-My-X-Forwarded-For": True
        }
    )

    awsclient.put_integration(
        restApiId=create_api_response["id"],
        resourceId=create_resource_response["id"],
        type="HTTP_PROXY",
        httpMethod="ANY",
        integrationHttpMethod="ANY",
        uri=f"{self.site}/{{proxy}}",
        connectionType="INTERNET",
        requestParameters={
            "integration.request.path.proxy": "method.request.path.proxy",
            "integration.request.header.X-Forwarded-For": "method.request.header.X-My-X-Forwarded-For"
        }
    )

    # Creates deployment resource, so that our API to be callable
    awsclient.create_deployment(
        restApiId=rest_api_id,
        stageName="ProxyStage"
    )

    # Return endpoint name and whether it show it is newly created
    return {
        "success": True,
        "endpoint": f"{rest_api_id}.execute-api.{region}.amazonaws.com",
        "new": True
    }`

Is it possible to perform opening and closing tasks in batch?

I have been attempting to use this package to scrape Google News - I am using the most recent release (v1.0.10), and have configured the AWS-CLI. The exact code sequence resulting in a failure is as follows:

1. Get blocked by Google :) (run this, and you'll likely be blocked after 750 to 1000 requests)

import requests
for i in range(1,10000):
    response = requests.get("http://www.google.com/search?q=barry+bonds after:2021-12-22 before: 2021-12-23&tbm=nws&hl=en&num=10")
    if response.status_code != 200:
        print(i)
        print(response.status_code)
        break

2. After getting blocked on my IP, I should still be able to access google using the module (i.e. after running the above block, I should be able to run the below block, and get a 200 response).

with ApiGateway("https://google.com") as g:
    session = requests.Session()
    session.mount("https://google.com", g)
    response = session.get("http://www.google.com/search?q=elon+musk after:2021-12-22 before:    2021-12-23&tbm=nws&hl=en&num=10")
    print(response.status_code)

Unfortunately, the result is a 429 response for me ... on the other hand, when I tried using a proxy from scrapingbee.com after initially getting blocked by Google (performing step 1), I actually did get a 200 response. I configured the AWS CLI, and I also tried inputting the keys as arguments and creating new users with the API Gateway enabled, as well as using the root key, but have had no luck.

Are you able to replicate this issue/first artificially block yourself from Google, and then being unable to scrape using this ip-rotator module? Thank you very much for an excellent module, and Merry Christmas and happy holidays!

hi, im running the request ip rotator in a while infinite loop. But i dont know if its really working, is it sending requests with different ip, how can i check that? im asking because i see in the terminal "IP Rotate API' (0 new)"

"Starting API gateways in 17 regions. Using 17 endpoints with name 'https://www.google.com - IP Rotate API' (0 new). running #3 - Status code: 200 - Host: 2000qnx4dm5b6.execute-api.us-east-1.amazonaws.com - Request headers X-My-X-Forwarded-For: XX.XXX.XXX.XX"

why do i see 0 new?

The site has Cloudflare protection to bypass it as I know I need to spoof tls fingerprint. How I can change tls fingerprint with code or in aws? Or mb there is another method to bypass it?

When there are many gateways on an account / region and the relevant ones aren't on the first page (25 per page) they start to accumulate since they may not be found to be deleted.

If you try to view a video, the response has status code 500 with message {"message": "Internal server error"}

You can try it out with this sample:

if __name__ == '__main__':
    gateway = ApiGateway("https://video.xx.fbcdn.net", regions=["regionhere"], access_key_id=ACCESS_KEY_ID, access_key_secret=ACCESS_KEY_SECRET)
    gateway.start()

    session = requests.Session()
    session.mount("https://video.xx.fbcdn.net", gateway)

    response = session.get("https://video.xx.fbcdn.net/v/t66.36240-2/10000000_5162062970516874_3952172667405771162_n.mp4?_nc_cat=110&_nc_ht=video.fmnl8-2.fna&_nc_ohc=RuGOgev5TvEAX8FI4Hk&_nc_sid=985c63&ccb=1-7&efg=eyJybHIiOjIzNzQsInJsYSI6MTc5MSwidmVuY29kZV90YWciOiJvZXBfaGQifQ%3D%3D&oe=628E6A4A&oh=00_AT__3AG9wy4Vz9QqimNjwzLTspBsfs0E5-VyVRaLCqGAUw&rl=2374&vabr=1583")
    print(response.status_code) // this prints 500

    # Delete gateways
    gateway.shutdown()

Also tried with: -https://previews.customer.envatousercontent.com/h264-video-previews/34bc5a70-9db3-4ee9-821d-b4304d0c5dc5/33244341.mp4

Any idea whats going on? It seems that default aws gateway configuration does not support video.

Hello @Ge0rg3! Thank you for putting together this library. It was proven to be effective over months of use.

However I'd like to report that some urls fail to fetch due to, at least I think so, how API Gateway is constructed. Take a look at this url for example:

https://scontent.xx.fbcdn.net/v/t39.25447-2/315991144_6377206065657734_5025935354679506890_n.mp4?_nc_cat=106&vs=e0ece699e5d69073&_nc_vs=HBksFQAYJEdHaWsxUktHOXpQX0NLZ1dBTXEzdjVlVHRiOUZibWRqQUFBRhUAAsgBABUAGCRHTlI1MVJKU3ZyWjZlYmtFQUM5WExHZ25HS0F6YnJGcUFBQUYVAgLIAQBLB4gScHJvZ3Jlc3NpdmVfcmVjaXBlATENc3Vic2FtcGxlX2ZwcwAQdm1hZl9lbmFibGVfbnN1YgAgbWVhc3VyZV9vcmlnaW5hbF9yZXNvbHV0aW9uX3NzaW0AKGNvbXB1dGVfc3NpbV9vbmx5X2F0X29yaWdpbmFsX3Jlc29sdXRpb24AHXVzZV9sYW5jem9zX2Zvcl92cW1fdXBzY2FsaW5nABFkaXNhYmxlX3Bvc3RfcHZxcwAVACUAHAAAJtLalcytxbYCFQIoAkMzGAt2dHNfcHJldmlldxwXQDQaXjU%2FfO4YIGRhc2hfdjRfNXNlY2dvcF9ocTFfZnJhZ18yX3ZpZGVvEgAYGHZpZGVvcy52dHMuY2FsbGJhY2sucHJvZDgSVklERU9fVklFV19SRVFVRVNUGwqIFW9lbV90YXJnZXRfZW5jb2RlX3RhZwZvZXBfaGQTb2VtX3JlcXVlc3RfdGltZV9tcwEwDG9lbV9jZmdfcnVsZQd1bm11dGVkE29lbV9yb2lfcmVhY2hfY291bnQGMTA5NTIxEW9lbV9pc19leHBlcmltZW50AAxvZW1fdmlkZW9faWQPNDcwODM4MDA1MTIwMzYxEm9lbV92aWRlb19hc3NldF9pZBAxMDk3NTM2MDE3NjIxNjU2FW9lbV92aWRlb19yZXNvdXJjZV9pZA82ODI4ODg3Mzk4NjIxODUcb2VtX3NvdXJjZV92aWRlb19lbmNvZGluZ19pZA81MTk4NzQyODMzNjkyNTEOdnRzX3JlcXVlc3RfaWQAJQIcACWOAhsIiAFzAzYzOAJjZAoyMDIyLTExLTE4A3JjYgYxMDk1MDADYXBwDFBvd2VyIEVkaXRvcgJjdBlDT05UQUlORURfUE9TVF9BVFRBQ0hNRU5UE29yaWdpbmFsX2R1cmF0aW9uX3MJMjAuMTM4NjY3AWYCYWQCdHMVcHJvZ3Jlc3NpdmVfZW5jb2RpbmdzAA%3D%3D&ccb=1-7&_nc_sid=4ea082&efg=eyJ2ZW5jb2RlX3RhZyI6Im9lcF9oZCJ9&_nc_ohc=Cfpbu-XsZBsAX87uM0c&_nc_ht=scontent.fltn3-2.fna&oh=00_AfAJY6nikmbiUBPnm8G-_aKrghM7WeryZX3b0Sk4zrarZw&oe=63A51240&_nc_rid=089394793026750

You can copy paste this url to browser and it plays as it should. However if you make a request via API Gateway, you will get URL signature mismatch error. That is the error you get from fbcdn when the url is malformed. This is a video, so you will also need to send a Range header.

url = "URL above"
headers["Range"] = f"bytes=0-9999999"
 with current_session.get(url, stream=True, headers=headers, allow_redirects=False) as response:
        with open(location["absolute_output_file"], 'wb') as f:
            for chunk in response.iter_content(chunk_size=8192):
                f.write(chunk)

I tried escaping % by replacing them with %25, but it didn't work. Passing URL(url) as request url also didn't work. I uploaded a media file on a server and used a name with %, if fetched normally so I am not sure what is the cause for a problem here. If the url is expired when you see this, I'll supply a new one. Thanks for your help.

Hi All -

Great project!

Just started testing this repo but it looks like CloudWatch stats are delayed.

I'm wondering if anyone knows whether the API Gateway charges for failed (4XX/5XX) requests?

I believe it does, but just want to be sure.

Thanks!

Hey there, thanks a lot for this great lib! I noticed that it uses the REST api. Would it theoretically be possibe to use the HTTP Api, which is only 1/3 the price?

Hey, I am back again:) I was happily parsing last few days away, blew past the free tier already, but I wanted to scale up my operation further to make it even faster. I am limited with max 60 workers in the pool, so I decided to rewrite it from multiprocessing to asynchronous concurrency. This is where I realized that I can't use Requests module, but your module was made to work only with Requests.

How difficult would it be to rewrite it to make it compatible with aiohttp? Is there any way to make it work with requests, even thou the module is inherently blocking at the sockets level? The higher latency, the more beneficial would it be to move this work from multiprocessing to asynchronous loop. I could try to work on it, but I am new to python so I appreciate any kind of feedback or advice you can give me.

I just found Dask and am looking into it, if it could help me keep using Requests. Another possibility is to rent a server that has enough virtual cores to go beyond 60 workers.