Aiotor - a pool of proxies, shifting on each request

Overview

▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ░████████╗░██████╗░██████╗░░█████╗░██╗░██████╗░░ ░╚══██╔══╝██╔═══██╗██╔══██╗██╔══██╗██║██╔═══██╗░ ░░░░██║░░░██║░░░██║██████╔╝███████║██║██║░░░██║░ ░░░░██║░░░██║░░░██║██╔══██╗██╔══██║██║██║░░░██║░ ░░░░██║░░░╚██████╔╝██║░░██║██║░░██║██║╚██████╔╝░ ░░░░╚═╝░░░░╚═════╝░╚═╝░░╚═╝╚═╝░░╚═╝╚═╝░╚═════╝░░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒▓▒

Aiotor - a pool of proxies, shifting on each request

What?

using aiotor you can easily create a small pool of tor proxies so you can take advantage of having multiple circuits at your disposal and multiple ip addresses, while running just 1 process. It also brings a ClientSession, which is the ClientSession from aiohttp with a small modification: it shifts proxies at every request.

Does it work for requests package? and urllib? and plain sockets ?

yes! aitor.Pool() is a context manager which, while in context, patches the low level socket library, so it's usage is easy and compatible for almost all setups/libraries.

Cleaning up

Pool will automatically cleanup the temp folder after your program ends. the folder is needed for tor to store circuit and consensus data. if you specify a folder explicitly, like Pool(user_data_dir="c:\some\folder\tordata"), it is assumed you want a dedicated folder. in that case cleanup will not happen. using a dedicated folder will greatly improve loading time on next runs.

It is blocking!

only once! aiotor should ideally be started on your mainthread. it will block to bootstrap the tor connections, and this only happens once during the full lifecycle of your program. while subprocesses are supported by asyncio, it would render useless as proxies can't be used anyway, until initialization is finished. It also brings a lot of other unexpected behavior.

aiotor.Pool is a true singleton. Every Pool() you create will refer to the same instance. If you need more proxies, just use the amount parameter (defaults to 10, which is more than sufficient, and actually too much).

proxies in the pool are refreshed at least once 15 minutes, getting new circuits from different countries, and new ip's.

Why another library around Onion/Tor?

for most people, setting up tor is already quite a task, implementing them as proxies in their programs requires much more work. the purpose of aiotor is to make it as easy as it can possibly get.

Show me how easy it is!

import logging
import asyncio
import aiotor

async def main():
    async with aiotor.ClientSession() as session:
        for _ in range(10):
            async with session.get('http://httpbin.org/ip') as response:
                print ( await response.json() )


logging.basicConfig(level=10)
asyncio.run(main())

this will output

DEBUG:asyncio:Using proactor: IocpProactor
INFO:pool:starting proxy pool 0x2a214dc9be0
INFO:pool:tor bootstrap 0.0 completed
INFO:pool:tor bootstrap 5.0 completed
INFO:pool:tor bootstrap 10.0 completed
INFO:pool:tor bootstrap 14.0 completed
INFO:pool:tor bootstrap 15.0 completed
INFO:pool:tor bootstrap 20.0 completed
INFO:pool:tor bootstrap 25.0 completed
INFO:pool:tor bootstrap 30.0 completed
INFO:pool:tor bootstrap 40.0 completed
INFO:pool:tor bootstrap 45.0 completed
INFO:pool:tor bootstrap 50.0 completed
INFO:pool:tor bootstrap 55.0 completed
INFO:pool:tor bootstrap 60.0 completed
INFO:pool:tor bootstrap 68.0 completed
INFO:pool:tor bootstrap 75.0 completed
INFO:pool:tor bootstrap 80.0 completed
INFO:pool:tor bootstrap 85.0 completed
INFO:pool:tor bootstrap 89.0 completed
INFO:pool:tor bootstrap 90.0 completed
INFO:pool:tor bootstrap 95.0 completed
INFO:pool:tor bootstrap 100.0 completed

DEBUG:aiotor._client_session:proxy switched
{'origin': '109.70.100.32'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '107.189.31.241'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '89.163.143.8'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '45.153.160.133'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '104.244.75.33'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '199.249.230.187'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '185.220.102.245'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '213.61.215.54'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '193.189.100.199'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '176.10.99.200'}

Lets run this again

asyncio.run(main())
# no initialization is done
# it returns immediately 

DEBUG:aiotor._client_session:proxy switched
{'origin': '109.70.100.32'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '107.189.31.241'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '89.163.143.8'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '45.153.160.133'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '104.244.75.33'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '199.249.230.187'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '185.220.102.245'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '213.61.215.54'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '193.189.100.199'}
DEBUG:aiotor._client_session:proxy switched
{'origin': '176.10.99.200'}

Showing different ways to get a pool and showing how to use the pool as context manager

import aiotor

pool = aiotor.start()   # this gives you a running pool

pool2 = aiotor.Pool()   # this gives you the same pool (because singleton) if already bootstrapped

pool2.start()           # will start and bootstrap the pool, if not already running
                        # start() has the pool itself as return value

pool3 = aiotor.Pool.get_instance()  # same story

pool4 = aiotor.start()   # same story


import requests

# context manager

with pool:       
    # in this block, all network will be handled by the proxies and it will return 
    # different ip's on subsequent calls
    print(requests.get('http://httpbin.org/ip').text)

# not in context
# you will see your own ip
print(requests.get('http://httpbin.org/ip').text)

Owner
Leon
DevOps | Reverse-engineering (mobile) apps | Hacking (certified) | IDA, Frida, RetDec, Ghidra, Simplify | Python, Flutter/Dart, PHP, js , Powershell, html5
Leon
Proxlist - Retrieve proxy servers.

Finding and storing a list of proxies can be taxing - especially ones that are free and may not work only minutes from now. proxlist will validate the proxy and return a rotating random proxy to you

Justin Hammond 2 Mar 17, 2022
A project that forwards data it receives in a URL POST Request to a Discord Webhook link

Mailman Mailman is a project that basically just forwards data it receives in a URL POST Request to a Discord Webhook link and act as a sort of messag

Prakhar Trivedi 2 Mar 14, 2022
A Python package for handling free proxies from sslproxies.org

SSLProxies Get free working proxy from https://www.sslproxies.org/ and use it in your script This is a port/rewrite of free-proxy with additional feat

Nate Harris 2 Mar 17, 2022
A light-weight open-source project CLI utility for showing services running on ports in a host

Portable Port Scanner (ppscanner) Portable Port Scanner (ppscanner) is a light-weight open-source CLI utility that leverages on nmap to make quick and

1 Oct 30, 2021
TicTacToe using Socket Server

TicTacToe using Socket Server This is a project for the class : 18CSC302J - Computer Networks by Dr. S.Babu Contributors Suvodeep Sinha RA191100301010

Suvodeep Sinha 12 Nov 30, 2022
Ipscanner - A simple threaded IP-Scanner written in python3 that can monitor local IP's in your network

IPScanner 🔬 A simple threaded IP-Scanner written in python3 that can monitor lo

4 Dec 12, 2022
EUserv - A Python script which can help you renew your free EUserv IPv6 VPS

English | 简体中文 This project comes from https://github.com/a-beam-of-light/eu_ex

阿两 0 Jan 06, 2022
ThorFI: A Novel Approach for Network Fault Injection as a Service

ThorFI: a Novel Approach for Network Fault Injection as a Service This repo includes ThorFI, a novel fault injection solution for virtual networks in

DESSERT research lab (Federico II University of Naples, Italy) 6 Dec 14, 2022
this is demo of tool dosploit for test and dos in network with python

this tool for dos and pentest vul SKILLS: syn flood udp flood $ git clone https://github.com/amicheh/demo_dosploit/ $ cd demo_dosploit $ python3 -m pi

yaser amir chehrazi 5 Sep 22, 2022
A database-based CDN node supporting PostgreSQL and MongoDB backends.

A simple to use database-based deployable CDN node for hobbyist developers who wish to have their own CDN!

Vish M 10 Nov 19, 2022
TsuserverMoS - A Python-based server for Attorney Online,

tsuserverMoS A Python-based server for Attorney Online, forked from RealKaiser/tsuserverCC Requires Python 3.7+ and PyYAML. Changes/additions from tsu

1 Dec 30, 2021
A repository dedicated to IoT(internet of things ) and python scripts

📑 Introduction Week of Learning is a weekly program in which you will get all the necessary knowledge about Circuit-Building, Arduino and Micro-Contr

27 Nov 22, 2022
Converts Cisco formatted MAC Addresses to PC formatted MAC Addresses

Cisco-MAC-to-PC-MAC Converts a file with a list of Cisco formatted MAC Addresses to PC formatted MAC Addresses... Ex: abcd.efgh.ijkl to AB:CD:EF:GH:I

Stew Alexander 0 Jan 04, 2022
Tiny JSON RPC via HTTP library.

jrpc Simplest ever possible Asynchronous JSON RPC via HTTP library for Python, backed by httpx. Installation pip install async-jrpc Usage Import JRPC

Onigiri Team 2 Jan 31, 2022
MQTT Explorer - MQTT Subscriber client to explore topic hierarchies

mqtt-explorer MQTT Explorer - MQTT Subscriber client to explore topic hierarchies Overview The MQTT Explorer subscriber client is designed to explore

Gambit Communications, Inc. 4 Jun 19, 2022
A Python framework for interacting with Solana's Pyth network.

Pyth Network A basic Python framework for reading and decoding data regarding the Pyth network

1 Nov 29, 2021
Linkedin Connection Automation

Why spend an hour+ a week, connecting with the correct people on LinkedIn when you can go for lunch and let your computer do the hard work?

1 Nov 29, 2021
Process incoming JSON-RPC requests in Python

August 16, 2021: Version 5 has been released. Read about the changes in version 5, or read the full documentation. Version 5 is for Python 3.8+ only.

Exploding Labs 156 Dec 31, 2022
Simple client for the Sirah Matisse Commander TCP server.

Simple client for the Sirah Matisse Commander TCP server.

Nelson Darkwah Oppong 1 Nov 02, 2021
A TrueCharts automatic and bulk update utility

trueupdate A TrueCharts automatic and bulk update utility How to install run pip install trueupdate Please be aware you will need to reinstall after e

TrueCharts 125 Jan 04, 2023