ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete

Overview

ShadowClone

ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete.

You can make full use of the Free Tiers provided by cloud providers and supercharge your mundane cli tools with shadow clone jutsu (Naruto style)!

Installation

Please visit the wiki for installation and intial configuration instructions

Usage

⚡ python shadowclone.py -h
usage: shadowclone.py [-h] -i INPUT [-s SPLITNUM] [-o OUTPUT] -c COMMAND

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
  -s SPLITNUM, --split SPLITNUM
                        Number of lines per chunk of file
  -o OUTPUT, --output OUTPUT
  -c COMMAND, --command COMMAND
                        command to execute

How it works

We create a container image during the initial setup and register it as a runtime for our function in AWS/GCP/Azure whatever. When you execute ShadowClone on your computer, instances of that container are activated automatically and are only active for the duration of its execution. How many instances to activate is dynamically decided at runtime depending on the size of the input file provided and the split factor. The input is then split into chunks and equally distributed between all the instances to execute in parallel. For example, if your input file has 10,000 lines and you set the split factor to 100 lines, then it will be split into 100 chunks of 100 lines each and 100 instances will be run in parallel!

Features

  • Extremely fast
  • No need to maintain a VPS (or a fleet of it :))
  • Costs almost nothing per month
    • Compatible with free tiers of most cloud services
  • Cloud agnostic
    • Same script works with AWS, GCP, Azure etc.
  • Supports upto 1000 parallel invocations
  • Dynamically decide the number of invocations
  • Run any tool in parallel on the cloud
  • Pipe output to other tools

Comparison

This tool was inspired by the awesome Axiom and Fleex projects and goes beyond the concept of VPS for running the tools by using serverless functions and containers.

Features Axiom/Fleex ShadowClone
Instances 10-100s* 1000s
Cost Per instance/per minute Mostly Free**
Startup Time 4-5 minutes 2-3 seconds
Idle Cost $++ Free
On Demand Scalability No

*Most cloud providers do not allow spinning up too many instances by default, so you are limited to around 10-15 instances at max. You have to make a request to the support to increase this number.

** AWS & Azure allow 1 million invocations per month for free. Google allows 2 million invocations per month for free. You will be charged only if you go above these limits

Demo

DNS Bruteforcing using a 43mb file - 34 seconds

asciicast

Running httpx on 94K subdomains in 1 min

asciicast

References

Lithops documentation

Free Tiers

Cloud Provider Free Allowance Link
Google Functions 2 Million invocations, 400,000 GB-seconds per month Google Cloud Free Program
AWS Lambda 1 Million invocations, Up to 3.2 million seconds of compute time per month Free Cloud Computing Services - AWS Free Tier
Azure Functions 1 Million invocations Microsoft Azure Free Services

Obviously, you can make any number of function invocations per month. The table above only shows how many invocations are free.

Similar Tools

Disclaimer

This tool is designed as a proof-of-concept implementation and it's usage is intended for educational purpose only. Usage for attacking targets without prior mutual consent is illegal. It's the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program.

Using multiple API sources, create an app that allows users to filter through random locations based on their temperature range choices.

World_weather_analysis Overview Using multiple API sources, create an app that allows users to filter through random locations based on their temperat

Jason Boyer 2 Sep 16, 2022
TypeRig is a Python library aimed at simplifying the current FontLab API

TypeRig TypeRig is a Python library aimed at simplifying the current FontLab API while offering some additional functionality that is heavily biased t

Vassil Kateliev 41 Nov 02, 2022
Most Simple & Powefull web3 Trade Bot (WINDOWS LINUX) Suport BSC ETH

Most Simple & Powefull Trade Bot (WINDOWS LINUX) What Are Some Pros And Cons Of Owning A Sniper Bot? While having a sniper bot is typically an advanta

GUI BOT 4 Jan 25, 2022
BeeDrive: Open Source Privacy File Transfering System for Teams and Individual Developers

BeeDrive For privacy and convenience purposes, more and more people try to keep data on their own hardwires instead of third-party cloud services such

Xuansheng Wu 8 Oct 31, 2022
A modular Telegram Python bot running on python3 with a sqlalchemy database.

Nao Tomori Robot Found Me On Telegram As Nao Tomori 🌼 A modular Telegram Python bot running on python3 with a sqlalchemy database. How to setup/deplo

Stinkyproject 1 Nov 24, 2021
A Python API wrapper for the Twitter API!

PyTweet PyTweet is an api wrapper made for twitter using twitter's api version 2! Installation Windows py3 -m pip install PyTweet Linux python -m pip

TheFarGG 1 Nov 19, 2022
Presentation and code files for the talk at PyCon Indonesia

pycon-indonesia Presentation and code files for the talk at PyCon Indonesia. Files used for the PyCon Indonesia presentation. [Directory Includes:] Be

Neeraj Pandey 2 Dec 04, 2021
Send song lyrics to iMessage users using the Genius lyrics API

pyMessage Send song lyrics to iMessage users using the Genius lyrics API. Setup 1.) Open the main.py file, and add your API key on line 7. 2.) Install

therealkingnull 1 Jan 23, 2022
Hacktoberfest2021 - Submit Just 4 PRs to earn SWAGS and Tshirts🔥

dont contribute in this repo, contribute only in below mentioned repo Special Note For Everyone ''' always make more then 4 pull request lets you have

Keshav Singh 820 Jan 02, 2023
BleachBit system cleaner for Windows and Linux

BleachBit BleachBit cleans files to free disk space and to maintain privacy. Running from source To run BleachBit without installation, unpack the tar

1.9k Jan 06, 2023
AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications

AWS Serverless Application Model (AWS SAM) The AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications

Amazon Web Services 8.9k Dec 31, 2022
E-Commerce Telegram Bot for UCA Students

ucaStudentStore To buy from and sell to other students Features Register the first time, after that you will always be recognised You can login either

Shukur Sabzaliev 5 Jun 26, 2022
Jalali version of python calendar :date:

jcalendar jcalendar is Jalali implementation of Python's calendar module Status Install pip install jcalendar Documents This module almost follows Py

Iman Kermani 7 Aug 09, 2022
A bot framework for Reddit to manage threads, wiki pages, widgets, menus and more.

Sub Manager Sub Manager is a bot framework for Reddit to automate a variety of tasks on one or more subreddits, and can be configured and run without

r/SpaceX 3 Aug 26, 2022
Python Wrapper for handling payment requests through the Daraja MPESA API

Python Daraja Description Python Wrapper for handling payment requests through the Daraja MPESA API Contribution Refer to the CONTRIBUTING GUIDE. Usag

William Otieno 18 Dec 14, 2022
Lending-Club-Loans - Using TensorFlow to create an ANN model to predict whether people would charge off or pay back their loans.

Lending Club Loans: Brief Introduction LendingClub is a US peer-to-peer lending company, headquartered in San Francisco, California.[3] It was the fir

Ali Akram 1 Jan 03, 2022
A python bot that stops muck chains

muck-chains-stopper-bot a bot that stops muck chains this is the source code of u/DaniDevChainBreaker (the main r/DaniDev muck chains breaker) guys th

24 Jan 04, 2023
📢 Video Chat Stream Telegram Bot. Can ⏳ Stream Live Videos, Radios, YouTube Videos & Telegram Video Files On Your Video Chat Of Channels & Groups !

Telegram Video Chat Bot (Beta) 📢 Video Chat Stream Telegram Bot 🤖 Can Stream Live Videos, Radios, YouTube Videos & Telegram Video Files On Your Vide

brut✘⁶⁹ // ユスフ 15 Dec 24, 2022
Cloudkeeper is “housekeeping for clouds” - find leaky resources, manage quota limits, detect drift and clean up.

Cloudkeeper Housekeeping for Clouds! Table of contents Overview Docker based quick start Cloning this repository Component list Contact License Overvi

Some Engineering 1.2k Jan 03, 2023
Github action for automatically determine the version for next release by using repository tags

This action will automatically determine the version for next release by using repository tags

Igor Gov 7 Oct 25, 2022