ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete

Overview

ShadowClone

ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete.

You can make full use of the Free Tiers provided by cloud providers and supercharge your mundane cli tools with shadow clone jutsu (Naruto style)!

Installation

Please visit the wiki for installation and intial configuration instructions

Usage

⚡ python shadowclone.py -h
usage: shadowclone.py [-h] -i INPUT [-s SPLITNUM] [-o OUTPUT] -c COMMAND

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
  -s SPLITNUM, --split SPLITNUM
                        Number of lines per chunk of file
  -o OUTPUT, --output OUTPUT
  -c COMMAND, --command COMMAND
                        command to execute

How it works

We create a container image during the initial setup and register it as a runtime for our function in AWS/GCP/Azure whatever. When you execute ShadowClone on your computer, instances of that container are activated automatically and are only active for the duration of its execution. How many instances to activate is dynamically decided at runtime depending on the size of the input file provided and the split factor. The input is then split into chunks and equally distributed between all the instances to execute in parallel. For example, if your input file has 10,000 lines and you set the split factor to 100 lines, then it will be split into 100 chunks of 100 lines each and 100 instances will be run in parallel!

Features

  • Extremely fast
  • No need to maintain a VPS (or a fleet of it :))
  • Costs almost nothing per month
    • Compatible with free tiers of most cloud services
  • Cloud agnostic
    • Same script works with AWS, GCP, Azure etc.
  • Supports upto 1000 parallel invocations
  • Dynamically decide the number of invocations
  • Run any tool in parallel on the cloud
  • Pipe output to other tools

Comparison

This tool was inspired by the awesome Axiom and Fleex projects and goes beyond the concept of VPS for running the tools by using serverless functions and containers.

Features Axiom/Fleex ShadowClone
Instances 10-100s* 1000s
Cost Per instance/per minute Mostly Free**
Startup Time 4-5 minutes 2-3 seconds
Idle Cost $++ Free
On Demand Scalability No

*Most cloud providers do not allow spinning up too many instances by default, so you are limited to around 10-15 instances at max. You have to make a request to the support to increase this number.

** AWS & Azure allow 1 million invocations per month for free. Google allows 2 million invocations per month for free. You will be charged only if you go above these limits

Demo

DNS Bruteforcing using a 43mb file - 34 seconds

asciicast

Running httpx on 94K subdomains in 1 min

asciicast

References

Lithops documentation

Free Tiers

Cloud Provider Free Allowance Link
Google Functions 2 Million invocations, 400,000 GB-seconds per month Google Cloud Free Program
AWS Lambda 1 Million invocations, Up to 3.2 million seconds of compute time per month Free Cloud Computing Services - AWS Free Tier
Azure Functions 1 Million invocations Microsoft Azure Free Services

Obviously, you can make any number of function invocations per month. The table above only shows how many invocations are free.

Similar Tools

Disclaimer

This tool is designed as a proof-of-concept implementation and it's usage is intended for educational purpose only. Usage for attacking targets without prior mutual consent is illegal. It's the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program.

Just a simple discord bot a create for try

WAIFU MATERIAL DISCORD BOT! French ver. here HOW TO LAUNCH First, clone this rep

1 Jan 08, 2022
Projeto de estudantes do primeiro período do CIn - UFPE voltado para a criação de um sistema interativo no fechamento da disciplina IF669 - Introdução a Programação.

Projeto Game: Dona da Lua Alunos: Beatriz Férre Clara Kenderessy Matheus Silva Rafael Baltar Roseane Oliveira Samuel Marsaro Sinopse O Cebolinha apron

Maria Clara Kenderessy 5 Dec 20, 2021
Darkflame Universe Account Manager

Darkflame Universe Account Manager This is a quick and simple web application intended for account creation and management for a DLU instance created

31 Nov 29, 2022
Modular Python-based Twitch bot optimized for customizability and ease of use.

rasbot Modular Python-based Twitch bot optimized for customizability and ease of use. rasbot is a Python-based Twitch bot that runs on your Twitch acc

raspy 9 Dec 14, 2022
thumbor is an open-source photo thumbnail service by globo.com

Survey If you use thumbor, please take 1 minute and answer this survey? It's only 2 questions and one is multiple choice!!! thumbor is a smart imaging

Thumbor (by @globocom) 9.3k Dec 31, 2022
Video-Player - Telegram Music/ Video Streaming Bot Using Pytgcalls

Video Player 🔥 ᴢᴀɪᴅ ᴠᴄ ᴘʟᴀyᴇʀ ɪꜱ ᴀ ᴛᴇʟᴇɢʀᴀᴍ ᴘʀᴏᴊᴇᴄᴛ ʙᴀꜱᴇᴅ ᴏɴ ᴘʏʀᴏɢʀᴀᴍ ꜰᴏʀ ᴘʟᴀʏ

Zaid 16 Nov 30, 2022
Online Marketplace API

Online Marketplace API Table of Contents Setup Instructions Documentation Setup instructions Make sure you have python installed Clone the repository

Kanat 3 Jul 13, 2022
My attempt to reverse the Discord nitro token generation function.

discord-theory-I PART: I My attempt to reverse the Discord nitro token generation function. The Nitro generation tools thing is common in Discord now,

Jakom 29 Aug 14, 2022
A telegram bot does not allow channels to send messages to the telegram supergroup

Channel Message Handler Getting started Installation $ git clone https://github.com/AbhijithNT/GroupChannelHandler.git Change directory $ cd ChannelMe

Abhijith N T 0 Dec 26, 2021
A Python API for Connected 2

connected API for Connected 2 api for the { connected 2 } programmer : api report api follow api check username api forget password api Search api cha

2 Jun 05, 2022
Extrait les informations contenues dans le code QR de la preuve de vaccination générée par le gouvernement du Québec

DecodeurPreuveVaccinationQC Extrait les informations contenues dans le code QR de la preuve de vaccination générée par le gouvernement du Québec Utili

Guillaume Morissette 8 Jul 26, 2022
BT CCXT Store

bt-ccxt-store-cn backtrader是一个非常好的开源量化回测平台,我自己也时常用它,backtrader也能接入实盘,而bt-ccxt-store就是帮助backtrader接入数字货币实盘交易的一个插件,但是bt-ccxt-store的某些实现并不是很好,无节制的网络轮询,一些

moses 40 Dec 31, 2022
My personal discord bot using discord.py

Rara-chan My personal discord bot that I use for stuff that I find interesting. Features Responds to certain messages ChatBot capabilities NHentai scr

Mikask 3 Nov 06, 2022
Rbx-mass-send - mass sends trades to item owners

mass sends trades to item owners proxies should be in ip:port format itemsToSend

0 Feb 20, 2022
Companion "receiver" to matrix-appservice-webhooks for [matrix].

Matrix Webhook Receiver Companion "receiver" to matrix-appservice-webhooks for [matrix]. The purpose of this app is to listen for generic webhook mess

Kim Brose 13 Sep 29, 2022
DonLee Robot

🤖 𝐃𝐎𝐍 𝐋𝐄𝐄 𝐑𝐎𝐁𝐎𝐓 𝐕𝟐 🤖 👋 Hey Muhammed, Iam DonLee RoBoT Make me an admin for your group and channel then connect me.... 🎉 🙂 To build a

Muhammed 27 Dec 01, 2022
A Python library wrapping the iFixit (Dozuki) API.

A Python library wrapping the iFixit API. Status Working, but incomplete. Fully tested and documented. Hacking I highly recommend using virtualenv: [$

James Pearson Hughes 13 May 24, 2021
Best Buy Bot used to add products to cart for purchase.

To Install the Best Buy Bot These instructions are for Mac users only. Clone this Repo to your machine. BestBuyBot Open in VScode. Is Python installed

Robert Estrella 1 Dec 11, 2021
The first open-source PyTgCalls-based project.

Calls Music — Telegram bot + userbot for streaming audio in group calls ✍🏻 Requirements FFmpeg Python 3.7+ 🚀 Deployment 🛠 Configuration Copy exampl

Calls Music 74 Nov 19, 2022