AWS Blog post code for running feature-extraction on images using AWS Batch and Cloud Development Kit (CDK).

Overview

Batch processing with AWS Batch and CDK

Welcome

This repository demostrates provisioning the necessary infrastructure for running a job on AWS Batch using Cloud Development Kit (CDK). The AWS Batch job reads images from an S3 bucket, runs inference over image-to-vector computer vision model, and stores the results in DynamoDB. Code can be easily modified to fit other batch job transformations you might want to perform.

This code repository is part of the Deep learning image vector embeddings at scale using AWS Batch and CDK AWS DevOps Blog post.

Pre-requisites

  1. Create and source a Python virtualenv on MacOS and Linux, and install python dependencies:
$ python3 -m venv .env
$ source .env/bin/activate
$ pip install -r requirements.txt
  1. Install the latest version of the AWS CDK CLI:
$ npm i -g aws-cdk

Usage

Current code creates a the AWS Batch infrastructure, S3 Bucket for reading the data from, a DynamoDB table to write te batch operation results. Once the infrastructure is provisioned trough AWS CDK, you need to upload the images you want to process to the created S3 bucket. Once you've done that, go to the created AWS Lambda and submit a job. This will trigger a job execution on AWS Batch and you should see the results in the created DynamoDB table.

To deploy and run the batch inference, follow the following steps:

  1. Make sure you have AWS CDK installed and working, all the dependencies of this project defiend in the requirements.txt file, as well as having an installed and configured Docker in your environment;
  2. Set the CDK_DEPLOY_ACCOUNT ENV variable to the name of the AWS account you want to use (pre-defined with AWS CLI);
  3. Set the CDK_DEPLOY_REGION ENV variable to the name of the region you want to deploy the infra in (e.g. 'us-west-2');
  4. Run cdk deploy in the root of this project and wait for the deployment to finish successfully;
  5. Upload the images you need to proccess to the newly created S3 bucket under a S3 bucket path (e.g. /images). Use this path in the next step;
  6. Go to the created AWS Lambda and execute the lambda function with the following JSON:
{
"Paths": [
    "images"
   ]
}
  1. In the AWS console, go to AWS batch and make sure the jobs are submitted and are running successfully;
  2. Open the created DynamoDB table and validate the results are there;
  3. You can now use a DynamoDB client to read and consume the results;

License

This library is licensed under the MIT-0 License. See the LICENSE file.

SOCMINT tool to get personal infos from an Instagram account via analysis of its followers and/or following

S T E R R A 🔭 A SOCMINT tool to get infos from an Instagram acc via its Followers / Following Allows you to analyse someone's followers, following, a

aet 316 Dec 28, 2022
Creates Spotify playlists from Spinitron playlists.

spin2spot Creates Spotify playlists from Spinitron playlists. Quick Start You can use spin2spot as a command-line tool: Erik Didriksen 1 Aug 28, 2021

Scanner and Checker for Binance Scam Contracts

Money Printer by Warranty Voider well this isnt exactly a printer, but it helps you find and check new token startups. In the end its a nice scam cont

12 Nov 24, 2022
Nonebot2 简易群管

简易群管 ✨ NoneBot2 简易群管 ✨ _ 踢 改 禁 欢迎issue pr 权限说明:permission=SUPERUSER 安装 💿 pip install nonebot-plugin-admin 导入 📲 在bot.py 导入,语句: nonebot.load_plugin("n

幼稚园园长 74 Dec 22, 2022
Georeferencing large amounts of data for free.

Geolocate Georeferencing large amounts of data for free. Special thanks to @brunodepauloalmeida and the whole team for the contributions. How? It's us

Gabriel Gazola Milan 23 Dec 30, 2022
A discord bot written in discord.py to manage custom roles assigned to boosters of your server.

BBotty A discord bot written in discord.py to manage custom roles assigned to boosters of your server. v0.0.1-alpha released! This version is incomple

Oui002 1 Nov 27, 2021
Posts word definitions on Twitter daily

Word Of The Day bot Post daily word definitions on social media. Twitter account: https://twitter.com/WordOfTheDay_B Introduction The goal of this pro

Lucas Rijllart 1 Jan 08, 2022
Wechat-file-cleaner - Clean files in PC WeChat FileStorage directory

Wechat-file-cleaner - Clean files in PC WeChat FileStorage directory

Xingjian Zhang 1 Feb 06, 2022
Easy & powerful bot to check if your all Telegram bots are working or not. This bot status bot updates every 45 minutes & runs for 24x7 hours.

PowerfulBotStatus-IDN-C-X Easy & powerful bot to check if your all Telegram bots are working or not. This bot status bot updates every 45 minutes & ru

IDNCoderX 5 Oct 06, 2022
Diablo II Resurrected Diablo Clone Running Room Mgr

d2rdc Diablo II Resurrected Diablo Clone Running Room Mgr Install Dependencies pip install fastapi pip install uvicorn Running uvicorn init:app INFO:

1 Dec 03, 2021
Auto-commiter - Auto commiter Github

auto committer Github Follow the steps below to use this repository: 1-install c

Arman Ebtekari 8 Nov 14, 2022
🔮 Uncover some followers of a private instagram account

Private Instagram Chaining 🔮 Uncover part of followers of an instagram private account I have this private instagram account julianakhao. I need to g

аэт 69 Dec 17, 2022
A zero-dependency Python library for getting the Kubernetes token of a AWS EKS cluster

tokeks A zero-dependency Python library for getting the Kubernetes token of a AWS EKS cluster. No AWS CLI, third-party client or library (boto3, botoc

Chris Karageorgiou Kaneen 6 Nov 04, 2022
This is RequestTrackerBot and it used for tracking request made by user in a group

This is a Request Tracker Bot repo, It is for those who upload content like movies, anime, etc. It can be used for tracking request of content that your members asked for.

Abhijeet 27 Dec 29, 2022
NFTs Upload to OpenSea CuseEdition

NFTs-Upload-to-OpenSea-CuseEdition YOUTUBE VIDEO - Soon... Download Python and

Lil Cuse 2 Jan 04, 2022
Slack bot for monitoring your Metaflow flows!

Metaflowbot - Slack Bot for your Metaflow flows! Metaflowbot makes it fun and easy to monitor your Metaflow runs, past and present. Imagine starting a

Outerbounds 21 Dec 07, 2022
This Instagram app created as a clone of instagram.Developed during Moringa Core.

Instagram This Instagram app created as a clone of instagram.Developed during Moringa Core. AUTHOR By: Nyagah Isaac Description This web-app allows a

Nyagah Isaac 1 Nov 01, 2021
PyHoroscope - Observational Indian lunisolar calendar, horoscope and matching using the Swiss ephemeris

PyHoroscope Observational Indian lunisolar calendar, horoscope and matching usin

4 Jun 05, 2022
Implement SAST + DAST through Github actions

Implement SAST + DAST through Github actions The repository is supposed to implement SAST+DAST checks using github actions against a vulnerable python

Syed Umar Arfeen 3 Nov 09, 2022
Asyncio SDK for Azure Cosmos DB

Asyncio SDK for Azure Cosmos DB. This library is intended to be a very thin asyncio wrapper around the Azure Comsos DB Rest API. It is not intended to have feature parity with the Microsoft Azure SDK

Grant McDonald 4 Dec 04, 2021