AWS Blog post code for running feature-extraction on images using AWS Batch and Cloud Development Kit (CDK).

Overview

Batch processing with AWS Batch and CDK

Welcome

This repository demostrates provisioning the necessary infrastructure for running a job on AWS Batch using Cloud Development Kit (CDK). The AWS Batch job reads images from an S3 bucket, runs inference over image-to-vector computer vision model, and stores the results in DynamoDB. Code can be easily modified to fit other batch job transformations you might want to perform.

This code repository is part of the Deep learning image vector embeddings at scale using AWS Batch and CDK AWS DevOps Blog post.

Pre-requisites

  1. Create and source a Python virtualenv on MacOS and Linux, and install python dependencies:
$ python3 -m venv .env
$ source .env/bin/activate
$ pip install -r requirements.txt
  1. Install the latest version of the AWS CDK CLI:
$ npm i -g aws-cdk

Usage

Current code creates a the AWS Batch infrastructure, S3 Bucket for reading the data from, a DynamoDB table to write te batch operation results. Once the infrastructure is provisioned trough AWS CDK, you need to upload the images you want to process to the created S3 bucket. Once you've done that, go to the created AWS Lambda and submit a job. This will trigger a job execution on AWS Batch and you should see the results in the created DynamoDB table.

To deploy and run the batch inference, follow the following steps:

  1. Make sure you have AWS CDK installed and working, all the dependencies of this project defiend in the requirements.txt file, as well as having an installed and configured Docker in your environment;
  2. Set the CDK_DEPLOY_ACCOUNT ENV variable to the name of the AWS account you want to use (pre-defined with AWS CLI);
  3. Set the CDK_DEPLOY_REGION ENV variable to the name of the region you want to deploy the infra in (e.g. 'us-west-2');
  4. Run cdk deploy in the root of this project and wait for the deployment to finish successfully;
  5. Upload the images you need to proccess to the newly created S3 bucket under a S3 bucket path (e.g. /images). Use this path in the next step;
  6. Go to the created AWS Lambda and execute the lambda function with the following JSON:
{
"Paths": [
    "images"
   ]
}
  1. In the AWS console, go to AWS batch and make sure the jobs are submitted and are running successfully;
  2. Open the created DynamoDB table and validate the results are there;
  3. You can now use a DynamoDB client to read and consume the results;

License

This library is licensed under the MIT-0 License. See the LICENSE file.

A Video Streaming Telegram Bot written in Python with Pyrogram and PyTgcalls

Video Stream Bot A Video Streaming Telegram Bot written in Python using Pyrogram and PyTgcalls Requirements Python 3.9 Telegram API Telegram Bot Token

Aarav Arora 61 Dec 10, 2022
Python Library for Accessing the Cohere API

Cohere Python SDK This package provides functionality developed to simplify interfacing with the Cohere API in Python 3. Documentation See the API's d

cohere.ai 42 Jan 03, 2023
Boto3 code assistance for any API in any IDE, always up to date

botostubs Gives you code assistance for any boto3 API in any IDE. Get started by running pip install botostubs Demo Features PyPI package automaticall

Jeshan Giovanni BABOOA 94 Nov 14, 2022
1.本项目采用Python Flask框架开发提供(应用管理,实例管理,Ansible管理,LDAP管理等相关功能)

op-devops-api 1.本项目采用Python Flask框架开发提供(应用管理,实例管理,Ansible管理,LDAP管理等相关功能) 后端项目配套前端项目为:op-devops-ui jenkinsManager 一.插件python-jenkins bug修复 (1).插件版本 pyt

3 Nov 12, 2021
A Python API wrapper for the Twitter API!

PyTweet PyTweet is an api wrapper made for twitter using twitter's api version 2! Installation Windows py3 -m pip install PyTweet Linux python -m pip

TheFarGG 1 Nov 19, 2022
Jalali version of python calendar :date:

jcalendar jcalendar is Jalali implementation of Python's calendar module Status Install pip install jcalendar Documents This module almost follows Py

Iman Kermani 7 Aug 09, 2022
✨ A Telegram mirror/leech bot By SparkXcloud Group ✨

SparkXcloud-Gdrive-MirrorBot SparkXcloud-Gdrive-MirrorBot is a multipurpose Telegram Bot writen in Python for mirroring files on the Internet to our b

119 Oct 23, 2022
python library to the bitly api

bitly API python library Installation pip install bitly_api Run tests Your username is the lowercase name shown when you login to bitly, your access

Bitly 245 Aug 14, 2022
A minimal caching proxy to GitHub's REST & GraphQL APIs

github-proxy A caching forward proxy to GitHub's REST and GraphQL APIs. GitHub-Proxy is a thin, highly extensible, highly configurable python framewor

Babylon Health 26 Oct 05, 2022
Simple software that can send WhatsApp message to a single or multiple users (including unsaved number**)

wp-automation Info: this is a simple automation software that sends WhatsApp message to single or multiple users. Key feature: -Sends message to multi

3 Jan 31, 2022
Prime Mega is a modular bot running on python3 with autobots theme and have a lot features.

PRIME MEGA Prime Mega is a modular bot running on python3 with autobots theme and have a lot features. Easiest Way To Deploy On Heroku This Bot is Cre

『TØNIC』 乂 ₭ILLΣR 45 Dec 15, 2022
This bot is created by AJTimePyro and It accepts direct downloading url & then return file as telegram file.

URL Uploader Bot This is the source code of URL Uploader Bot. And the developer of this bot is AJTimePyro, His Telegram Channel & Group. You can use t

Abhijeet 23 Nov 13, 2022
Pogodasbot - Telegram bot sending channel weather info

Pogodasbot - Telegram bot sending channel weather info

Qayrat Sultan 1 Dec 15, 2022
Python lib to control HottoH based stove devices

Project desciption This library can be used to discuss with HootoH based stove devices Actually tested and validated with a CMG Drum stove. To use thi

3 May 16, 2022
An advanced automatic top.gg dank memer voter that votes automatically for you.

Auto Dank Memer Voter An automatic dank memer voter that sends votes onto top.gg every 12 hours, unless their is captcha. I am working on a captcha de

6 Aug 27, 2022
A Python Library to interface with LinkedIn API, OAuth and JSON responses

#Overview Here's another library based on the LinkedIn API, OAuth and JSON responses. Hope this documentation explains everything you need to get star

Mike Helmick 69 Dec 11, 2022
Example notebooks for working with SageMaker Studio Lab. Sign up for an account at the link below!

SageMaker Studio Lab Sample Notebooks Available today in public preview. If you are looking for a no-cost compute environment to run Jupyter notebooks

Amazon Web Services 304 Jan 01, 2023
Download archived malware from ActiveState's source code mirror

malware-archivist (ma) Tool to aid security researchers in dissecting malware. Often, repository maintainers will remove malicious packages entirely f

ActiveState Software 28 Dec 12, 2022
Hassium Server Manager For Python

Hassium Server Manager This is meant to be a tool for mostly internal use. I decided that I would make it open souce in case anyone wanted to use it.

0 Nov 24, 2022
Dante, my discord bot. Open source project in development and not optimized for other filesystems, install and setup script in development

DanteMode (In private development for ~6 months) Dante, my discord bot. Open source project in development and not optimized for other filesystems, in

2 Nov 05, 2021