Event-driven-model-serving - Unified API of Apache Kafka and Google PubSub

Overview

event-driven-model-serving

Unified API of Apache Kafka and Google PubSub

1. Project Structure

.event-driven-model-serving
+-- Modelling
|   +-- model_checkpoints
|       +-- mnist_cnn_model.h5
|   +-- plots
|       +-- loss_plot.png
|
|   +-- __init__.py
|   +-- data_preparation.py
|   +-- train_cnn.py
|   +-- predict_cnn.py
|
+-- MessageBroker
|   +-- __init__.py
|   +-- broker_interface.py
|   +-- google_pubsub_handler.py
|   +-- kafka_handler.py
|   +-- message_broker_factory.py
|   +-- kafka_config.ini
|   +-- pubsub_config.ini
|
+-- Queue
|   +-- log.txt
|
+-- DB
|   +-- db_for_app_2.txt
|
+-- app_1.py
+-- model_server_sub.py
+-- model_server_pub.py
+-- app_2.py

2. Cloud service configuration

2-1. Confluent Kafka

  1. In MessageBroker/kafka_config.ini, fill in bootstrap.servers, sasl.username, and sasl.password.

  2. Create two topics in your Kafka cluster. Name one as new_image_topic, another as inference_topic.

    Note:
    There is a create_topic() function in each handler, which are not working yet. I will make an update on this soon. For now, you can creat topics by manually in the confluent console.

2-2. Google Pub/Sub

  1. In MessageBroker/pubsub_config.ini, fill in your project_id.

    Note:
    If you are using gcloud CLI, after you set you project with gcloud config set project <Your project name>, project_id can be found by gcloud config get-value project.

  2. Set up Service account and Cloud IAM role.

    Go to https://console.cloud.google.com/iam-admin/ and pick "service account" tab.
    Make one and create JSON key.

    Then set a environment variable with your JSON key like below:
    export GOOGLE_APPLICATION_CREDENTIALS=~/Downloads/pubsub-trial-key.json

  3. Like you did in Kafka, please make two topics.
    Name one as new_image_topic, another as inference_topic.

    In GCP console, when you make these topic, check Add a default subscription as an option. This will create two according subscription_id named, new_image_topic-sub and inference_topic-sub

Every thing is ready for the cloud configuration!

3. About the architecture

  1. Queue directory.

    In the log.txt file, I simulate Redis Queue and FIFO strategy.
    This is to get results queued here before sending the results to application 2.

  2. DB directory.

    In the db_for_app_2.txt file, I simulate a simple database for application 2.
    This shows that application 2 successfully pull result from model server's publisher.

  3. BrokerInterface class is a abstract base class that makes interface for two handlers: KafkaHandler and PubSubHandler.

  4. MessageBrokerFactory creates instance object of either two Handler class.
    Default is set to KafkaHandler.

  5. However, no matter what Handler is created, this unified API ensures the same method names, parameters, and the behaviours.

Side note

  • Currently, the image recognition model is only trained with 28x28 pixel grey scale image. It does not support different size nor color image.
Owner
Danny Toeun Kim
MEng. Computer Science at @UCL
Danny Toeun Kim
An Open Source ALL-In-One Telegram RoBot, that can do lot of things.

URL Uploader Bot An Open Source ALL-In-One Telegram RoBot, that can do lot of things. My Features Installation The Easy Way You can also tap the Deplo

NT BOTS 1 Oct 23, 2021
Pixiv 爬虫,使用 Python 实现。支持批量下载、上传到图床。

用 Python 实现的 Pixiv 爬虫,支持批量下载和上传。 随机图片 API: https://loliapi.ml/ Deploy Github Action 集成部署 建议使用本方法部署,相较于本地部署,无需搭建环境,全程在线上完成。并且使用国外服务器下载、上传,网络更加通畅。 Fork

18 Feb 26, 2022
一个基于Python3的Bot。目前支持以Docker的方式部署在vps上。支持Aria2、本子下载、网易云音乐下载、Pixiv榜单下载、Youtue-dl支持、搜图。

介绍 一个基于Python3的Bot。目前支持以Docker的方式部署在vps上。 主要功能: 文件管理 修改主界面为 filebrowser,账号为admin,密码为admin,主界面路径:http://ip:port,请自行修改密码 FolderMagic自带的webdav:路径:http://

Ben 650 Jan 08, 2023
Make WhatsApp ChatBot and use WhatsApp API to send the WhatsApp messages in python .

Ultramsg.com WhatsApp Bot using WhatsApp API and ultramsg Demo WhatsApp API ChatBot using Ultramsg API with python. Opportunities and tasks: The outpu

Ultramsg 64 Dec 29, 2022
Estimate the total emissions for popular CryptoArt platforms.

cryptoart-footprint Estimate the total CO2 footprint for popular CryptoArt platforms. The goal is to accurately quantify the ecological damage of Ethe

Kyle McDonald 182 Oct 12, 2022
Make a command interpreter that manages AirBnb objects

AirBnB Clone Project Description This is part 1 of our AirBnb Clone project. The purpose of this project is to make a command interpreter that manages

Firdaus H. Salim 1 Nov 14, 2021
通过GitHub的actions 自动采集节点 生成订阅信息

VmessActions 通过GitHub的actions 自动采集节点 自动生成订阅信息 订阅内容自动更新再仓库的 clash.yml 和 v2ray.txt 中 然后PC端/手机端根据自己的软件支持的格式,订阅对应的链接即可

skywolf627 372 Jan 04, 2023
Python client library for Google Maps API Web Services

Python Client for Google Maps Services Description Use Python? Want to geocode something? Looking for directions? Maybe matrices of directions? This l

Google Maps 3.8k Jan 01, 2023
Asyncio SDK for Azure Cosmos DB

Asyncio SDK for Azure Cosmos DB. This library is intended to be a very thin asyncio wrapper around the Azure Comsos DB Rest API. It is not intended to have feature parity with the Microsoft Azure SDK

Grant McDonald 4 Dec 04, 2021
Un petit tool qui est la pour envoier des message avec des webhook en bêta

📎 Webhook-discord Le but de se tool c'est que tu peux envoier vos webhook discord sur vos serveur et les customiser Pour lancer le projet il faut avo

2 Oct 10, 2021
A Discord Bot coded using Python. Open to collaboration

DisPy-Bot A Discord Bot coded using Python. Open to collaboration La syntax pour intégrer le bot (imaginons la fonction lol_reponse dans le fichier au

BiMathAx 2 Mar 03, 2022
Discord Voice Channel Automatic Online

Discord-Selfbot-voice Features: Discord Voice Channel Automatic Online FAQ Q: How can I obtain my token? A: 1. How to obtain your token in android 2.

Pranav Ajay 3 Oct 31, 2022
Free and Open Source Machine Translation API. 100% self-hosted, no limits, no ties to proprietary services. Built on top of Argos Translate.

LibreTranslate Try it online! | API Docs Free and Open Source Machine Translation API, entirely self-hosted. Unlike other APIs, it doesn't rely on pro

UAV4GEO 3.5k Jan 03, 2023
Darkflame Universe Account Manager

Darkflame Universe Account Manager This is a quick and simple web application intended for account creation and management for a DLU instance created

31 Nov 29, 2022
Jika ada pertanyaan lebih lanjut, hubungi kontak dibawah ini. Terimakasih...

⚡ Lynx Userbot ⚡ Userbot Used for Fun on Telegram, and for Maintianing Your Group. This is a Repo Lynx-Userbot. This is Repo was Created by Axel From

29 Aug 30, 2021
Extrait les informations contenues dans le code QR de la preuve de vaccination générée par le gouvernement du Québec

DecodeurPreuveVaccinationQC Extrait les informations contenues dans le code QR de la preuve de vaccination générée par le gouvernement du Québec Utili

Guillaume Morissette 8 Jul 26, 2022
An open source development framework to help you build data workflows and modern data architecture on AWS.

AWS DataOps Development Kit (DDK) The AWS DataOps Development Kit is an open source development framework for customers that build data workflows and

Amazon Web Services - Labs 111 Dec 23, 2022
A simple Python API wrapper for Cloudflare Stream's API.

python-cloudflare-stream A basic Python API wrapper for working with Cloudflare Stream. Arbington.com started off using Cloudflare Stream. We used the

Arbington 3 Sep 08, 2022
Telegram Group Chat Statistics With Python

Telegram Group Chat Statistics How to Run First add PYTHONPATH in repository root directory enviroment variable by running: export PYTHONPATH=${PWD}

Sina Nazem 3 Apr 18, 2022
BaiduPCS API & App 百度网盘客户端

BaiduPCS-Py A BaiduPCS API and An App BaiduPCS-Py 是百度网盘 pcs 的非官方 api 和一个命令行运用程序。

Peter Ding 450 Jan 05, 2023