Event-driven-model-serving - Unified API of Apache Kafka and Google PubSub

Overview

event-driven-model-serving

Unified API of Apache Kafka and Google PubSub

1. Project Structure

.event-driven-model-serving
+-- Modelling
|   +-- model_checkpoints
|       +-- mnist_cnn_model.h5
|   +-- plots
|       +-- loss_plot.png
|
|   +-- __init__.py
|   +-- data_preparation.py
|   +-- train_cnn.py
|   +-- predict_cnn.py
|
+-- MessageBroker
|   +-- __init__.py
|   +-- broker_interface.py
|   +-- google_pubsub_handler.py
|   +-- kafka_handler.py
|   +-- message_broker_factory.py
|   +-- kafka_config.ini
|   +-- pubsub_config.ini
|
+-- Queue
|   +-- log.txt
|
+-- DB
|   +-- db_for_app_2.txt
|
+-- app_1.py
+-- model_server_sub.py
+-- model_server_pub.py
+-- app_2.py

2. Cloud service configuration

2-1. Confluent Kafka

  1. In MessageBroker/kafka_config.ini, fill in bootstrap.servers, sasl.username, and sasl.password.

  2. Create two topics in your Kafka cluster. Name one as new_image_topic, another as inference_topic.

    Note:
    There is a create_topic() function in each handler, which are not working yet. I will make an update on this soon. For now, you can creat topics by manually in the confluent console.

2-2. Google Pub/Sub

  1. In MessageBroker/pubsub_config.ini, fill in your project_id.

    Note:
    If you are using gcloud CLI, after you set you project with gcloud config set project <Your project name>, project_id can be found by gcloud config get-value project.

  2. Set up Service account and Cloud IAM role.

    Go to https://console.cloud.google.com/iam-admin/ and pick "service account" tab.
    Make one and create JSON key.

    Then set a environment variable with your JSON key like below:
    export GOOGLE_APPLICATION_CREDENTIALS=~/Downloads/pubsub-trial-key.json

  3. Like you did in Kafka, please make two topics.
    Name one as new_image_topic, another as inference_topic.

    In GCP console, when you make these topic, check Add a default subscription as an option. This will create two according subscription_id named, new_image_topic-sub and inference_topic-sub

Every thing is ready for the cloud configuration!

3. About the architecture

  1. Queue directory.

    In the log.txt file, I simulate Redis Queue and FIFO strategy.
    This is to get results queued here before sending the results to application 2.

  2. DB directory.

    In the db_for_app_2.txt file, I simulate a simple database for application 2.
    This shows that application 2 successfully pull result from model server's publisher.

  3. BrokerInterface class is a abstract base class that makes interface for two handlers: KafkaHandler and PubSubHandler.

  4. MessageBrokerFactory creates instance object of either two Handler class.
    Default is set to KafkaHandler.

  5. However, no matter what Handler is created, this unified API ensures the same method names, parameters, and the behaviours.

Side note

  • Currently, the image recognition model is only trained with 28x28 pixel grey scale image. It does not support different size nor color image.
Owner
Danny Toeun Kim
MEng. Computer Science at @UCL
Danny Toeun Kim
Bavera is an extensive and extendable Python 3.x library for the Discord API

Bavera is an extensive and extendable Python 3.x library for the Discord API. Bavera boasts the following major features: Expressive, functiona

Bavera 1 Nov 17, 2021
Discord bot for playing Werewolf game on League of Legends.

LoLWolf LoL人狼をプレイするときのDiscord用botです。 (Discord bot for playing Werewolf game on League of Legends.) 以下のボタンを押してbotをあなたのDiscordに招待することで誰でも簡単に使用することができます。

Hatsuka 4 Oct 18, 2021
Administration Panel for Control FiveM Servers From Discord

FiveM Discord Administration Panel Version 1.0.0 If you would like to report an issue or request a feature. Join our Discord or create an issue. Contr

NIma 9 Jun 17, 2022
An unoffcial python API client for primeuploads.com

primeuploads-py An unoffcial python API wrapper for primeuploads.com Installation pip3 install primeuploads-py Usage example from prime import PrimeUp

W4RR10R 41 Dec 05, 2022
A simple Discord bot that notifies users of new Abitti versions

A simple Discord bot that notifies users of new Abitti versions. New features might be added later on. If you have good ideas, feel free to do a PR.

1 Feb 11, 2022
Pixoo-Awesome is a tool to get more out of your Pixoo Devices.

Pixoo-Awesome is a tool to get more out of your Pixoo Devices. It uses the Pixoo-Client to connect to your Pixoo devices and send data to them. I targ

Horo 10 Oct 27, 2022
Get an SNS alert for High Severity GuardDuty findings

Automation AWS-GuardDuty findings Get an SNS alert for High Severity GuardDuty findings Problem: Getting notified when there is Red finding in AWS Gua

Giten Mitra 4 Nov 18, 2022
Instagram GiftShop Scam Killer

Instagram GiftShop Scam Killer A basic tool for Windows which kills acess to any giftshop scam from Instagram. Protect your Instagram account from the

1 Mar 31, 2022
Download archived malware from ActiveState's source code mirror

malware-archivist (ma) Tool to aid security researchers in dissecting malware. Often, repository maintainers will remove malicious packages entirely f

ActiveState Software 28 Dec 12, 2022
High-Resolution Differential Z-Belt Mod for V0 (with optional Kirigami support)

V0-DBM This is a high-resolution differential pulley system belt mod for the Z-axis on Voron 0 with optional Kirigami Bed support. NOTE: Alpha version

Simon Küppers 11 Jan 07, 2023
Just a python library to make reddit post caching easier

Reddist Just a python library to make reddit post caching easier. Caching Options In Memory Caching Redis Caching Pickle Caching Usage Installation: D

Samrid Pandit 3 Jan 16, 2022
This is Pdisk Upload Bot made using Python with Pyrogram Framework. Its capable of uploading direct download link with thumbnail or without thumbnail & with Title Support.

Pdisk-Upload-Bot Introduction This Is PDisk Upload Bot Used To Upload Direct Link To Pdisk With Thumb Support Deploy Heroku Deploy Local Deploy pip in

HEIMAN PICTURES 32 Oct 21, 2022
Braintree Python library

Braintree Python library The Braintree Python library provides integration access to the Braintree Gateway. TLS 1.2 required The Payment Card Industry

Braintree 230 Dec 18, 2022
This is a tutorial on how to make a Discord Bot using the discord.py library

HowToMakeADiscordBot This Github repository is here to help you code a Discord Bot using the discord.py library! 1 - Setup: Download the code inside t

Baz 1 Oct 31, 2021
Satoshi is a discord bot template in python using discord.py that allow you to track some live crypto prices with your own discord bot.

Satoshi ~ DiscordCryptoBot Satoshi is a simple python discord bot using discord.py that allow you to track your favorites cryptos prices with your own

Théo 2 Sep 15, 2022
C Y B Ξ R UserBot is a project that simplifies the use of Telegram. All rights reserved.

C Y B Ξ R USΞRBOT 🇦🇿 C Y B Ξ R UserBot is a project that simplifies the use of Telegram. All rights reserved. Automatic Setup Android: open Termux p

C Y B Ξ R 0 Sep 20, 2022
A Telegram Repo For Devs To Controll The Bots Under Maintenance.This Bot Is For Developers, If Your Bot Is Down, Use This Repo To Give Your Dear Subscribers Some Support By Providing Them Response.

Maintenance Bot A Telegram Repo For Devs To Controll The Bots Under Maintenance About This Bot This Bot Is For Developers, If Your Bot Is Down, Use Th

Vɪᴠᴇᴋ 47 Dec 29, 2022
Anime Themed Telegram Group Manager Bot By WaifuNetwork

🤍 Yukino Yukinoshita 🤍 #This Is The OLD version Of Yukino Bot New Version Of Yukino Yukinoshita is private. Thanks to everyone who starred Yukino, T

TR0J3N 4 Jan 10, 2022
Auto-Rollnumber-sender - Auto Rollnumber sender with python

Auto-Rollnumber-sender The above code fits better on my system but it can vary s

Riya Tripathi 2 Feb 14, 2022
Google Search Results via SERP API pip Python Package

Google Search Results in Python This Python package is meant to scrape and parse search results from Google, Bing, Baidu, Yandex, Yahoo, Home depot, E

SerpApi 254 Jan 05, 2023