Demonstrates how to divide a DL model into multiple IR model files (division) and introduce a simplest way to implement a custom layer works with OpenVINO IR models.

Overview

Demonstration of OpenVINO techniques - Model-division and a simplest-way to support custom layers

Description:

Model Optimizer in Intel(r) OpenVINO(tm) toolkit supports model division function. User can specify the region in the model to convert by specifying entry point and exit point with --input and --output options respectively.
The expected usage of those options are:

  • Excluding unnecessary layers: Removing non-DL related layers (such as JPEG decode) and layers not required for inferencing (such as accuracy metrics calculation)
  • Load balancing: Divide a model into multiple parts and cascade them to get the final inferencing result. Each individual part can be run on different device or different timing.
  • Access to the intermediate result: Divide a model and get the intermediate feature data to check the model integrity or for the other purposes.
  • Exclude non-supported layers: Convert the model without OpenVINO non-supprted layers. Divide the model and skip non-supported layers to get the IR models. User needs to perform the equivalent processing for the excluded layers to get the correct inferencing result.

This project demonstrates how to divide a DL model, and fill the hole for skipped leyers.
The project includes Python and C++ implementations of naive 2D convolution layer to perform the Conv2D task which was supposed to have done by the skipped layer. This could be a good reference when you need to implement a custom layer function to your project but don't want to develop full-blown OpenVINO custom layers due to some restrictions such as development time.
In this project, we will use a simple CNN classification model trained with MNIST dataset and demonstrate the way to divide the model with skipping a layer (on purpose) and use a simple custom layer to cover the data processing for the skipped layer.

image

Prerequisites:

  • TensorFlow 2.x
  • OpenVINO 2021.4 (2021.x may work)

How to train the model and create a trained model

You can train the model by just kicking the training.py script. training.py will use keras.datasets.mnist as the training and validation dataset and train the model, and then save the trained model in SavedModel format.
training.py also generates weights.npy file that contains the weight and bias data of target_conv_layer layer. This weight and bias data will be used by the special made Conv2D layer.
Since the model we use is tiny, it will take just a couple of minutes to complete.

python3 training.py

How to convert a TF trained model into OpenVINO IR model format

Model Optimizer in OpenVINO converts TF (savedmodel) model into OpenVINO IR model.
Here's a set of script to convert the model for you.

script description
convert-normal.sh Convert entire model and generate single IR model file (no division)
convert-divide.sh Divide the input model and output 2 IR models. All layers are still contained (no skipped layers)
convert-divide-skip.sh Divide the input model and skip 'target_conv_layer'
  • The converted models can be found in ./models folder.

Tip to find the correct node name for Model Optimizer

Model optimizer requires MO internal networkx graph node name to specify --input and --output nodes. You can modify the model optimizer a bit to have it display the list of networkx node names. Add 3 lines on the very bottom of the code snnipet below and run the model optimizer.

mo/utils/class_registration.py

def apply_replacements_list(graph: Graph, replacers_order: list):
    """
    Apply all transformations from replacers_order
    """
    for i, replacer_cls in enumerate(replacers_order):
        apply_transform(
            graph=graph,
            replacer_cls=replacer_cls,
            curr_transform_num=i,
            num_transforms=len(replacers_order))
        # Display name of available nodes after the 'loader' stage
        if 'LoadFinish' in str(replacer_cls):
            for node in graph.nodes():
                print(node)

You'll see something like this. You need to use one of those node names for --input and --output options in MO.

conv2d_input
Func/StatefulPartitionedCall/input/_0
unknown
Func/StatefulPartitionedCall/input/_1
StatefulPartitionedCall/sequential/conv2d/Conv2D/ReadVariableOp
StatefulPartitionedCall/sequential/conv2d/Conv2D
   :   (truncated)   :
StatefulPartitionedCall/sequential/dense_1/BiasAdd/ReadVariableOp
StatefulPartitionedCall/sequential/dense_1/BiasAdd
StatefulPartitionedCall/sequential/dense_1/Softmax
StatefulPartitionedCall/Identity
Func/StatefulPartitionedCall/output/_11
Func/StatefulPartitionedCall/output_control_node/_12
Identity
Identity56

How to infer with the models on OpenVINO

Several versions of scripts are available for the inference testing.
Those test programs will do inference 10,000 times with the MNIST validation dataset. Test program displays '.' when inference result is correct and 'X' when it's wrong. Performance numbers are measured from the start of 10,000 inferences to the end of all inferences. So, it is including loop overhead, pre/post processing time and so on.

script description (reference execution time, Core i7-8665U)
inference.py Use simgle, monolithic IR model and run inference 3.3 sec
inference-div.py Take 2 divided IR models and run inference. 2 models will be cascaded. 5.3 sec(*1)
inference-skip-python.py Tak2 2 divided IR models which excluded the 'target_conv_layer'. Program is including a Python version of Conv2D and perform convolution for 'target_conv_layer'. VERY SLOW. 4338.6 sec
inference-skip-cpp.py Tak2 2 divided IR models which excluded the 'target_conv_layer'. Program imports a Python module written in C++ which includes a C++ version of Conv2D. Reasonably fast. Conv2D Python extension module is required. Please refer to the following section for details. 10.8 sec

Note 1: This model is quite tiny and light-weight. OpenVINO can run this model in <0.1msec on Core i7-8665U CPU. The inferencing overhead introduced by dividing the model is noticeable but when you use heavy model, this penalty might be negligible.

How to build the Conv2D C++ Python extnsion module

You can build the Conv2D C++ Python extension module by running build.sh or build.bat.
myLayers.so or myLayers.pyd will be generated and copied to the current directory after a successful build.

How to run draw-and-infer demo program

Here's a simple yet bit fun demo application for MNIST CNN. You can draw a number on the screen by mouse or finger-tip and you'll see the real-time inference result. Right-click will clear the screen for another try. Several versions are available.

script description
draw-and-infer.py Use the monolithic IR model
draw-and-infer-div.py Use divided IR models
draw-and-infer-skip-cpp.py Use divided IR models which excluded 'target_conv_layer'. Conv2D Python extension is requird.

draw-and-infer

Tested environment

  • Windows 10 + VS2019 + OpenVINO 2021.4
  • Ubuntu 20.04 + OpenVINO 2021.4
Owner
Yasunori Shimura
Yasunori Shimura
Course materials for Fall 2021 "CIS6930 Topics in Computing for Data Science" at New College of Florida

Fall 2021 CIS6930 Topics in Computing for Data Science This repository hosts course materials used for a 13-week course "CIS6930 Topics in Computing f

Yoshi Suhara 101 Nov 30, 2022
A simple Tensorflow based library for deep and/or denoising AutoEncoder.

libsdae - deep-Autoencoder & denoising autoencoder A simple Tensorflow based library for Deep autoencoder and denoising AE. Library follows sklearn st

Rajarshee Mitra 147 Nov 18, 2022
Machine Translation Implement By Bi-GRU And Transformer

Seq2Seq Translation Implement By Bidirectional GRU And Transformer In Pytorch Before You Run The Code You should download the data through the link be

He Wang 2 Oct 27, 2021
Official implementation for the paper: Multi-label Classification with Partial Annotations using Class-aware Selective Loss

Multi-label Classification with Partial Annotations using Class-aware Selective Loss Paper | Pretrained models Official PyTorch Implementation Emanuel

99 Dec 27, 2022
[ICCV 2021 Oral] Mining Latent Classes for Few-shot Segmentation

Mining Latent Classes for Few-shot Segmentation Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, Yang Gao. This codebase contains baseline of our paper Mini

Lihe Yang 66 Nov 29, 2022
LightningFSL: Pytorch-Lightning implementations of Few-Shot Learning models.

LightningFSL: Few-Shot Learning with Pytorch-Lightning In this repo, a number of pytorch-lightning implementations of FSL algorithms are provided, inc

Xu Luo 76 Dec 11, 2022
HiFi-GAN: High Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks

HiFiGAN Denoiser This is a Unofficial Pytorch implementation of the paper HiFi-GAN: High Fidelity Denoising and Dereverberation Based on Speech Deep F

Rishikesh (ऋषिकेश) 134 Dec 27, 2022
No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency

This repository contains the implementation for the paper: No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consiste

Alireza Golestaneh 75 Dec 30, 2022
CKD - Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding

Collaborative Knowledge Distillation for Heterogeneous Information Network Embed

zhousheng 9 Dec 05, 2022
An air quality monitoring service with a Raspberry Pi and a SDS011 sensor.

Raspberry Pi Air Quality Monitor A simple air quality monitoring service for the Raspberry Pi. Installation Clone the repository and run the following

rydercalmdown 24 Dec 09, 2022
Official PyTorch implementation of CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds

CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds Introduction This is the official PyTorch implementation of o

Yijia Weng 96 Dec 07, 2022
Res2Net for Instance segmentation and Object detection using MaskRCNN

Res2Net for Instance segmentation and Object detection using MaskRCNN Since the MaskRCNN-benchmark of facebook is deprecated, we suggest to use our mm

Res2Net Applications 55 Oct 30, 2022
Code for the IJCAI 2021 paper "Structure Guided Lane Detection"

SGNet Project for the IJCAI 2021 paper "Structure Guided Lane Detection" Abstract Recently, lane detection has made great progress with the rapid deve

Jinming Su 27 Dec 08, 2022
A Marvelous ChatBot implement using PyTorch.

PyTorch Marvelous ChatBot [Update] it's 2019 now, previously model can not catch up state-of-art now. So we just move towards the future a transformer

JinTian 223 Oct 18, 2022
Official PyTorch implementation of the paper "Self-Supervised Relational Reasoning for Representation Learning", NeurIPS 2020 Spotlight.

Official PyTorch implementation of the paper: "Self-Supervised Relational Reasoning for Representation Learning" (2020), Patacchiola, M., and Storkey,

Massimiliano Patacchiola 135 Jan 03, 2023
SEOVER: Sentence-level Emotion Orientation Vector based Conversation Emotion Recognition Model

SEOVER-Master This code is the implementation of paper: SEOVER: Sentence-level Emotion Orientation Vector based Conversation Emotion Recognition Model

4 Feb 24, 2022
Wikidated : An Evolving Knowledge Graph Dataset of Wikidata’s Revision History

Wikidated Wikidated 1.0 is a dataset of Wikidata’s full revision history, which encodes changes between Wikidata revisions as sets of deletions and ad

Lukas Schmelzeisen 11 Aug 16, 2022
Perspective: Julia for Biologists

Perspective: Julia for Biologists 1. Examples Speed: Example 1 - Single cell data and network inference Domain: Single cell data Methodology: Network

Elisabeth Roesch 55 Dec 02, 2022
Fully Convlutional Neural Networks for state-of-the-art time series classification

Deep Learning for Time Series Classification As the simplest type of time series data, univariate time series provides a reasonably good starting poin

Stephen 572 Dec 23, 2022
Neural Radiance Fields Using PyTorch

This project is a PyTorch implementation of Neural Radiance Fields (NeRF) for reproduction of results whilst running at a faster speed.

Vedant Ghodke 1 Feb 11, 2022