Demonstrates how to divide a DL model into multiple IR model files (division) and introduce a simplest way to implement a custom layer works with OpenVINO IR models.

Overview

Demonstration of OpenVINO techniques - Model-division and a simplest-way to support custom layers

Description:

Model Optimizer in Intel(r) OpenVINO(tm) toolkit supports model division function. User can specify the region in the model to convert by specifying entry point and exit point with --input and --output options respectively.
The expected usage of those options are:

  • Excluding unnecessary layers: Removing non-DL related layers (such as JPEG decode) and layers not required for inferencing (such as accuracy metrics calculation)
  • Load balancing: Divide a model into multiple parts and cascade them to get the final inferencing result. Each individual part can be run on different device or different timing.
  • Access to the intermediate result: Divide a model and get the intermediate feature data to check the model integrity or for the other purposes.
  • Exclude non-supported layers: Convert the model without OpenVINO non-supprted layers. Divide the model and skip non-supported layers to get the IR models. User needs to perform the equivalent processing for the excluded layers to get the correct inferencing result.

This project demonstrates how to divide a DL model, and fill the hole for skipped leyers.
The project includes Python and C++ implementations of naive 2D convolution layer to perform the Conv2D task which was supposed to have done by the skipped layer. This could be a good reference when you need to implement a custom layer function to your project but don't want to develop full-blown OpenVINO custom layers due to some restrictions such as development time.
In this project, we will use a simple CNN classification model trained with MNIST dataset and demonstrate the way to divide the model with skipping a layer (on purpose) and use a simple custom layer to cover the data processing for the skipped layer.

image

Prerequisites:

  • TensorFlow 2.x
  • OpenVINO 2021.4 (2021.x may work)

How to train the model and create a trained model

You can train the model by just kicking the training.py script. training.py will use keras.datasets.mnist as the training and validation dataset and train the model, and then save the trained model in SavedModel format.
training.py also generates weights.npy file that contains the weight and bias data of target_conv_layer layer. This weight and bias data will be used by the special made Conv2D layer.
Since the model we use is tiny, it will take just a couple of minutes to complete.

python3 training.py

How to convert a TF trained model into OpenVINO IR model format

Model Optimizer in OpenVINO converts TF (savedmodel) model into OpenVINO IR model.
Here's a set of script to convert the model for you.

script description
convert-normal.sh Convert entire model and generate single IR model file (no division)
convert-divide.sh Divide the input model and output 2 IR models. All layers are still contained (no skipped layers)
convert-divide-skip.sh Divide the input model and skip 'target_conv_layer'
  • The converted models can be found in ./models folder.

Tip to find the correct node name for Model Optimizer

Model optimizer requires MO internal networkx graph node name to specify --input and --output nodes. You can modify the model optimizer a bit to have it display the list of networkx node names. Add 3 lines on the very bottom of the code snnipet below and run the model optimizer.

mo/utils/class_registration.py

def apply_replacements_list(graph: Graph, replacers_order: list):
    """
    Apply all transformations from replacers_order
    """
    for i, replacer_cls in enumerate(replacers_order):
        apply_transform(
            graph=graph,
            replacer_cls=replacer_cls,
            curr_transform_num=i,
            num_transforms=len(replacers_order))
        # Display name of available nodes after the 'loader' stage
        if 'LoadFinish' in str(replacer_cls):
            for node in graph.nodes():
                print(node)

You'll see something like this. You need to use one of those node names for --input and --output options in MO.

conv2d_input
Func/StatefulPartitionedCall/input/_0
unknown
Func/StatefulPartitionedCall/input/_1
StatefulPartitionedCall/sequential/conv2d/Conv2D/ReadVariableOp
StatefulPartitionedCall/sequential/conv2d/Conv2D
   :   (truncated)   :
StatefulPartitionedCall/sequential/dense_1/BiasAdd/ReadVariableOp
StatefulPartitionedCall/sequential/dense_1/BiasAdd
StatefulPartitionedCall/sequential/dense_1/Softmax
StatefulPartitionedCall/Identity
Func/StatefulPartitionedCall/output/_11
Func/StatefulPartitionedCall/output_control_node/_12
Identity
Identity56

How to infer with the models on OpenVINO

Several versions of scripts are available for the inference testing.
Those test programs will do inference 10,000 times with the MNIST validation dataset. Test program displays '.' when inference result is correct and 'X' when it's wrong. Performance numbers are measured from the start of 10,000 inferences to the end of all inferences. So, it is including loop overhead, pre/post processing time and so on.

script description (reference execution time, Core i7-8665U)
inference.py Use simgle, monolithic IR model and run inference 3.3 sec
inference-div.py Take 2 divided IR models and run inference. 2 models will be cascaded. 5.3 sec(*1)
inference-skip-python.py Tak2 2 divided IR models which excluded the 'target_conv_layer'. Program is including a Python version of Conv2D and perform convolution for 'target_conv_layer'. VERY SLOW. 4338.6 sec
inference-skip-cpp.py Tak2 2 divided IR models which excluded the 'target_conv_layer'. Program imports a Python module written in C++ which includes a C++ version of Conv2D. Reasonably fast. Conv2D Python extension module is required. Please refer to the following section for details. 10.8 sec

Note 1: This model is quite tiny and light-weight. OpenVINO can run this model in <0.1msec on Core i7-8665U CPU. The inferencing overhead introduced by dividing the model is noticeable but when you use heavy model, this penalty might be negligible.

How to build the Conv2D C++ Python extnsion module

You can build the Conv2D C++ Python extension module by running build.sh or build.bat.
myLayers.so or myLayers.pyd will be generated and copied to the current directory after a successful build.

How to run draw-and-infer demo program

Here's a simple yet bit fun demo application for MNIST CNN. You can draw a number on the screen by mouse or finger-tip and you'll see the real-time inference result. Right-click will clear the screen for another try. Several versions are available.

script description
draw-and-infer.py Use the monolithic IR model
draw-and-infer-div.py Use divided IR models
draw-and-infer-skip-cpp.py Use divided IR models which excluded 'target_conv_layer'. Conv2D Python extension is requird.

draw-and-infer

Tested environment

  • Windows 10 + VS2019 + OpenVINO 2021.4
  • Ubuntu 20.04 + OpenVINO 2021.4
Owner
Yasunori Shimura
Yasunori Shimura
AI pipelines for Nvidia Jetson Platform

Jetson Multicamera Pipelines Easy-to-use realtime CV/AI pipelines for Nvidia Jetson Platform. This project: Builds a typical multi-camera pipeline, i.

NVIDIA AI IOT 96 Dec 23, 2022
Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Fastformer-Keras Unofficial Tensorflow-Keras implementation of Fastformer based on paper Fastformer: Additive Attention Can Be All You Need. Tensorflo

Yam Peleg 10 Jan 30, 2022
This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021.

GMPQ: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation This is the pytorch implementation for the paper: Generalizable Mix

18 Sep 02, 2022
This repository is related to an Arabic tutorial, within the tutorial we discuss the common data structure and algorithms and their worst and best case for each, then implement the code using Python.

Data Structure and Algorithms with Python This repository is related to the Arabic tutorial here, within the tutorial we discuss the common data struc

Mohamed Ayman 33 Dec 02, 2022
Unoffical implementation about Image Super-Resolution via Iterative Refinement by Pytorch

Image Super-Resolution via Iterative Refinement Paper | Project Brief This is a unoffical implementation about Image Super-Resolution via Iterative Re

LiangWei Jiang 2.5k Jan 02, 2023
Official codebase for "B-Pref: Benchmarking Preference-BasedReinforcement Learning" contains scripts to reproduce experiments.

B-Pref Official codebase for B-Pref: Benchmarking Preference-BasedReinforcement Learning contains scripts to reproduce experiments. Install conda env

48 Dec 20, 2022
General purpose Slater-Koster tight-binding code for electronic structure calculations

tight-binder Introduction General purpose tight-binding code for electronic structure calculations based on the Slater-Koster approximation. The code

9 Dec 15, 2022
Moment-DETR code and QVHighlights dataset

Moment-DETR QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries Jie Lei, Tamara L. Berg, Mohit Bansal For dataset de

Jie Lei 雷杰 133 Dec 22, 2022
VISNOTATE: An Opensource tool for Gaze-based Annotation of WSI Data

VISNOTATE: An Opensource tool for Gaze-based Annotation of WSI Data Introduction Requirements Installation and Setup Supported Hardware and Software R

SigmaLab 1 Jun 14, 2022
Using fully convolutional networks for semantic segmentation with caffe for the cityscapes dataset

Using fully convolutional networks for semantic segmentation (Shelhamer et al.) with caffe for the cityscapes dataset How to get started Download the

Simon Guist 27 Jun 06, 2022
A simple version for graphfpn

GraphFPN: Graph Feature Pyramid Network for Object Detection Download graph-FPN-main.zip For training , run: python train.py For test with Graph_fpn

WorldGame 67 Dec 25, 2022
Automatic meme generation model using Tensorflow Keras.

Memefly You can find the project at MemeflyAI. Contributors Nick Buukhalter Harsh Desai Han Lee Project Overview Trello Board Product Canvas Automatic

BloomTech Labs 2 Jan 13, 2022
Pytorch0.4.1 codes for InsightFace

InsightFace_Pytorch Pytorch0.4.1 codes for InsightFace 1. Intro This repo is a reimplementation of Arcface(paper), or Insightface(github) For models,

1.5k Jan 01, 2023
Code for paper "Vocabulary Learning via Optimal Transport for Neural Machine Translation"

**Codebase and data are uploaded in progress. ** VOLT(-py) is a vocabulary learning codebase that allows researchers and developers to automaticaly ge

416 Jan 09, 2023
A practical ML pipeline for data labeling with experiment tracking using DVC.

Auto Label Pipeline A practical ML pipeline for data labeling with experiment tracking using DVC Goals: Demonstrate reproducible ML Use DVC to build a

Todd Cook 4 Mar 08, 2022
This repository contains an implementation of the Permutohedral Attention Module in Pytorch

Permutohedral_attention_module This repository contains an implementation of the Permutohedral Attention Module

Samuel JOUTARD 26 Nov 27, 2022
TorchOk - The toolkit for fast Deep Learning experiments in Computer Vision

TorchOk - The toolkit for fast Deep Learning experiments in Computer Vision

52 Dec 23, 2022
Code for layerwise detection of linguistic anomaly paper (ACL 2021)

Layerwise Anomaly This repository contains the source code and data for our ACL 2021 paper: "How is BERT surprised? Layerwise detection of linguistic

6 Dec 07, 2022
PyTorch implementation for NED. It can be used to manipulate the facial emotions of actors in videos based on emotion labels or reference styles.

Neural Emotion Director (NED) - Official Pytorch Implementation Example video of facial emotion manipulation while retaining the original mouth motion

Foivos Paraperas 89 Dec 23, 2022
QA-GNN: Question Answering using Language Models and Knowledge Graphs

QA-GNN: Question Answering using Language Models and Knowledge Graphs This repo provides the source code & data of our paper: QA-GNN: Reasoning with L

Michihiro Yasunaga 434 Jan 04, 2023