From above , This article will explain the route of the shopping guide system project in detail .

Commodity selection module

The first is our commodity selection module , We use mediapipe Human feature point detection framework , Detect the of human trunk 33 Two feature points and hand 21 A key point , In order to reduce the confusion caused by multiple key points , We use the palm of our right hand as the only positioning point , The projection effect is realized through the screen spacing ratio , Calculate the projection point of the key point of the user's palm on the screen , Through coordinate transformation to achieve the effect . adopt xy The normalized value displays the position information of the identification point on the plane , adopt z The value shows the depth information of the identification point , Achieve three-dimensional detection effect .

Timestamp synchronization

MediaPipe The diagram structure is decentralized during execution ： No global lock , Different nodes can process data with different timestamps at the same time . This allows the pipeline to have higher throughput . However, time information is very important for many perceptual workflows . Nodes that receive multiple input streams at the same time need to coordinate them in some way . for example , A target detector may generate a series of candidate boxes , Then this information is transmitted to the rendering node , This node should be processed with the original frame .
therefore MediaPipe One of the main functions is to synchronize node inputs . In terms of framework , The main function of timestamp is to act as synchronization key . Besides ,MediaPipe Designed to support deterministic operations , This is in many scenarios ( test 、 simulation 、 Batch processing, etc ) It's very important , At the same time, it allows the graph designer to relax the certainty where the real-time constraints need to be met .
The two goals of synchronization and determinism are the basis of several design choices . It is worth noting that , A given packet must have an incremental timestamp : This is not just a useful assumption for many nodes , And the synchronization logic also depends on this . Each data stream has a timestamp limit , This is the minimum timestamp allowed for new packets on the data stream . When a timestamp is T When the packet arrives , The boundary is automatically pushed to T+1, Reflects monotonous requirements . This allows the framework to determine that no timestamp is less than T The packet will arrive .

Enter policy

from DefaultInputStreamHandler The defined default input policy provides deterministic input synchronization , Packets with the same timestamp on multiple input streams can be guaranteed , The input data stream is processed in strict ascending order according to the timestamp . The cell-based approach allows the graph to control where packets are discarded , It also allows flexible adaptation and customization of graph behavior according to resource constraints .

GPU Calculation and rendering

MediaPipe Support for GPU Cell nodes for calculation and rendering , And allows you to merge multiple GPU node , And combine them with CPU Cell node mix .MediaPipe in GPU The design principle is to ensure GPU Computing units can appear anywhere in the diagram , Frame data in GPU Copying from a cell to another cell should not be required ,CPU and GPU Data transmission between should be efficient .
MediaPipe Allow the graph to be in multiple GL Run... In context OpenGL. for instance , This can be very useful in graph structures , Combine slower GPU Reasoning path ( for example , stay 10 frame / second ) And faster GPU Render path ( Such as 30 FPS)： Because a GL The context corresponds to a continuous command queue , So using the same context for both tasks will reduce the frame rate of rendering .
MediaPipe One challenge addressed with multiple contexts is the ability to communicate across them . For example, an example scenario , Send the input video to the display and reasoning path at the same time , Show the results of the reasoning that needs to be accessed first .
One OpenGL Context cannot be accessed by multiple threads at the same time . Besides , In some Android On the device , Switch activities on the same thread GL Context can be slow . therefore , Our approach is to set up a dedicated thread for each context . Each thread emits GL command , Establish a serial command queue in its context , Then from GPU Asynchronous execution .

Virtual fitting module

It integrates the ability to understand the real environment and the ability to render the model , The ability to understand the real environment includes identifying the plane of the real world 、 The ambient light 、 Human body, etc , Model rendering capabilities include models and real-world planes 、 The ambient light 、 Interactive rendering of human body, etc . After identifying the ground , Place the model on the ground for display .

Use this component , You can easily place a virtual model in a real environment , After the virtual and real main body shielding switch is turned on , If the virtual model interacts with the human body , For example, the virtual model is blocked by the human body , The blocked part will be deducted and displayed , Achieve a more realistic AR Display function .

It integrates the ability of face understanding and model rendering , Among them, face understanding ability includes face position 、 Pose recognition and facial expression recognition , Use this component , Easily realize face AR effect .

meanwhile , Just load the face model with expression , That is, the real facial expression can be used to drive the model , Realize the real-time expression change of the model .

Hongmeng app

Our front-end uses Hongmeng app.
Hongmeng system was officially released at Huawei developer conference , It is a distributed operating system for the whole scene , With distributed soft bus 、 Distributed data management 、 Features of distributed security . Distributed soft bus allows multiple devices to be integrated into one device , High throughput within and between devices 、 Low latency 、 Highly reliable smooth connection experience . Distributed data management makes cross device data access as local data access , Greatly improve the performance of remote reading, writing and retrieval of cross device data . Distributed security by defining the security level of data and devices , Classify and protect data and equipment , Ensure safe and reliable data flow .

Use distributed Hongmeng app Replace traditional Android Applications , We have the following advantages ：
1、 Increase system capacity . We're doing more and more business , And to be able to cope with the increasing volume of business , The performance of a machine is not enough , We need multiple machines to cope with large-scale application scenarios . therefore , We need to split business systems vertically or horizontally , Make it a distributed architecture .
2、 Enhanced system available . Our business is becoming more and more critical , We need to improve the availability of the whole system architecture , This means that there can't be a single point of failure in the architecture . such , The whole system will not be unavailable due to the failure of one machine . therefore , We need to use distributed architecture to redundancy system to eliminate single point of failure , So as to improve the availability of the system .
3、 The reusability of system modules is higher

Collaborative filtering algorithm

Collaborative filtering , Literally , Including collaboration and filtering . The so-called collaboration is to use the behavior of groups to make decisions ( recommend ), There is a biological theory of coevolution , Through synergy , Let the group evolve to a better state . For recommendation systems , Through the continuous synergy of users , The final recommendation to users will be more and more accurate . And filtering , Is to make feasible decisions ( recommend ) programme ( Subject matter ) The scheme that users like ( Subject matter ) look for ( Filter ) come out .

say concretely , The idea of collaborative filtering is to find some similarity through group behavior ( The similarity between users or between objects ), Through this similarity to make decisions and recommendations for users .
Here, we realize the clothing recommendation effect through collaborative filtering algorithm , Displayed in Hongmeng app On , Generate unique personalized tags for users , Divide the crowd into small groups through personalized labels , Realize the data sharing of favorite clothing in small groups , That is, like to recommend .

Hardware exploration

Used raspberry pie before , The code that carried the project on the raspberry pie , But because the raspberry pie has too low computing power, it leads to frame loss 、 Frame loss, etc .
The temperature monitoring module of little bear pie is being developed , Use bear pie , adopt STM32 The hardware IIC obtain SHT20 Temperature and humidity data , And print in the serial port . Master the bear hardware IIC The use of and SHT20 Data reading , Upload the read data to Huawei ECs , And process the data in web End to end rendering .