Application practice of low-latency real-time audio and video in 5G remote control scenarios

    首先,Introduce the application of remote real-time control to you.

    What is a remote real-time control application?？We give a simple definition：突破空间限制,Remote control of people and objects with the help of network and audio and video capabilities,Experience close to field operations.From this definition, we can see that the medium of interaction in remote real-time control is mainly the network,The target effect of interaction is mainly close to the experience of live operation,The main value of interaction is to break through space constraints,To liberate people and things interact space distance.Compared with the original traditional remote control,The remote real-time control is mainly enhanced in the following three aspects.The first is that the information is more specific,In addition to traditional sensor information, there are often multiple video channels in remote real-time control.,These videos from multiple perspectives will be presented in all directions at the remote end to allow the operator to achieve an immersive experience.The second is to control more realistic,To improve control efficiency,In remote real-time control, control equipment similar to the original equipment is often used,For example, a car will use an analog steering wheel,Dig opportunity with corresponding control mechanical lever in order to improve the efficiency of the manipulator.The last and more important point is that in order to maintain the smoothness of operation in remote real-time control,Its interaction needs to be more real-time,The remote operation and on-site feedback should be seamlessly connected,Makes maneuvering smooth.

   Next, I would like to introduce to you why remote real-time control has risen in the past two years,It is mainly inseparable from the driving of two technical directions.第一个是5G：5GThe technology itself can improve the uplink bandwidth and network latency of the network,Provides an important network foundation for remote real-time control scenarios.In addition, in the past two years, the country5GPolicies are also promoting the rapid upgrading of the digital industry,像在2020年,工信部、网信办、Ten departments including the Development and Reform Commission have launched《5G应用“扬帆”行动计划（2021-2023年）》,which mentioned that the port involved、矿山、factories, etc5GIndustry remote application scenarios,To achieve unmanned on-site、少人化作业,Improve from local work to remote work.The second is the maturity of autonomous driving technology.In fact, after several years of development of autonomous driving,At present, we are gradually trying to enter the unmanned stage in some scenarios.,Realizing the business value of autonomous driving by removing local safety officers from vehicles.But there are still some unexpected situations in autonomous driving that need to be solved.International last year（SAE）和国内（GB）In the automatic driving classification standard,remote intervention,Defining Remote Drivers to Address Takeovers in Remote Driving Unexpected Situations,应对L3/L4Intervention problems in accidental situations of staged autonomous driving,Completed the transformation from the original local security officer to the remote security officer.

    下面,Mainly introduce the application scenarios of remote real-time control,We can roughly divide it into three categories of scenarios.The first type of scenario is the emergency takeover of autonomous driving,In this type of scenario, it is mainly through the remote remote supervisor to take over the unexpected automatic driving situation,In order to achieve a remote security officer to watch multiple unmanned equipment at the same time,提高效率.At present, the more mature scenarios are mainly unmanned mining cards in mines and unmanned collecting cards in ports,Has now entered the actual landing operation stage.另外,Park logistics and passenger car scenarios are also exploring unmanned remote control.The second type of scenario is high-risk and harsh environment operations,It is also a direction supported by the state.,It can liberate the original operators from the difficult working environment,Working in an air-conditioned room is no longer a dream.At present, there are mainly these types of application scenarios：One is that mine drilling rigs and underground roadheaders need to be operated remotely;In addition, there are also some high-altitude working environments in the port.,Like field bridges and tire cranes;There are also higher temperatures in the steel workshop,At this time and try to be less humanizing,In addition, remote excavators and drones can be used to deal with emergency natural disasters.,Reduce the risk to personnel.The third scenario is online-to-offline service and experience,Traditional real-time audio and video are more online interactions between people,But remote control brings new possibilities for online-to-offline interaction,This is also a direction that is more suitable for mass innovation..Online to offline interaction is mainly divided into two aspects,One is online manpower to offline services,For example, cloud driving and parking in travel scenarios、Real estate built figure scene Ozzie robot、Remote for medical scenariosBChaodu allows online doctors and staff to serve on-site remotely.The second aspect is that online customers experience offline resources in a convenient way,Such as the drone cloud tourism scene,At home you can gain actual travel experience,View the corresponding landscape.And the scene of cloud racing,You can experience the racing service in the special arena on the remote line.

   The following may be a topic that people care about,What is the video indicator for remote real-time control?？In fact, the challenge of remote real-time control to real-time audio and video is relatively large,We are currently developing a5GThree more important indicators are excerpted from the remote driving standard.The first is the screen delay,Refers to the delay from the capture of the camera to the remote display,It contains the acquisition delay of the camera、Video transmission and display three parts of the display.Screen delay indicator for remote real-time control,is defined in the standard150毫秒到200毫秒,But in practice the more desirable indicator is100毫秒到150in the millisecond range.In remote control, it is often necessary to use a camera device with a car specification or a higher specification,The latency of such devices is much higher than that of mobile phones orUSBcamera delay.Typical values ​​for camera latency like car gauges are at70毫秒到100毫秒之内,This means that the delay time left for video codec transmission and display is less than100毫秒.The second is the stall rate,This is also a very demanding indicator,Because many objects are moving in the remote control scene,It is easier to feel the stuttering in the moving picture,The frame interval defined by the standard is100毫秒到150毫秒,The traditional audio and video definition limit is200毫秒.The requirements for the freeze rate have also increased10倍,The traditional audio and video freeze rate indicator may be1%,And the remote control standard changed the value to0.1%,Of course this is normal value,If there is an extremely weak net,It's still hard to achieve.The third is the anti-weak network indicator,Remote real-time control against weak network requirements is not so high,This mainly takes into account real-world network scenarios,Generally, the remote control network is relatively better.,标准定义是20%丢包和50millisecond network jitter,Under such indicators, it is possible to achieve1%的卡顿率.

   The following mainly introduce tencent in the remote controlRTCSome experience with products.

   The first is product architecture.From this structure,It is similar to the traditional real-time audio and video structure,It is also divided into the gateway side of push flow、Watch the control side and the service side of the pull stream,The service side is also the traditional 3 big（信令服务器、媒体服务器、存储服务器）.slightly different because5GRemote control security requirements,It is necessary to increase the authentication of the device on the signaling server for more robust security protection.另外,From the deployment scenario,On the media server and storage server, some lightweight improvements need to be made to improve the scalability of the scene and the ability to deploy across regions.在端侧,It is roughly divided into several parts that are different from traditional audio and video.第一个是5GThere are many remote control scenarios,A bit of a point-to-point connection,There are also server transfer scenarios,So it may need to support multi-mode video streaming.The second is that the requirements for the transmission of status and control signaling in remote control are very high,Not only need low latency and continuous transmission,Therefore, corresponding enhancements are also required on the control channel..The third point is some changes in the operating platform,Because the remote control is different from the traditional actual audio and video scenes,It's mostly somethingLinuxIoT devices are also availablewindowsthe control end,In such cases it support platform will change too.

   下面,Introduce the characteristics and indicators of Tencent remote control products.The first is low latency,one of the most important indicators.We have achieved end-to-end picture delay,本地100毫秒左右,省内150毫秒左右,全国200毫秒,and provides frame-by-frame video latency measurement,Help customers to do some fallback strategies in the case of high latency.The second is safe and controllable,Because remote control requires very high security,On the basis of encrypted transmission of traditional audio and video stream content,We have added device access authentication and video rights management to help customers better provide security protection.The third is the anti-weak network,While achieving the ultimate low latency, it also retains the ability to resist weak networks equivalent to the current real-time audio and video,可以抵抗30%丢包和50Milliseconds above network fluctuations,And there are some blind spots for the network coverage of some ports or mines,In this case we also provide less than1Second network interruption fast recovery ability.The fourth is full network scenario support,Scenarios where there is a public network in remote control,乘用车、Public network scenarios such as unmanned taxis,There are also private network scenarios,Like mines and ports,There are also some hybrid network scenarios,So we provide an automatic and fast path selection function,It can automatically find the nearest access route for corresponding access to reduce end-to-end delay.

   接下来,Share two application cases of Tencent remote control products.One is the port,一个是矿山,In both cases we did it locally5GThe screen delay under the private network is reduced to120毫秒左右,This is based on the car gauge camera,That is to say, the real video transmission may be in50毫秒左右.The highlight in the case of the port on the left is that it supports unmanned driving monitoring and remote driving,Help customers to realize a practical unmanned operation.The main highlight of the case of the mine on the right is that we have demonstrated remote control of the mine truck in Ordos at the Guangzhou and Wuhan exhibitions.,The control distance from Wuhan to Ordos is1000多公里,And guangzhou to ordos2000多公里,At such a distance scale,Continuous driving is achieved,The picture delay is less than200毫秒.

   那么,You may be concerned about the new technical points behind Tencent's remote control products.

   这里,We simply share a few technical points.The first is the congestion control technology,under a low latency budget,Congestion control algorithm in video transmission need to be better,Once the network is congested,Bit rate can be reduced as much as possible,避免拥塞发生,Reduce retransmission of lost packets and increase in delay caused by congestion.We added to the traditional congestion algorithm5GAir interface congestion perception link,Including based on air interfaceBuffer的测量, IECNCongestion feedback field scan and signal quality andMCSMade an estimate of the transmit code rate,以减少网络拥塞的发生.

    A feature of remote control video is multi-channel video transmission,Most of the cases like we do have at least4video transmission,In some special machinery there will be6-8road video transmission,And more video transmissions can have appropriate trade-offs when congestion occurs,Generally, we need to ensure the clarity of the main screen,And for the sharpness of non-main viewing angles can do with some additional drop.基于这样的情况,We do a joint congestion control,The sending rate of the entire network is shared,In the congestion control algorithm, the corresponding output transmission rate is allocated to the corresponding video stream based on the weight.In this way, the quality of multiplex video transmission can be flexibly chosen.

      The second is anti-weak network technology,It is also very important in low-latency transmission.The first thing to talk about is the retransmission technology..Retransmission is currently one of the most effective techniques to resist packet loss,But retransmissions are highRTTDecoding will be added belowready时间的波动,Brings longer cumulative delay at the receiving end,As a result, the delay of the entire end-to-end picture increases.In order to reduce the delay fluctuation caused by retransmission,We made an algorithm for encoding streams across frames,The encoded stream is similarSVC,Two-layer sharding is done,后一个PThe frame will refer to the previous one and the previous oneP帧,Form a cross-frame reference.The lost package has several advantages,The first is when the previous frame was lost,The current frame is decoded without waiting for retransmission,This effectively solves the delay fluctuation problem caused by retransmission.另外,is when severe packet loss occurs,The retransmission and transmission of one of the fragments can be discarded as a whole,Stream video at half frame rate,Reduced transmission bandwidth,Alleviate congestion.

      Anti-weak network in addition to the retransmission just mentioned,Another important technique isFEC.Multi-channel video toFECMixed with more dimensions,In the case of multi-channel video,We in order to reduce the effects of packet loss,Transmit interleaving for transmitted multiple video streams,For sudden continuous packet loss will be scattered on every single video all the way,Pass each way your ownFECCoding done fixes.另外,In order to balance the relationship between redundancy and error correction ability,You can also superimpose on the sendFEC,The redundant package added here can be shared by multiple video channels.

     In fact, the most direct and effective way to reduce video delay is to improve the network,Choose a network with lower latency for transmission.在5GRemote control scene in the network,大部分caseThe equipment side and the operator side are in the same city or in the same park.In such a city and local scene,The video link can have a variety of options,Such as through direct connectionP2Pway to do the link,The second is through the edge of the media server to do the edge of the forward,The third is to forward through the cloud in the traditional way,However, the network link delay of the cloud forwarding method tends to be relatively large..To support flexible forwarding links,We have done multi-mode support on the end side,P2P+边缘分流+There are three forms of remote forwarding,Provides smart scheduling options.

   The following is the actual test data of our delay,The test camera is here1080P 30fps GMSLⅡ车载相机,Its delay is about 70毫秒左右,我们在WIFIin a local area network scenario,网络RTT大约在1-3毫秒左右,The entire screen is delayed by approx90+毫秒.第二个是在5GCity network to do the test,网络RTT大约在30毫秒左右,The picture delay rises to approx.120毫秒.

   Here's another camera,型号是USB3.0 720P工业相机,The overall delay of this camera is much smaller than the previous car network camera,Its camera lag is about20毫秒左右.我们在WIFIIn the LAN scenario, the screen delay is approximated40+毫秒,This is a very low latency.Some drones like DJI are also difficult to do with such clarity50毫秒以内的延迟.右边是在5GCity network to do the test,网络RTT大约在30毫秒左右,The screen delay is done70milliseconds.

    我的分享就到这里,感谢大家.