Discussion on the outline of short video technology

Time transformation , Short video from emergence to development , Experienced a gradual to rapid process , The future is full of imagination . Compared with traditional video , Its cultural and social attributes are different . That is, consumers 、 Content richness 、 Production costs 、 Production tools 、 The channel distribution relationship is different . As the bottom support , Technology brings great vitality to the development of short video , As far as the video itself is concerned , Short video technology is the same as video technology , Let's talk about it today ( short ) The technical category of video .

1、 Technology base

Short video technology is video capture first 、 Compress 、 code 、 Storage 、 Play, etc . Associated with related hardware , This is the technical basis of short video .

1.1、 Video capture

Video capture is a hardware category , Convert analog video into digital video , Save in the format of digital video file . Video capture is to simulate the camera 、 video recorder 、LD Videodisc machine 、 The video signal output from the TV , By simulating 、 Digital conversion equipment , The process of converting to binary digital information . Video capture , At present, video capture card is the main equipment .

1.2、 video compression

Obtain a large amount of analog data after video acquisition , Direct processing and transmission is unrealistic , The data must be compressed . Video compression is divided into hardware level compression and software level compression , Compression standards are associated with coding , The coding corresponds to the specific file format .

There is spatial redundancy in video data 、 Time redundancy 、 Structural redundancy 、 Knowledge redundancy 、 Visual redundancy , These redundancies provide the premise for video compression . The following figure shows a typical video compression process .

1.3、 Video coding

Video coding is through compression technology , Convert the format of the original video file into another video file . At present, the important codec standards are , ITU H.261、H.263、H.264, Motion and still images expert group M-JPEG, Group of motion experts MPEG series ,Real-Networks Of RealVideo、 Microsoft's WMV,Apple The company's QuickTime etc. .

Video compression and video coding are integrated , Each video coding corresponds to a video compression algorithm . Shannon information theory It has a wide range of guiding significance for video coding , Based on set theory , Probability and mathematical model describe the source expansion . The development of video coding is based on Shannon's information theory , Constantly improve and develop . 

Coding methods can be divided into two generations ： The first generation is based on data statistics , Remove data redundancy , Low level compression coding method ; The second generation is content-based , Remove content redundancy . The object-based method is called middle-level compression coding method , The method based on semantics is called high-level compression coding method .

Content based compression coding is a new generation of compression methods , Is the most active field , It was first developed in Sweden Forchheimer Proposed , Then Japan Harashima Et al .

1.4、 Video storage

Video storage has little to do with hardware , It is generally carried out under the operating system , Have perfect programming interface , Use traditional IO Technology to deal with . The concept of storage has also developed greatly today , From traditional stand-alone storage 、 Server storage 、 Then to cloud storage . In the past two years, there has been a hot concept , Blockchain storage . At present, stand-alone storage and cloud storage are the most used . Blockchain storage is an enhanced version of cloud storage , For blockchain storage of real video data , future 10 Years exist only in books .

1.5、 Video playback

Video playback is divided into local playback and online playback . Play locally with the meter

The emergence and development of computers , Online broadcasting comes with the development of Internet technology .

1.5.1、 There are four stages of local video playback .

A、acess visit , receive 、 obtain 、 obtain

B、demux Demultiplexing , Combined audio and video separation ( There are also possible subtitles )   

C、decode decode , Audio and video decoding

D、output Output , Audio and video output （aout and vout）

The core unit of playback is multimedia engine management , Responsible for loading audio and video data 、 Process and present . With FFmpeg For example , Its basic operation process is shown in the figure below ：

1.5.2、 There are four stages of online video playback ：

A、 Solutions of the agreement ： Remove the signaling data from the original streaming media protocol data, and only retain the audio and video data , If RTMP Protocol transmitted data , After resolving the protocol, output flv Format data .

B、 decapsulation ： Separate audio compressed coding data and video compressed coding data , Common packaging formats mp4、mkv、rmvb、ts、flv、avi etc. . The function of these formats is to put the compressed video data and audio data together , for example FLV The data in the format is output after unpacking H.264 Encoded video streams and AAC Encoded audio stream .

C、 decode ： The video 、 Audio compression encoding data , Restore to uncompressed video 、 Audio raw data . Audio compression standards include AAC,MP3,AC-3 etc. . Video compression coding standards include H.264,MPEG2,VC-1 etc. . After decoding, uncompressed video color data is obtained , Such as YUV420P,RGB And uncompressed audio data such as PCM etc. .

D、 Audio video synchronization ： Simultaneously decoded audio and video data are sent to the system sound card and video card to play .

Limited space , This topic does not explain the playback in detail . Refer to the following topics for relevant contents .

2、 The core technology

The essence of video is a collection of images , Depth processing is to process the image .GPU、OpenGL&Opengl ES、DirectX、OpenCV、DirectX、CUDA、OpenCL、Metal And other technologies are professional secondary processing of images . This is the core technology of short video .

2.1、GPU

GPU Also known as display core 、 Visual processor 、 Display chip , On personal computers 、 The workstation 、 Microprocessors that do graphics and image operations on game consoles and some mobile devices . GPU Technology is the hardware technology of image and video programming , Yes GPU Hardware 、 The understanding of instructions is of great significance to video programming .

GPU It makes the graphics card less sensitive to CPU Dependence , Carry on the part CPU The job of , Especially in 3D When processing graphics GPU essential , For example, hardware T&L（ Geometry conversion and lighting ）、 Cubic environment material map 、 Vertex blending 、 Texture compression 、 Bump mapping 、 Double texture, etc , Hardware T&L The technology is GPU Technical mark of .GPU The main manufacturers are NVIDIA and ATI.

2.2、OpenGL & OpenGL ES

OpenGL It's rendering 2D、3D Cross language of vector graphics 、 Cross platform application programming interface . From close to 350 Different function calls make up , It can draw simple graphics to complex three-dimensional scenes .OpenGL The application platform has ：Windows、UNIX、Mac OS. The implementation is provided by the display equipment manufacturer , Rely on the hardware provided by the manufacturer .

OpenGL ES yes  OpenGL Subset , For mobile phones 、PDA And embedded devices such as game consoles . from Khronos Define and promote ,Khronos It's a graphics software and hardware industry association , Focus on open standards for graphics and multimedia ,OpenGL ES It is one of the core technologies of Internet terminal graphics, image and video processing .

2.3、OpenCV

OpenCV It's based on BSD Licensed cross platform computer vision and machine learning libraries , Running on the Linux、Windows、Android and Mac OS On the operating system . Lightweight and efficient —— By a series of C Functions and small quantities C++ Class a , It provides the interface of mainstream programming language , It includes many algorithms in image processing and computer vision ,OpenCV utilize MMX and SSE Instructions , Tend to real-time visual applications .

2.4、DirectX

DirectX Is Microsoft's graphics, image and video engine , Give Way Windows Platform games or multimedia programs get higher 3D graphics 、 Image and audio and video effects , Provide common hardware driver standards for designers .

DirectX To base on Windows Computers run and display richer multimedia elements , Including panchromatic graphics 、 video 、3D An efficient platform for animation and audio rich applications .DirectX Including security and performance updates , And the advanced core functions at the bottom of audio and video .

2.5、CUDA

CUDA It's the video card manufacturer NVIDIA The new computing platform ,CUDA It's a kind of NVIDIA General parallel computing architecture , send GPU It can solve complex graphics, images and data operation problems . Contains CUDA Instruction set architecture （ISA） as well as GPU Internal parallel computing engine .

Used by developers C Language comes for CUDA Architecture writer , The program can support CUDA Run with ultra-high performance on your processor .CUDA3.0 Has begun to support C++ and FORTRAN.

2.6、OpenCL

OpenCL Parallel programming for heterogeneous systems , It's a free and open 、 Unified programming environment , It is convenient for developers to provide high-performance computing servers 、 Desktop computing systems 、 Write efficient code for handheld devices , It is widely used in multi-core processors (CPU)、 Graphics processor (GPU)、Cell Type architecture and digital signal processor (DSP) And other parallel processors . stay GPU Programming 、 Complex video processing ,OpenCL It provides a set of reference solutions .

2.7、Metal

Metal Is Apple's graphics programming library , Great progress has been made in the past two years . stay Mac The platform and IOS Platform audio and video development ,Metal Is a good choice .

Metal Provide right GPU Direct access , Technicians give full play to iOS、macOS and Apple tvOS app Graphics hardware computing potential in .Metal Built on an easy-to-use, low overhead Architecture , Provides precompiled GPU Shaders and fine-grained resource control , Multithreading support , Support GPU Drive command creation . Support GPU Array programming , make the best of  Mac Pro  and  Pro Display XDR  Professional multimedia functions .

3、 Short video SDK technology

Streaming media development has always been a professional field , Compared with conventional application information system development , There are many streaming media development specifications 、 Strong professionalism , As a result, domestic streaming media development has been dominated by shallow applications , Foreign countries mainly focus on the development of editing and compositing rendering .

Streaming media development is short video SDK The foundation of technology , More professional and comprehensive . Short video SDK Technology mainly includes beauty 、 Cute face 、 Beautiful type 、 The props 、 Filter 、 Special effects 、AR intelligence 、 Dynamic scene, etc . Short video SDK The essence of technology is the application of multimedia core technology , Secondary processing development of video .

3.1、 At home SDK present situation

Years of effort , Great progress has been made in domestic software technology , From software workshops to giants that affect the world . China is not limited to the application of foreign tools , Start in software framework technology ,SDK Technological breakthroughs .

At present, domestic short video SDK Mostly with cloud platforms 、 Live platform 、 Audio and video industry association . There are typical beautiful photos SDK、 Alibaba cloud SDK、 Tencent cloud SDK、 headlines SDK、 Hua Wei Yun SDK、 Netease Yunxin SDK、 Jin Shan Yun SDK、 Qiniuyun SDK etc. . In terms of professional and technical implementation , Beautiful photos SDK Technology is better .

3.2、 Beautiful photos SDK

Beautiful photos SDK Provide all-round technical support for short video . Meishe technology has long been committed to software research and development in the field of radio and television , He has many years of research and development experience in nonlinear editing . In order to adapt to the development of Internet video , Americo's Internet video business line is committed to the bottom development , Provide audio and video editing services for software companies SDK, So that the majority of programmers in the absence of audio and video technology , Quickly and efficiently develop high-quality audio and video programs .

Americo technology is committed to the bottom research and development , Complex audio and video signal processing in development 、 Hardware processing 、 Data transformation is encapsulated , So that the majority of developers can adopt Meishe SDK Development process , Avoid contact with the bottom layer . So that the software developers of other technologies , Efficiently develop professional high-performance video software .

Beautiful photos SDK It's a toolkit , Right picture 、 Audio and video processing . You can edit streaming media . Including video shooting 、 edit 、 synthesis 、 Special effects 、 Make audio and video animation . Help program developers quickly carry out audio and video 、 Image rendering processing . Quickly develop high-performance audio and video programs . Meet the needs of the development and application of audio and video in the Internet era . Beautiful photos SDK stay PC End 、 Mobile 、WEB End 、 The server side provides all-round technical support for short video processing .

Beautiful photos SDK Features include ： video recording 、 Video editing 、 Music editor 、 Caption processing 、 Animation stickers 、 Transition 、 Special effects 、 Synthesis and so on .

4、 Epitaxial technology

No video , No network . Short video technology is widely used on the Internet platform . Epitaxial technology is also mostly related to video platform . The popular technologies in recent years are rtmp、 Push flow 、 Pull flow 、HTTP、CDN、 Load balancing 、 The server 、Android End 、IOS End 、 Desktop, etc .

4.1、HTTP

HTTP It's a simple request - Response protocol , Running on the TCP above . The communication mechanism between client and server is agreed . The header of the request and response message is ASCII Code form given ; The message content has a similar MIME The format of . This simple model is Web Made outstanding contributions to early development , Make the development and deployment simple and convenient .

Business flow processing of short video technology platform 、 Logical processing is attached to Web platform ,Web The platform is based on HTTP Protocol standards are established .

4.2、CDN

CDN It's a content distribution network ,1996 year , MIT , A research group proposed to improve the service quality of the Internet . In order to be able to IP Publish a lot of broadband media content on the Internet , It is proposed to establish a content distribution platform based on the existing Internet , Dedicated to providing services for websites ,1999 In, a special service company was established , by Yahoo Provide services .CDN It is a network coverage layer optimized to speed up network access , So it's figuratively called “ Network accelerator ”. 

CDN The Internet improves the service quality of the Internet , Large network operators began to build their own networks CDN The Internet , Such as AT&T、 Telecom Germany 、 China Telecom, etc . Short video technology has network properties , It is closely related to the network platform . Short video, huge data stream , Need to attach CDN distributed .

4.3、 Push flow Pull flow

Streaming and streaming are professional terms of video platform , Streaming is to transmit the encapsulated content in the collection stage to the server . The process of transmitting the live video signal to the network . Pull streaming means that the server already has live content , The process of pulling from a specified address .

Push stream and pull stream have high requirements for network , Network instability ,（ live broadcast ） The playback effect will be very poor ,（ live broadcast ） Caton and other phenomena will occur when playing , Poor effect and experience .

4.4、RTMP

RTMP Real time message transfer protocol , yes Adobe The real-time messaging protocol for audio, video and data transmission between servers developed by the company .RTMP Based on the agreement TCP, Include RTMP Basic agreement and RTMPT/RTMPS/RTMPE And so on .

RTMP Agreement , The video must be H264 code , The audio must be AAC or MP3 code , And more flv Format packet .RTMP It is the most mainstream streaming media transmission protocol at present , Yes CDN Good support , Less difficult to achieve , It is the choice of most live broadcasting platforms .

RTMP There is one deficiency , Browser not supported ,Adobe No more updates . If the live broadcast service needs to support the browser , Need additional push protocol support .RTMP And HTTP equally , All belong to TCP/IP The application layer of the four layer model . At present, short video streaming RTMP Protocols are widely used .

4.5、RTSP

RTSP It's a real-time streaming protocol , yes TCP/IP An application layer protocol in protocol architecture , By Columbia University 、 Netscape and RealNetworks Standards submitted by the company . Defines how one to many applications can effectively pass through IP Multimedia data over the Internet .RTSP Architecturally located in RTP and RTCP above , It USES TCP or UDP Complete the data transfer .

HTTP And RTSP comparison ,HTTP The request is made by the client , Server response ; Use RTSP when , Both the client and the server can make requests , namely RTSP It can be two-way .RTSP It's a multimedia streaming protocol used to control sound or video , Allow multiple stream demand control at the same time , The server can choose to use TCP or UDP To deliver streaming content . relative RTMP,RTSP Yes UDP The choice of .

4.6、HLS

HLS yes Apple The definition is based on HTTP Streaming media real-time transmission protocol . The principle is to divide the whole stream into several small files to download , Download only a few at a time . The server generates new small files from the latest live broadcast data , The client keeps playing the files obtained from the server in sequence , It's live .

HLS It is the on-demand technology that realizes the live broadcast experience , The length of each small file is very short , The client can quickly switch the bit rate , To adapt to different bandwidth conditions . Technical features of segmented push , To determine the HLS The delay is generally higher than that of ordinary streaming media live broadcast protocols . Today's short video platform , Pull flow HLS Protocols are widely used .

4.7、 Load balancing

Load balancing is based on the existing network structure , Provide cheap ways to expand the bandwidth of network devices and servers 、 Increased throughput 、 Strengthening network data processing capacity 、 Improved network flexibility and availability .

Load balancing means that the load is distributed to multiple operating units for execution , for example Web The server 、FTP The server 、 Application server, etc , Work together , Meet high concurrent access . Short video platform is a traffic platform , Must have high concurrency property .

4.8、WebRTC

WebRTC, Web im . Support browser for real-time voice 、 Open source protocol for video dialogue .Google、Mozilla、Opera Push WebRTC Become W3C The recommended standard of .

WebRTC Support current mainstream browsers , be based on SRTP and UDP, The network signal also has good stability in general .WebRTC Realize point-to-point communication , The delay of both sides of communication is low , It's the realization of “ Lian Mai ” Better choice of functions . On the live platform 、 Short video platform has practical applications .

4.9、OBS

Open Broadcaster Software Is a video recording and video real-time streaming software , Widely used in video capture 、 Live broadcast and other fields .

OBS The following functions are supported ： Support H264（X264） and  AAC  code ; Support Intel Quick Sync Video (QSV) and NVENC; Unlimited number of scenes and video sources ; Support real-time RTMP Stream push ; Support  MP4  and  FLV  Format output ; Support capture based on  GPU  Game Capture , High performance output video stream ; Support DirectShow Acquisition device ; Bilinear or lanczos3 Resampling support, etc .

4.10、 client

At present, short video technology clients are mainly Android App、IOS App、 Desktop application . Use the relevant technology of the topic , Short video for terminal customers to operate applications .

5、 summary

Great experimental psychologist Treicher confirmed , Human access to information 83% From vision ,11% From hearing , That adds up to 94%. also 3.5% From smell ,1.5% From touch ,1% From taste . Unfortunately , On the vast Internet , There is little more information about the master .

The epidemic forced short videos to accelerate the upgrade ,2020 Usher in the bonus period . future 10 year , online 70% Traffic in short videos . Studies have shown that , human beings 91% My memory comes from “ Images + voice ”, and “ Vision + auditory ” The best carrier is short video .

Technology has revolutionized video , Short video technology is in the ascendant ,SDK Applications are surging , The development of financial media is booming . I firmly believe that , Short video applications can be expected in the future , American Photography Technology SDK Will make use of the technical advantages of traditional video and audio processing , Provide developers with more innovative playing methods and imagination space , Helping partners achieve business innovation and create huge user value will , The great vision of promoting the overall development of the Internet industry , Accelerate the empowerment of the Cenozoic , Embrace a new era of digital creativity .