Experts have information | Zhang Zuyou: Tencent cloud devsecops practice and open source governance exploration

In recent days, ,CIS2021 Network Security Innovation Conference DevSecOps The special session on application and technology was successfully held in Shanghai . This time Special session by DevSecOps Agile security leaders —— Founder and of hanging mirror safety CEO Ziya product , focusing DevSecOps Technology development and application practice . At the meeting , Many experts gave keynote speeches and round table forums in combination with their own research or business , Wonderful . among , Tencent cloud product security director 、 Zhang Zuyou, safety director of Tencent Yunding laboratory Shared Tencent cloud in DevSecOps And practical experience in open source governance , The following is a transcript of the speech ：

The theme of my speech is “ Tencent cloud in the past DevSecOps practice ”. Let's make a simple self introduction first , I am now in Tencent security Yunding Laboratory , Do cloud security and be responsible for the security of Tencent cloud products .

First, share the challenges of Tencent cloud security . There is a big difference between Internet companies and traditional companies such as financial companies , The overall infrastructure and R & D system are not unified , Different departments 、 The R & D systems of different centers are different , And Tencent has a lot of DevOps Platform or code analysis platform . Besides , Tencent cloud has 300 A variety of first-class products , The number of products subdivided into class II or sub class will be more . So when we do the product safety of the whole business line or business group , The challenges are relatively big , It's hard to land . imagine , Security needs fall into the R & D and operation and maintenance system , If the R & D and operation and maintenance systems are not unified , So it's hard to be safe , Therefore, for the construction of the whole Tencent cloud product security system , We are thinking about how to do . In the early days , We adopt a vulnerability defense system , More from the perspective of vulnerability convergence , Mainly manual vulnerability mining , Although this method is fast and effective , It may be applicable in the early stage , But later you will find , In fact, there are omissions in this way , So we are now using more life-cycle convergence .

The following figure is our system diagram based on the risk introduction process , It is divided into risk introduction 、 Risk discovery and protection 、 Response disposal and closed-loop improvement , It basically covers the activities that can be carried out in all dimensions of product safety .

  Based on the risk introduction process for convergence

 

In the overall construction process , We divided 4 Stages .

Because it may be the same as traditional manufacturers , We need to do more anti invasion , So in phase zero , What we do is border control , For some open assets , For example, Internet IP、 Domain name etc. . In the following first stage , Is to supplement security capabilities and security activities from scratch .

The second stage is to pursue fine operation . Inside us , There are several forms of products , Including... Introduced by third parties 、OEM And cooperative research and development , In addition, online products will be involved, including privatized delivery products . These types of products , The focus is different , For example, online products , You may pay more attention to whether it has the actual vulnerability risk that will lead to intrusion or control of the server ; and OEM Or jointly developed products , Its R & D stage may not be within the enterprise , Therefore, a security access mechanism is adopted ; Like privatized products , For example, Tencent cloud privatized products , In fact, it does not affect Tencent's own safety , Because it is delivered to the customer , But safety issues affect product reputation . So we take more refinement in this stage of operation , It is to build different systems for different product safety scenarios .

In the third stage , What we want more is to improve efficiency , Increase business team involvement in security , Measures to improve security , So at this stage , Our focus is on practice DevSecOps. Why practice DevSecOps？ This may be understood by everyone , The purpose of security shift left is actually because of the convergence of the whole vulnerability , The lower the processing cost in the more left stage . What used to be known was Microsoft SDL That mechanism , Now DevSecOps Why is it so hot ？ Because DevSecOps And SDL There is a big difference between ： stay SDL in , A lot of times it's safe to tell the truth , Security is always beyond R & D, operation and maintenance , however DevSecOps More emphasis is on , Security governance decisions are not solely the responsibility of the security team , Development 、 Even QA, Are one of the roles of security governance , It requires everyone to be responsible for safety , Embed security into the development process system . The requirements for safety are different , The reason lies in the change of R & D operation and maintenance mode , And security is to adapt to its own R & D operation and maintenance mode . The reason why Tencent cloud landed DevSecOps, Because inside Tencent , Tencent cloud is also a relatively early practice DevOps Business line of , Then we need to follow up from the perspective of security DevSecOps.

DevOps Patterns focus on technologies such as minimum modularity , This may not be possible in the traditional mode . for instance , When Tencent conference is still an internal application , Its security is what we do . Later, Tencent conference was widely used ,260 Oh, my God 29 A version , For this iterative speed, if we still use the traditional security mode to do security construction , I'm sure I can't keep up with , Even then we set up a special project team to ensure its safety . So we need a new security model, that is DevSecOps.

In my submission SDL and DevSecOps There is no great conflict between the two , Not a replacement relationship , Just adapt to different scenes . Compare with SDL Pattern ,DevSecOps What we pay attention to is the integration of more automation 、 Agile iteration . And through the annual RSAC The theme of the meeting can be seen ,DevSecOps Itself is an evolving security framework , It's not static .DevSecOps Divided into 10 Stages , Among them the first 6 This stage was originally called configure To configure , Now called prevent The prevention of . It itself is a theory that is constantly improving , So every company is interested in DevSecOps The use of is different , It should adapt to its own R & D operation and maintenance system . We are combing the past security activities of Tencent cloud , It can be found that many safety activities in the past are in Ops（ Operation and maintenance ） Stage , For example, often do black box test , Do relatively little before going online . So we choose to practice DevSecOps The system is based on the results of previous combing and summary , More activities need to be added during the development phase .

DevSecOps The core of is the process 、 Technology and culture , The hardest thing is culture . There are four key points for the implementation of the whole system ： At the bottom is the construction of tool chain ; The second is the core automated testing capability , such as IAST、SCA The ability of ; The third is based on CI/CD Secure embedding of pipeline ; Finally, we need to pay attention to some new key technologies , Like container security 、API Security .

Actually building DevSecOps System time , We have two stages . The first stage is the internal multi platform stage of Tencent cloud ,DevOps Platform and code analysis platform are not unified . So we adopted a way , Is to adapt the security capability to multiple platforms , Encapsulate various security tools into multiple DevOps platform , In some unified platforms, such as Git Warehouse , We'll do an active scan . Then the business coverage is counted and confirmed through unified data reporting , And realize the alarm of security problems through a unified security operation platform 、 Measurement and other work .

It's the second stage , A technical committee has been established in the industrial business group within Tencent cloud , Unified DevOps platform , So the landing of the security system has also changed the direction , We put the security capability into the function of this unified platform . This unified DevOps The platform contains several different stages of functionality , Therefore, we need to analyze which corresponding security capabilities can be directly embedded in the functions based on different functions , At present, the more mature one is development 、 There are three stages of testing and product management . The local development phase is mainly based on DevOps Local pipeline of the platform , adopt ECI To implement ; Code analysis phase , We mainly embed SAST Code component analysis 、SCA And sensitive information checks ; The automated testing phase mainly includes black boxes DAST as well as IAST, About IAST, In addition to relying on internal capabilities , I'm also trying the spirit pulse of hanging mirror IAST; More capabilities are required in the product management stage , Such as container image scanning 、 Binary file analysis 、 Virus, Trojan horse detection, etc .

Why do we need to package tools here , A very important point is that many of the security tools originally used are now available DevOps Not necessarily applicable on the assembly line , For example, there are some security engines developed by ourselves in the past , Put it on the assembly line for business students to execute , But it takes more than ten hours , This is unacceptable to the business team . So the original tools can't be used directly after being taken over , It involves encapsulation . There are also local pipeline inspection and online pipeline inspection , The complexity of the rules is different , Local is relatively simple , Online may be a little more complicated , But also consider the time factor . And for some aspects that we can't touch at present , More will use the way of commercial procurement to quickly supplement .

Code analysis phase , Just now we talked about three abilities ,SAST、SCA And sensitive information checks , Mainly in local and CI/CD The two scenes are embedded . Automated scanning tools SAST Is directly embedded . The automated testing phase involves DAST, Mainly through the traffic forwarding of the test environment , Just now, Ziya, the safety of hanging mirror, also introduced , To be exact, this is in a broad sense IAST. We have done a lot in this area in the past , It is to realize the scanning before going online through traffic forwarding . On the other IAST, We mainly integrate and test in the test environment .

WIP safety this stage , Our platform has a product library capability , You can directly integrate security checks in the product library , That is to say, all DevOps The products produced in the process will be placed in the product library , By default, all products in the product library should be subject to safety inspection and safety tips . However, it is still impossible to intercept the download , This is the next step to improve , We will be in CD Add an ability to identify products with safety problems , Intercept it , It is not allowed to use .

At an earlier stage of threat modeling , We adopt a semi-automatic way , Through questionnaires and automated API Call this combination of multiple modes to achieve semi automation . For most products , We let the business fill in the questionnaire of the automatic evaluation process by itself , Then the system makes an automatic judgment , Judge whether it is reasonable and identify which points are at risk . For some key products , For example, Tencent Conference , Manual review may also be involved , Before the new feature goes online , Mainly in the form of manual evaluation .

This part of the security coding specification , It is based on the inspection of the company's overall coding specification . We actually have two versions of the secure coding specification , One version is an open source version of Tencent based on the development language , There was another version based on vulnerability dimension .

We are also trying framework security or default security , For example, in governance SSRF（ Server side Request Forgery ） When there's a leak , We try to do one-stop SDK（ Software development kit ）, So that the business does not need to consider too many R & D problems when accessing .

And in DevOps In the platform , We consider adding a security view . This security attempt allows business personnel to see the security related situation and access situation in one stop . Because there is too much business , Of every business DevOps Maturity and R & D, operation and maintenance processes are different , There is no uniform standard for , So we are right DevSecOps Unified process requirements and standards are graded , It is divided into 1 To 4 level . The unified requirement is 1 level , For some DevOps A highly mature business , For example, Tencent Conference , The level requirements will be higher .

It will also involve building metrics , In two parts , Part is the measurement of business security , For example, measuring whether the product is safe , The other part is about DevSecOps Measurement of practical effect . For the latter , There are three main dimensions , One is capacity-building , One is operation promotion , The last one is the effect .

After so much discovery work , It also involves risk convergence . Risk convergence is differentiated , Because it is bound to the overall security measurement . We internally implement an index called security reputation score , This is a great deterrent to the business , Because if a team score is less than 80 branch , The team leader is required to explain . Because we should encourage businesses to find problems before going online , Therefore, the measurement of security reputation points will be after going online , And there will be a mechanism to measure punishment , But it will not be included before going online . There will be a difference in convergence , For example, some assembly lines before going online , We are more likely to intercept through access control , The relevant parts of active scanning are not included in the measurement mechanism , Through the risk disposal platform to promote . Routine online scanning may be included in the safety work order to follow up . also Nday and 0day The vulnerability is unexpected , We won't include .

Finally, let's talk about open source governance . I think it's difficult to do open source governance in Internet companies , The reason is that we can't control the whole process from in to out like other companies , Especially in the entry stage, there are too few things to do , Because companies like Tencent , supplier 、 There are too many partners , There's no way to manage , So we do more in the use phase and management and maintenance phase , Here's a share .

This is what we talked about before log4j2 The process of event response .

 log4j2 Event response process

The first stage is the acquisition of intelligence , We have established a monitoring mechanism for vulnerability intelligence . The second stage is the study and judgment of intelligence . Not all intelligence will receive a high standard response , We will study and judge the impact of the vulnerability, especially on Tencent , For example, for some components , Tencent doesn't use this technology stack , There's no need to respond . In the impact inventory stage , We do more , In several ways , The first is online WAF Protection rules for , Secondly, three ways are adopted , One is SCA, Identify which part of the code uses Log4j2; The other is through PoC The way , Test which businesses exposed to the Internet actually have vulnerabilities ; The third kind of , adopt HIDS To identify the host level log4j2 Of Java package . For some key products , We will do manual screening . On the overall convergence priority , Take the vulnerability of the Internet as the first priority , adopt SCA and HIDS The identified vulnerabilities come second . 

In terms of intelligence based risk inventory , We basically adopt such a mechanism ： From monitoring 、 analysis 、 Asset matching 、 Influence surface output to push repair . It also involves the capacity-building of asset mapping , Because once a vulnerability breaks out , It's actually a little late to explore again , Therefore, we need to do a good job in asset mapping in advance , This can quickly match the influence surface .

About open source governance , We mainly do more in component identification and security control , Some specifications will also be involved , For example, Tencent has a software source , We embedded security checks in it , Some fake packages or packages with vulnerabilities , Is not allowed to be downloaded . But we can't ask users to only use Tencent software source , So more is done in the intermediate stage . The overall component identification adopts three ways , One is SCA, One is the traditional network detection ,Banner distinguish , The last one is host detection . There are also three types of security governance roles ： Part time business security interface 、 Security team and QA. In addition, there is a risk disposal platform . Because according to hundreds of thousands of safety problems , Follow up with forms , It's tiring , So we will continue to iterate and optimize later , Launch a risk disposal platform .

If we summarize from the whole life cycle of open source software , There are probably the following dimensions . In the introduction phase , One is to check the software source accordingly , The business will take the initiative to conduct security assessment and introduce reporting , In addition, the open source component security management specification is also established . In the use and maintenance management stage , We use a blacklist , such as Struts2 It's on Tencent's blacklist , Not allowed to be used .

That's what I want to share , Thank you. ！

About hanging mirror safety

The suspension mirror is safe ,DevSecOps Agile security leaders . By the network security technology research team of Peking University “XMIRROR” Initiate the establishment , Committed to AI Technology enables agile security , Focus on DevSecOps Software supply chain continuous threat integrated detection and defense . The core DevSecOps Smart Adaptive Threat Management solutions include threat modeling with deep learning technology as the core 、 Open source governance 、 Risk discovery 、 Threat simulation 、 Independent innovative products with multiple dimensions such as detection response and software supply chain security services characterized by actual attack and defense confrontation , For Finance 、 energy 、 Pan Internet 、IoT、 Users in cloud services, automobile manufacturing and other industries provide innovative and flexible intelligent adaptive security housekeeper solutions . For more information, please visit the hanging mirror security official website ：www.xmirror.cn