当前位置：网站首页>The art of automation

The art of automation

2022-04-23 13:50:00 【InfoQ】

For infrastructure , Many services have similar architectures 、 Database interaction and UI, This paper introduces Paypal Through the practice of template automation infrastructure micro service scaffold code , Maximize the time to develop services and ensure the consistency of services . original text ：The Art of Automating Automation
[1]

brief introduction

As PayPal Website reliability and cloud engineering (SRCE, Site Reliability & Cloud Engineering) Part of the team , We are committed to improving efficiency through automation . However , Writing software that allows people to automate their work intuitively is not a simple task . stay PayPal In infrastructure , The automation components of any operation must provide the highest level of reliability 、 Security 、 Efficiency and self-service ability . In this article, we will discuss how to serve in the global network (GNS, Global Network Services) To achieve automation in .

GNS The team has a lot of operations that need to be automated , It ranges from ordinary tasks that require a few hours of human investment , To complex tasks that require the operation and maintenance team to devote many days of energy . For us , The biggest challenge is how to automate these large and small operations without consuming a lot of development cycles , We need to work hard to prove “ROI( Return on investment )” The rationality of , This is particularly important .

Software service requirements

To turn any network function into a service , Need to complete the following series of tasks :

User interface development

Open to PayPal All services in the development community must have some form of user interface (UI), So that users can use the network like any other service .

Database interaction and table management

Most services need to design database tables and write database interaction code to manipulate data .

Core business logic

You need to write and test the actual business logic for a particular service .

And automation tools Terraform Integrate

Our basic users include the network operation team , They won't use based on UI Service for . contrary , They are based on GitOps Mode operation , This means that all services we develop need to be developed Terraform provider And Terraform Integrate .

Certification and authorization

This involves standard enterprise authentication ( Such as single sign on 、 Multifactor Authentication 、 Role management, etc ) Integration of .

Service account ,API Consumer throttling and speed limits

We decided to provide... For everything API. therefore , All services we develop must provide service accounts , And ensure throttling and speed limit .

As you can see from the list above , Building a viable network service requires a lot of time to develop important peripheral components . In an ideal situation , We can only spend time writing core business logic , And don't do anything else . In two ways , We have reached the ideal state of writing less code .

The first way to solve this problem is to migrate to real micro service components and build pluggable Library Architecture , The second method is template based automatic code generation , Can be generated based on a generic template UI、 Back end and database layer code .

Code generation

For a long time , Data modeling has always been the pillar of well-designed software products . Once the basic data building blocks are identified , It's easier to build software components that work around data . If the data is obtained in a standard format , Then there is a lot of automation space for various software components around data .

This is exactly the method we use . All of our team's projects start with identifying the data model . The data model uses YAML Format defined , In this way, our dynamic code generation module can generate code for all parts of the system . The following figure shows the process :

GNS Software architecture

UI Code generation

React The framework has been included PayPal Widely used in the industry including .PayPal The application team has a very modular React library , For building customer facing applications . We build on this library , Use in any suitable place .

Most system programmers are not familiar with UI Programming . In order to promote rapid UI Component development , We rely on Model Driven Design . Use model driven design , Typical GNS Developers only need to specify UI Layout , Don't write anything UI Code .

GNS The code generation engine adopts a user-defined model , And use React frame , Generate a separate React Code container , With fully functional UI. This approach allows developers to focus on the business logic of the application .

Back end code generation

In the past , Development and supply REST API Microservices with functions are very cumbersome , Developers need to write software for each endpoint and configure it web The service gateway . However ,Flask The framework has been adopted as a standard framework Python Widely recognized by developers . Allied ,Golang In addition to built-in support , also gorilla/mux frame .

majority GNS Service is to use Golang or Python Compiling . With the help of the above framework , Designed based on Jinja Templates generate generic code GNS Code generation engine .

Based on YAML Format defined data model , The code generation engine uses predefined Jinja Template to generate the basic code of the service .GNS Developers can implement precise business logic based on these codes , To reduce programming errors , And eliminate the tedious work of writing everything from scratch .

Database layer code generation

majority GNS Services need some form of database to store the information described in the modeling phase . Write code to add 、 Deleting and modifying entries in the database is cumbersome , And it's prone to bug、 Inconsistent style, etc .Python and Golang Have the ability to handle object relational mapping (ORM, Object Relationship Mapper ) Open source library , We choose to adopt sqlalchemy To standardize database operations .

Because the model is made of YAML Format defined , Our code generation engine not only generates back-end code , Also generate database processing methods .

Terraform Provider Code generation

Terraform It has become the most commonly used tool for automation infrastructure . As GNS Service provider , We have decided that all automation services will be targeted at Terraform The interface of , Thus, it can adapt to the continuous deployment pipeline being adopted .

Terraform The architecture is flexible and easy to expand , Be able to add... To any automation we like Provider. see also terrform Documentation for more details .

Terraform Provider framework

because Terraform Provider Is a well constructed library , Our code generation engine can be based on Jinja Template generation is based on golang Of Provider Program , Then build the code into a pluggable module , The module can be installed into the running terraform In the example . Once that is done , The user can be in GNS Carry out standard service terraform operation , Like any other service .

Conclusion

Most of the tasks required to automate any web service have been generated by our code generation engine , This allows us to focus only on the core logic of the service . The shift to microservice architecture and the adoption of template based code generation and model driven architecture enable us to produce services faster . At the time of writing , We can complete the working framework of any service in one day , This helps the team focus on the core logic , And generate more services in a shorter time .

References:
[1] The Art of Automating Automation: https://medium.com/paypal-tech/the-art-of-automating-automation-17b32594a41f

Hello , I'm Yu fan , stay Motorola Have done research and development , Now in Mavenir Do technical work , Pair communication 、 The Internet 、 Back end architecture 、 Cloud native 、DevOps、CICD、 Blockchain 、AI And other technologies have always maintained a strong interest , I usually like reading 、 reflection , Believe in continuous learning 、 Lifelong growth , Welcome to exchange and study together .
WeChat official account ：DeepNoMind

- END -

版权声明
本文为[InfoQ]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/04/202204231334474751.html