当前位置：网站首页>Best practices of Apache APIs IX high availability configuration center based on tidb
Best practices of Apache APIs IX high availability configuration center based on tidb
2022-04-23 15:41:00 【ApacheAPISIX】
What is? Apache APISIX？
API Gateway is an important component in the microservice architecture , It is the core entrance and exit of traffic , It is used for unified processing of business-related requests , It can effectively solve massive requests 、 Malicious access and other issues , Ensure business security and stability .
As an open source cloud native API gateway ,Apache APISIX Both dynamic 、 real time 、 Three advantages of high performance , Provides load balancing 、 Dynamic upstream 、 Grayscale Publishing 、 Service failure 、 Authentication 、 Observability and other rich traffic management functions , Help enterprises quickly 、 Handle safely API And microservice traffic , It can be applied to gateway 、Kubernetes Ingress And service grid .
meanwhile ,Apache APISIX Rich community ecology has been established through extensive ecological cooperation .Apache APISIX It also supports highly customized , Support Wasm, You can use Java、Go、Python And other mainstream computer languages .
Apache APISIX Technology Architecture
Apache APISIX The data plane and control plane are separated , Receive... Through the configuration center 、 Lower allocation , So that the data plane is not affected by the control plane .
In this architecture , The data plane is responsible for receiving and processing caller requests , Use Lua And Nginx Dynamic control of request traffic , Can be used to manage API The entire life cycle of the request . The control plane contains Manager API And the default configuration center etcd, Can be used to manage API gateway . When the administrator accesses and operates the console , The console will call Manager API Distribute the configuration to etcd, With the help of etcd watch Mechanism , The configuration will take effect in real time in the gateway .
The default configuration center is etcd, Also support Consul、Nacos、Eureka etc. .etcd Natural support for distributed 、 High availability , Support clusters , And in K8s And other fields have a lot of application practice , bring APISIX It can easily support millisecond configuration updates 、 Support thousands of mesh joints , And the mesh joint points are stateless , It can be expanded or reduced at will .
etcd The limitations of
1. Their own architecture problems
First ,etcd be based on BoltDB, Capacity has an upper limit .etcd The default storage limit is 2 GB, If the upper limit exceeds 2 GB, You can use the
--quota-backend-bytes Tag configuration store , Maximum adjustable to 8 GB. One etcd If there is a cluster 8 GB The amount of storage , Is enough to serve a gateway , But if you also serve N individual APISIX colony , There may not be enough capacity , It may cause some trouble .
secondly ,etcd It's essentially a CP System , nothing Method bearing Big Amount of client connections . because etcd It's through Raft To achieve distributed consensus , All read and write requests will pass through Raft Of Leader To deal with , A large number of client connections may lead to high load of the whole cluster , It may affect the caller .
2. Scene matching problem
stay Ingress and Service Mesh Such as the scene , Use etcd Relatively speaking, it's a little too heavy , There are some use Users do not want to deploy except control Noodles And data Noodles Components other than . such as NGINX Ingress Controller Just a mirror can run , but APISIX Ingress Controller except Ingress Controller Control surface and APISIX Data side , One more etcd. For users , This technology architecture is more expensive to deploy , And make sure that etcd Operation and maintenance of .
And in essence ,etcd Is a redundant component , You can get rid of .K8s It supports storage services , All configuration information 、 Stored in APISIX Back end Endpoints Information can be obtained from K8s Of API Server obtain . Use... In this scenario etcd It will make the whole selection more bulky .
The same is true of service grid . In the service grid scenario APISIX, If you want to deploy etcd, The whole selection will focus on . And in the service grid scenario ,Pod The number may be hundreds or even tens of thousands , This is very common . If there are tens of thousands Pod All connected to etcd,etcd It will become the bottleneck of the whole service .
3. The question of cost
First of all ,etcd The operation and maintenance cost is high , Some companies don't have specialized etcd O & M Engineer . Deploy etcd Need at least 3 Or 5 An example ,etcd After successful operation , You also need to do data backup regularly , Create a snapshot . For monitoring etcd Operating condition , Real time understanding etcd Health status of , We also need to build an observability system , Provide necessary alarm support . If a company doesn't have a special etcd O & M Engineer , May not be able to do well etcd Operation and maintenance work .
second , Some companies or organizations have long-term middleware or infrastructure , Switching the configuration center will bring some costs . For these companies or organizations , They often prefer to reply ⽤ Existing middleware or infrastructure as APISIX The configuration of ⼼,⽐ Such as TiDB、Consul、Apache ZooKeeper, So as to converge the technology stack , Avoid additional costs .
Based on the above considerations , We decided to study a new scheme , Change the current overweight technical architecture , by Apache APISIX Provide more flexible choices for users , Don't be etcd binding , Ease the of existing users etcd Operation and maintenance pressure , Lower maintenance cost , At the same time, it is expected to give users more 、 Better choice , breakthrough etcd Our own bottleneck .
relieve APISIX and etcd The strong connection of , Let users have more 、 More flexible options , In fact, it is also the charm of open source . If this restriction can be lifted , There are no restrictions on how users use , Users may create more surprises .
The project design
How to decouple APISIX and etcd？
At the beginning of the scheme design , The first question we consider is how to achieve APISIX And etcd The decoupling , because APISIX The core of ⼼ Code 、 Data structure and etcd Closely related to . Responsible for operation configuration Admin API It usually takes... In the return value etcd Metadata , such as etcd v3 Of Revision、etcd v2 Of
modifiedIndex, Even in APISIX In the core logic of , A route or a Upstream Objects will also carry these metadata .
If we fundamentally transform APISIX, The cost will exceed ⾼. Transformation in such a core place may also affect APISIX Existing stability , So directly modify APISIX It may not be a good plan .
broken ： Introduce additional middle tier
If the direct transformation cost is too high and the risk is too high , So can we consider introducing an additional middle layer ？ There is a famous saying in the computer industry ——“ There is no problem that can't be solved by adding one layer ”. If you want to add a layer , What is the specific responsibility of this layer ？ What to do ？ Sum up , This layer needs to complete two more important things .
First of all , This additional middle layer needs to Provide etcd v3 API and ⽀ a etcd gRPC Gateway. at present ,APISIX Only support etcd v3. Yes APISIX Come on , This middle tier is still a etcd, It must provide etcd v3 Of API. In addition to providing v3 Of API, It also supports etcd Of gRPC Gateway, because APISIX Now it's still through HTTP The protocol and etcd Interaction , and etcd v3 API Is based on gRPC, We need to etcd Of gRPC Gateway hold HTTP Turn your request into gRPC request , So that the whole interaction can go on smoothly .
second , This extra middle layer can Connect various storage schemes . We need to figure out how to support TiDB、PostgreSQL,SQLite, Even Consul、Apachce ZooKeeper These different schemes .
Only by doing these two things , This middle tier can connect different storage schemes , So as to give APISIX Bring complete configuration center functions .
Standing on the shoulders of giants TiDB
With this middle layer , How do we integrate TiDB Well ？ In fact, we have a similar project to refer to . although K8s Native supports the use of etcd As a storage solution , but Rancher Of K3s project It doesn't work etcd, Maybe it's because if K3s Deployed in some embedded environments ,etcd Some of the limitations make it impossible to operate well . therefore ,Rancher adopt Kine This project , Some additional components are supported , such as PostgreSQL、MySQL、SQLite、 Dqlite, bring K3s Users can choose other storage schemes flexibly . In a nutshell ,Kine This project has the following points that we can learn from .
First of all ,TiDB compatible MySQL, and Kine The project itself supports MySQL. We can learn from or refer to Kine Some of the implementations , So as to help us better support and connect this project TiDB.
second , Kine Complete realization of etcd Need support watch function . because APISIX It is based on push mode to sense the change of configuration , The delay of configuration change is usually on the millisecond level , The delay is very low . and watch The function just involves the push of configuration , therefore watch The mechanism is quite important .
Third ,Kine Also simulated etcd Of MVCC characteristic , Support Compact. Every change 、 write in 、 Update or delete in Kine or TiDB There is a line of data in the . The primary key of each row of data is etcd Of Revision, That's the counter , Record the number of latest changes . In this way ,Kine Multi version support is realized .
By introducing a similar architecture ,Apache APISIX No need to interact with a real storage center , But interact with this middle tier . Pictured above ,APISIX and etcd adapter The middle floor will go etcd Of KV API and Watch API,etcd adapter Will poll TiDB, Write aware configuration , complete watch operation , To push the data to APISIX.
The effect of the scheme
etcd adapter The birth of
With these thoughts and Kine The reference of this project , We developed it on the shoulders of giants etcd adapter project .
First , This project Support TiDB、MySQL as well as In-Memory B-Tree And other configuration centers , soon , Will also support SQLite and PostgreSQL. among ,In-Memory B Trees and APISIX Ingress Controller The architecture is too heavy . If you choose In-Memory Of B Tree selection , Users can directly put etcd adapter Embedded in the target program . One component is missing in this way , It can further improve the overall user experience .
secondly , This project Support etcd v3 Of API. at present , This project only supports APISIX The required API A subset of , such as KV API and Watch API. As for other types of API, such as Lease、 Partial authentication API Not yet fully implemented .
Last , This project **⽀ Hold on gRPC Gateway.** It will put the corresponding gRPC The interface is translated into the corresponding Restful Interface , for APISIX call .
Although we put etcd adapter On the control surface , But we can also put it in every APISIX On the edge , Exist as a side car . Both schemes have their own advantages , You can choose flexibly according to your actual situation .
About the follow-up plan and future direction of this project , We have the following ideas to share with you .
by Apache APISIX Of ⽤ Users provide more configurations ⼼ choice
We hope etcd adapter The project can make Apache APISIX Our users have more choices of configuration centers , Don't be etcd lock , Users can choose solutions according to their actual situation . If the operation and maintenance of the company 、 The developed technology stack is more inclined to Consul, You can use Consul.Consul KV Is based on Raft, High availability . in addition to , You can also consider more mainstream Apollo Or and etcd Benchmarking Apache ZooKeeper, also PostgreSQL Or other alternatives .
by Apache APISIX Ingress Controller Improved architecture helps ⼒
We hope etcd adapter The project can be APISIX Ingress Controller Improved architecture helps .etcd adapter Support In-Memory B-Tree,In-Memory B-Tree You can embed data into memory , Without actually storing .
In this way ,etcd adapter Can be a APISIX Ingress Controller Part of ,Apache Ingress Controller Just keep Ingress Controller Control surface and APISIX The data plane has two components . Because there is no etcd,APISIX Even with Ingress Controller Direct interaction , Get configuration change data .
in addition to , We can also put Ingress Controller Control surface and APISIX The data plane is placed in the same image , Achieve control ⾯ And data ⾯ Of ⼀ Integrated deployment . In the end, only one command is needed 、 A mirror image , You can go to K8s Put... In the target cluster APISIX Ingress Controller Run . If the control surface 、 Put the data faces together , You don't have to deploy another etcd And a control surface , Equivalent to directly missing two components , Can greatly improve the user experience .
Donate to Apache The foundation , As Apache APISIX Of ⼦ term ⽬ Into the ⾏ incubation
at present , The address of this project is https://github.com/api7/etcd-adapter, Put it in API7.ai In the warehouse . future , We expect to continue to polish this project , After the iteration of the project is relatively perfect , Will donate it to Apache Software foundation , As Apache APISIX The sub projects of , So as to attract more people with lofty ideals in the community , Work with us to improve this project , Give Way Apache APISIX Our ecology is even larger .
Configuration of multi spanning tree MSTP
Why disable foreign key constraints
Mobile finance (for personal use)
大厂技术实现 | 行业解决方案系列教程
pgpool-II 4.3 中文手册 - 入门教程
Why is IP direct connection prohibited in large-scale Internet
cadence SPB17. 4 - Active Class and Subclass
IronPDF for . NET 2022.4.5455
Node. JS ODBC connection PostgreSQL
Today's sleep quality record 76 points
一刷313-剑指 Offer 06. 从尾到头打印链表（e）
GFS distributed file system (Theory)
[leetcode daily question] install fence
How to test mobile app?
ICE -- 源码分析
Deep learning - Super parameter setting