API gateway / API gateway (III) - use of Kong - current limiting rate limiting (redis)

One 、 Preface

Kong There are three ways to limit current , They are local current limiting （Local）、 Database throttling （Cluster） and Redis Current limiting , The current limiting algorithms adopted by these three current limiting methods are counter method . Support by second / branch / Hours / Japan / month / Current limiting in different time units such as year , And can be combined , For example, you can set a maximum of... Per second 100 Times and at most per minute 1000 Time .

• Local current limiting （Local）, The counter uses Nginx The cache of
• Database throttling （Cluster）, The counter is saved in the database ,Kong Two types of databases are supported ,PostgreSQL and Cassandra
• Redis Current limiting , Similar to local current limiting , Just put the cache in Redis On

Next configure Kong API gateway , Use Redis The way of limiting current . 

Two 、Redis To configure

2.1  stay Docker-composer.yml add Redis Configuration of

  redis:
    image: redis
    container_name: redis
    restart: always    
    networks:
      - kong-net    
    ports:
      - 6379:6379
    volumes:
      - ./redis/data:/data
    command:
      redis-server /data/redis.conf --requirepass "123321"

2.2 redis.conf The configuration file

redis-server Start with profile parameters , Such as redis-server redis.conf, need redis.conf File exists in advance , hold redis.conf The file is placed on the host ./redis/data Under the table of contents , Click to download redis.conf 

2.3 Docker-compose.yml

version: "3"
 
networks:
  kong-net:
    #external: true #docker network create --subnet=172.18.0.0/16 kong-net
    driver: bridge
  
#volumes:
#  db-data:
#    external:
#      name: ~/db   #manually using `docker volume create --name=~/db`
  
services:
  redis:
    image: redis
    container_name: redis
    restart: always    
    networks:
      - kong-net    
    ports:
      - 6379:6379
    volumes:
      - ./redis/data:/data
    command:
      redis-server /data/redis.conf --requirepass "123321"
  kong-database:
    image: postgres:9.6
    restart: always
    networks:
      - kong-net 
    environment:
      POSTGRES_USER: kong
      POSTGRES_DB: kong
      POSTGRES_PASSWORD: your_pg_password
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "kong"]
      interval: 5s
      timeout: 5s
      retries: 5
    volumes:
      - ./postgres/data:/var/lib/postgresql/data
    container_name: kong-database
  kong-migration:
    image: kong:latest
    command: "kong migrations bootstrap"
    restart: on-failure
    networks:
      - kong-net 
    environment:
      KONG_DATABASE: postgres
      KONG_PG_HOST: kong-database
      KONG_PG_PASSWORD: your_pg_password
      KONG_CASSANDRA_CONTACT_POINTS: kong-database
    links:
      - kong-database
    depends_on:
      - kong-database
    container_name: kong-migration     
  kong:
    image: kong:latest
    restart: always
    networks:
      - kong-net 
    environment:
      KONG_DATABASE: postgres
      KONG_PG_HOST: kong-database
      KONG_PG_PASSWORD: your_pg_password
      KONG_CASSANDRA_CONTACT_POINTS: kong-database
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: 0.0.0.0:8001, 0.0.0.0:8444 ssl
    depends_on:
      - kong-migration
      - kong-database
    healthcheck:
      test: ["CMD", "curl", "-f", "http://kong:8001"]
      interval: 5s
      timeout: 2s
      retries: 15
    ports:
      - "8001:8001"
      - "8000:8000"
      - "8443:8443"
      - "8444:8444"
    container_name: kong
  konga-prepare:
    image: pantsel/konga:next
    command: "-c prepare -a postgres -u postgresql://kong:[email protected]:5432/konga_db"
    restart: on-failure
    networks:
      - kong-net 
    links:
      - kong-database
    depends_on:
      - kong-database
    container_name: konga-prepare
  konga:
    image: pantsel/konga:latest
    restart: always
    networks:
      - kong-net 
    environment:
      DB_ADAPTER: postgres
      DB_HOST: kong-database
      DB_USER: kong
      DB_PASSWORD: your_pg_password
      TOKEN_SECRET: km1GUr4RkcQD7DewhJPNXrCuZwcKmqjb
      DB_DATABASE: konga_db
      NODE_ENV: production
    depends_on:
      - kong-database
      - konga-prepare
    ports:
      - "1337:1337"  
    container_name: konga

2.4  test Redis

test Redis Whether the installation and data persistence of are normal , stay RedisDesktopManager In the middle, connect with Redis after , Open console , You can enter a command to change Redis Configuration information .

use “CONFIG SET” After modifying the configuration , If you restart Redis Words , The changed parameters will not be saved , Will revert to the original parameters in the original configuration file .

If you want to “CONFIG SET” The modified parameters can be saved all the time , You need to execute the command again “CONFIG REWRITE”, The configuration file will be updated Redis.conf, Then restart Redis Words , The parameter will also be the latest modified parameter .

2.5 Redis The persistence of

If you don't want Redis Data loss after restart , Need to use Redis Persistence configuration ,Redis The persistence of is 2 Ways of planting ： RDB and AOF.

2.5.1 RDB Persistence mode

RDB Under way ,Redis Will take a snapshot of the dataset dump To dump.rdb In file , We can also modify it through the configuration file Redis The server dump Frequency of snapshots .

• save 900 1              # stay 900 second (15 minute ) after , If at least 1 individual key change , be dump memory dump .

• save 300 10            # stay 300 second (5 minute ) after , If at least 10 individual key change , be dump memory dump .

• save 60 10000        # stay 60 second (1 minute ) after , If at least 10000 individual key change , be dump memory dump .

2.5.1.1 SAVE and BGSAVE command

 SAVE It's blocking RDB Persistence , When you execute this command Redis The main process of writes the data in memory to RDB file （ namely dump.rdb） in , Until the file is created Redis Will not be able to process any command requests .

  BGSAVE It's non blocking persistence , It will create a sub process to write the data in memory to RDB In file , At the same time, the main process can also handle the command request from the client . But the child process is basically a copied parent process , This is equal to two of the same size Redis Processes run on the system , Will cause a substantial increase in memory usage .

After execution bgsave after ,dump.rdb It will be the latest modification time .

（ Case study ：Redis Own memory usage 60%, Total memory usage 70-80%, When lasting, it immediately soared to more than 130% ,  Finally, I chose AOF Persistence ）

2.5.2 AOF Persistence mode

AOF Persistent configuration

appendonly yes                       # Turn on AOF Persistence , Off by default 
appendfilename "appendonly.aof"      #AOF File name （ Default ）
appendfsync no                       #AOF Persistence strategy 
auto-aof-rewrite-percentage 100      # Trigger AOF Conditions for file rewriting （ Default ）
auto-aof-rewrite-min-size 64mb       # Trigger AOF Conditions for file rewriting （ Default 

appendfsync There are three options ：always、everysec and no, They are ：

• appendfsync always     # Write every time a data change occurs AOF file .
• appendfsync everysec  # Sync every second , The strategy is AOF Default policy for .
• appendfsync no           # The system itself decides .

1、 choice always When the server executes an event, it will AOF The contents of the buffer are forced to be written to the hard disk AOF In the document , It can be seen as every time you perform redis Write the command to AOF This command is recorded in the file , This ensures the integrity of data persistence , But efficiency is the slowest , But it's also the safest ;

2、 configure everysec If the server performs a write operation every time （ Such as set、sadd、rpush） This command will also be appended to a separate AOF End of buffer , And will AOF Buffer write AOF file , Then, file synchronization will be carried out every other second to synchronize the files in the memory buffer AOF Cache data is actually written to AOF In the document , This model takes into account both efficiency and data integrity , Even if the server goes down, only one second will be lost redis Changes made to the database ;

3、 take appendfsync configure no Then means redis Even if the data in the database is lost, you can accept , It also appends each write command to AOF End of buffer , Then write the file , But when to synchronize files and really write data AOF In the file, it is up to the system itself , That is, when the space of the memory buffer is filled or exceeds the set time limit, the system will automatically synchronize . In this mode, the efficiency is the fastest , But it's also the least secure for data , If redis The data in the database is from the background database, such as mysql Out of , Belongs to data that can be retrieved at any time or is not important , Then you can consider setting it to this mode .

      comparison RDB Each persistence doubles the memory ,AOF In addition to persistence, a new child process will be created when it is first enabled AOF Files consume a lot of memory , After each persistence, the memory usage is very small . but AOF There is also a problem that cannot be ignored ：AOF File is too large. . You are right about redis Every write operation of the database will make AOF Add a piece of data to the file , Over time, this document will form a behemoth . Fortunately redis Put forward AOF The mechanism of rewriting , I.e. the one configured above auto-aof-rewrite-percentage and auto-aof-rewrite-min-size.auto-aof-rewrite-percentage and auto-aof-rewrite-min-size Configure trigger AOF The conditions for rewriting .Redis After the last rewrite AOF File size of the file , And the current AOF The file size is the same as that after last rewriting AOF The percentage of file size exceeds auto-aof-rewrite-percentage Set the value of the , At the same time, at present AOF The file size also exceeds auto-aof-rewrite-min-size Set the minimum value , It triggers AOF File rewriting . Take the above configuration as an example , When the present AOF File is larger than 64mb Also greater than the last rewrite AOF File size after , Then the file will be AOF rewrite .

Redis4.0 Start allowing the use of RDB and AOF Mixed persistence , Combining the advantages of the two, through aof-use-rdb-preamble The configuration item can turn on the mixing switch . The last thing to note is this , If Redis Open the AOF Persistence function , So when Redis It will be used first when rebooting AOF File to restore the database .

3、 ... and 、Kong To configure

3.1 add to Rate Limiting plug-in unit

 3.2 Set up minute = 5（ namely 1 No more than... In minutes 5 Time ）

 3.3 To configure policy=redis and redis Connection information

3.4 To configure hide client headers = yes

If hide client headers = no, HTTP Of header It's going to be like this

Date →Tue, 28 Apr 2020 16:14:18 GMT
Content-Type →application/json; charset=utf-8
Connection →keep-alive
Retry-After →42
Content-Length →37
X-RateLimit-Limit-Second →5
X-RateLimit-Remaining-Second →5
X-RateLimit-Remaining-Minute →0
RateLimit-Limit →5
RateLimit-Remaining →0
X-RateLimit-Limit-Minute →5
RateLimit-Reset →42
X-Kong-Response-Latency →1
Server →kong/2.0.3

Content-Type →text/plain;charset=UTF-8
Content-Length →4
Connection →keep-alive
Date →Tue, 28 Apr 2020 16:26:28 GMT
X-Kong-Upstream-Latency →8
X-Kong-Proxy-Latency →10
Via →kong/2.0.3

3.5 call API To test

stay 1 Within minutes 5 Secondary call ,HTTP The return code is 429 Too Many Requests