当前位置:网站首页>Explanation 2 of redis database (redis high availability, persistence and performance management)

Explanation 2 of redis database (redis high availability, persistence and performance management)

2022-04-23 15:25:00 C chord~

Catalog

One .redis High availability

Two 、Redis Persistence

2.1 Persistent function

2.2 Persistence mode  

RDB and AOF Comparison of the two :

3、 ... and 、RDB Persistence

3.1 The trigger condition

Manual trigger

Automatic triggering

Other automatic trigger mechanisms

3.2 Execution process

3.3 Load on startup

Four .AOF Persistence

4.1 AOF Persistence

4.2 Turn on AOF 

4.3 Execute the process

①、 Order to append (append)

②、 File is written to (write) Synchronize with files (sync)

③、 File rewriting (rewrite)

④、 The process of file rewriting is as follows

⑤、 Load on startup

5、 ... and 、RDB and AOF Advantages and disadvantages

1、RDB Persistence

2、AOF Persistence

6、 ... and 、Redis Performance management

1、 see Redis Memory usage

2. Memory fragmentation rate

3、 Memory usage

4、 Internal recovery key

summary

RDB and AOF Comparison of the two :


One .redis High availability

  • Persistence : Persistence is the simplest high availability method ( Sometimes it's not even classified as a highly available means ), The main function is data backup , Store data on hard disk , Ensure that data is not lost due to process exit .
  •   Master slave copy : Master slave replication is highly available Redis The basis of , Sentinels and clusters are highly available based on master-slave replication . Master-slave replication mainly realizes multi machine backup of data , And load balancing and simple fault recovery for read operations . defects : Failure recovery cannot be automated ; Write operations are not load balanced ; Storage capacity is limited by a single machine .
  •   sentry : On the basis of master-slave replication , Sentinels achieve automated recovery from malfunctions . defects : Write operations are not load balanced ; Storage capacity is limited by a single machine .
  •   colony : By clustering ,Redis It solves the problem that write operation cannot be load balanced , And storage capacity is limited by single machine , A relatively complete high availability scheme has been realized .

Two 、Redis Persistence

2.1 Persistent function

  • Redis Is a memory database , Data is stored in memory , In order to avoid server power failure and other reasons Redis Permanent loss of data after abnormal exit of the process , Need to regularly Redis In some form ( Data or commands ) Save from memory to hard disk ; The next time Redis Restart time , Using persistent files to achieve data recovery .
  •   besides , For disaster backup , You can copy persistent files to a remote location

2.2 Persistence mode  

  • RDB   Persistence : The principle is that Reids The database records in memory are regularly saved to disk .( Default on )
  • AOF  Persistence : The principle is that Reids The operation log of is written to the file as an append , Be similar to MySQL Of binlog.

RDB and AOF Comparison of the two :

  • RDB Pay attention to the results 、 Small volume 、 Low security 、 The priority is higher
  • AOF In addition to results, focus on process 、 Large size 、 High safety 、 High priority

3、 ... and 、RDB Persistence

  • RDB Persistence refers to saving a snapshot of the data in the current process in memory to the hard disk within a specified time interval ( So it's also called snapshot persistence ), Compress storage with binary , The saved file suffix is rdb; When Redis On reboot , Can read snapshot file recovery data .

3.1 The trigger condition

Manual trigger

  •  save Command and bgsave Commands can be generated RDB file .
  • save Orders will block Redis Server process , until RDB Until the file is created , stay Redis During server blocking , The server cannot process any command requests .
  • bgsave The command creates a child process , The child process is responsible for creating RDB file , The parent process ( namely Redis The main process ) Then continue processing the request .
  •  bgsave During command execution , Only fork Child process will block the server , And for save command , The whole process blocks the server , therefore save It's basically abandoned , Online environment should be eliminated save Use .
     

Automatic triggering

  • In auto trigger RDB Persistence ,Redis Will also choose bgsave instead of save To persist .
  •   The most common case of automatic triggering is in the configuration file through save m n, Designated as m Within seconds n The next change , Will trigger bgsave.
vim /etc/redis/6379.conf
#----219 That's ok ---- Here are three save When any condition is satisfied , Will cause bgsave Call to 
save 900 1 : When the time comes 900 seconds , If redis The data happened, at least 1 Changes , execute bgsave
save 300 10 : When the time comes 300 seconds , If redis The data happened, at least 10 Changes , execute bgsave
save 60 10000 : When the time comes 60 seconds , If redis The data happened, at least 10000 Changes , execute bgsave
#----242 That's ok ---- Open or not RDB File compression 
rdbcompression yes
#----254 That's ok ---- Appoint RDB file name 
dbfilename dump.rdb
#----264 That's ok ---- Appoint RDB Document and AOF File directory 
dir /var/lib/redis/6379

Other automatic trigger mechanisms

  except save m n outside , There are other situations that trigger bgsave:

  • In the master-slave replication scenario , If full replication is performed from a node , Then the master node will execute bgsave command , And will rdb The file is sent to the slave node .
  • perform shutdown On command , Automatic execution rdb Persistence .

3.2 Execution process

  1.  Redis The parent process first judges : Is it currently being implemented save, or bgsave/bgrewriteaof Can be inherited by child processes. , If it's being executed bgsave The command returns directly to . bgsave/bgrewriteaof Cannot be executed at the same time , Mainly based on performance considerations : Two concurrent subprocesses perform a large number of disk writes at the same time , Can cause serious performance problems .
  2. Parent process execution fork Action create subprocess , In this process, the parent process is blocked ,Redis Cannot execute any command from the client
  3. The parent process fork after ,bgsave Command return ”Background saving started” Information doesn't block the parent process anymore , And can respond to other commands
  4. Subprocess creation RDB file , Generate a temporary snapshot file according to the memory snapshot of the parent process , After the completion of the original file for atomic replacement
  5. The child process sends a signal to the parent to indicate completion , Parent process update statistics
     

3.3 Load on startup

  •  RDB File loading is performed automatically when the server starts , There is no special order . But because of AOF Higher priority , So when AOF On ,Redis Will load first AOF File to recover data ; Only when AOF closed , Will be in Redis Detect when the server starts RDB file , And automatically load . The server loads RDB Blocked during file , Until the load is complete .
  •  Redis(AOF When it's closed ) load RDB When you file , Would be right RDB Check the file , If the file is corrupted , Error will be printed in the log ,Redis Boot failure
     

Four .AOF Persistence

4.1 AOF Persistence

  • RDB Persistence is writing process data to a file , and AOF Persistence , Will be Redis Each write performed 、 The delete command is recorded in a separate log file , The query operation will not record ; When Redis Execute again on restart AOF Command in the file to recover data .
  • And RDB comparison ,AOF Better real-time , So it has become a mainstream persistence solution

4.2 Turn on AOF 

Redis The server is turned on by default RDB, close AOF; To turn on AOF, You need to configure... In the configuration file :

vim /etc/redis/6379.conf

#----700 That's ok ---- modify ; Turn on AOF
appendonly yes
#----704 That's ok ---- Appoint AOF File name 
appendfilename "appendonly.aof"
#----796 That's ok ---- Whether to ignore the last instruction that may have problems 
aof-load-truncated yes
# finger redis On recovery , The last instruction that may have a problem is ignored , The default is yes, That is to say aof When writing , There may be a problem with incorrect instructions ( Sudden power failure leads to non execution end ), In this case ,yes Meeting log And continue , and no Will directly recover from failure 

/etc/init.d/redis_6379 restart
# You need to cancel the password first 

4.3 Execute the process

Because of the need to record Redis Every command written by , therefore AOF No need to trigger , Let's introduce AOF The implementation process of .
 AOF The implementation process includes :

  • Order to append (append): take Redis The write command of is appended to the buffer aof_buf
  • File is written to (write) Synchronize with files (sync): According to different synchronization strategies, we will aof_buf The contents of the file are synchronized to the hard disk
  • File rewriting (rewrite): Rewrite regularly AOF file , Achieve the purpose of compression
     

①、 Order to append (append)


 Redis First append the write command to the buffer , Instead of writing directly to a file , It is mainly to avoid writing commands to the hard disk directly every time , Cause the hard disk IO Become Redis The bottleneck of the load .
  The format of the command append is Redis The protocol format of the command request , It's a plain text format , Good compatibility 、 High readability 、 Easy to handle 、 It is easy to operate and avoid secondary overhead . stay AOF In file , Except for the specified database select command ( Such as select 0 For the selection 0 The database ) By Redis Added , The others are written commands sent by the client .


②、 File is written to (write) Synchronize with files (sync)


 Redis Provides a variety of AOF Cache synchronization strategy , The policy involves the operating system write Functions and fsync function , The explanation is as follows :
In order to improve the efficiency of file writing , In modern operating systems , When the user calls write Function to write data to a file , Operating systems usually store data in a memory buffer , When the buffer is filled or exceeds the specified time limit , To write the buffer data to the hard disk . This kind of operation improves the efficiency , But it also brings security issues : If the computer goes down , Data in the memory buffer will be lost ; So the system also provides fsync、fdatasync And so on , It can force the operating system to write the data in the buffer to the hard disk immediately , So as to ensure the security of data .
 AOF There are three synchronization methods for the cache synchronization file policy , They are :(vim /etc/redis/6379.conf ----》 729 That's ok )
appendfsync always: Command write aof_buf Call the system immediately after fsync The operation is synchronized to AOF file ,fsync After completion, the thread returns . In this case , Every time you have a write command, you have to synchronize to AOF file , Hard disk IO Become a performance bottleneck ,Redis Only about a few hundred TPS write in , It's seriously reduced Redis Performance of ; Even with solid state drives (SSD), It can only process tens of thousands of commands per second , And it will greatly reduce SSD Life span of .
appendfsync no: Command write aof_buf After calling the system write operation , incorrect AOF File do fsync Sync ; Synchronization is the responsibility of the operating system , Usually the synchronization period is 30 second . In this case , The timing of file synchronization is uncontrollable , And there will be a lot of data piled up in the buffer , Data security cannot be guaranteed .
appendfsync everysec: Command write aof_buf After calling the system write operation ,write After completion, the thread returns ;fsync The synchronization file operation is called once per second by a dedicated thread .everysec It's a compromise between the two strategies , It's a balance between performance and data security , So it is Redis Default configuration , It's also our recommended configuration .


③、 File rewriting (rewrite)

  •   Over time ,Redis More and more write commands are executed by the server ,AOF The files will get bigger and bigger ; Too much AOF Files will not only affect the normal operation of the server , It can also cause data recovery to take too long .
  •   File rewriting refers to periodic rewriting AOF file , Reduce AOF Volume of file .

               AOF Rewriting is to put Redis In process data is converted to write commands , Sync to the new AOF file
                Not to the old AOF File for any read 、 Write operation

  •     about AOF In terms of persistence , File rewriting is highly recommended , But it's not necessary ; Even if there is no file rewriting , Data can also be persisted and stored in Redis Import at startup ; So in some implementations , Automatic file rewriting is turned off , And then through the timing task, at a certain time of the day .
 The reason why file rewriting can be compressed AOF file , The reason lies in :
	 Expired data is no longer written to the file 
	 Invalid commands are no longer written to the file : If some data is set repeatedly (set mykey v1, set mykey v2)、 Some data has been deleted (sadd myset v1, del myset) etc. .
	 Multiple commands can be combined into one : Such as sadd myset v1, sadd myset v2, sadd myset v3 Can be combined into sadd myset v1 v2 v3.

Trigger of file rewriting , It can be divided into manual trigger and automatic trigger :

  • Manual trigger : Call directly bgrewriteaof command , The execution of the order is related to bgsave Some similar : All are fork The subprocess does the specific work , And only in fork Time blocking .
  • Automatic triggering : By setting auto-aof-rewrite-min-size Options and auto-aof-rewrite-percentage Option to automate BGREWRITEAOF. Only when auto-aof-rewrite-min-size and auto-aof-rewrite-percentage When both options are met , Will automatically trigger AOF rewrite , namely bgrewriteaof operation .
  • auto-aof-rewrite-percentage 100 : At present AOF file size ( namely aof_current_size) Is the last time the log was rewritten AOF file size (aof_base_size) At twice , happen BGREWRITEAOF operation
  • auto-aof-rewrite-min-size 64mb : At present AOF File execution BGREWRITEAOF The minimum value of the command , Avoid starting Reids Due to the small file size, frequent BGREWRITEAOF
    vim /etc/redis/6379.conf
    #----729 That's ok ----
    auto-aof-rewrite-percentage 100
    # At present AOF file size ( namely aof_current_size) Is the last time the log was rewritten AOF file size (aof_base_size) Twice as much as , happen bgrewriteaof operation 
    auto-aof-rewrite-min-size 64mb 
    # At present AOF File execution bgrewriteaof The minimum value of the command , Avoid starting redis Due to the small file size, frequent bgrewriteaof
    
     Be careful :
    	 Rewriting by the parent process fork The subprocess goes on 
    	 During rewrite Redis Write commands executed , Need to be added to the new AOF In file , So Redis Introduced aof_rewrite_buf cache .
    

    ④、 The process of file rewriting is as follows

  • Redis The parent process first determines whether there is currently executing bgsave/bgrewriteaof Can be inherited by child processes. , If there is one bgrewriteaof The command returns directly to , If there is bgsave Orders wait bgsave Execute after the execution is completed .
  • Parent process execution fork Action create subprocess , In this process, the parent process is blocked .
  • The parent process fork after ,bgrewriteaof Command return ”Background append only file rewrite started” Information doesn't block the parent process anymore , And can respond to other commands .Redis All write commands for are still written to AOF buffer , And according to appendfsync Policy synced to hard disk , Keep the original AOF The right mechanism .
  • because fork The operation uses copy on write technology , Child processes can only share fork Memory data during operation . Because the parent process is still responding to commands , therefore Redis Use AOF Rewrite buffer (aof_rewrite_buf) Save this data , Prevent new AOF This data is lost during file generation . in other words ,bgrewriteaof During execution ,Redis At the same time, the write command of aof_buf and aof_rewirte_buf Two buffers .
  • Sub process according to memory snapshot , Write to the new... According to the command merge rule AOF file .
  • The subprocess writes a new AOF After the document , Signal the parent process , Parent process update statistics , The details can be obtained through info persistence see .
  • The parent process put AOF The data in the rewrite buffer is written to the new AOF file , This guarantees new AOF The database state saved by the file is consistent with the current state of the server .
  • Use the new AOF File replace old file , complete AOF rewrite .


⑤、 Load on startup

  •   When AOF On ,Redis When it starts, it takes precedence to load AOF File to recover data ; Only when AOF closed , Will load RDB File recovery data .
  •   When AOF Turn on , but AOF When the file does not exist , Even if RDB The file will not load if it exists .
  •  Redis load AOF When you file , Would be right AOF Check the file , If the file is corrupted , Error will be printed in the log ,Redis Boot failure . But if it is AOF The end of the file is incomplete ( The end of the file is not complete when the machine goes down suddenly ), And aof-load-truncated Parameter on , A warning will be output in the log ,Redis Ignore AOF End of file , Successful launch .aof-load-truncated The parameter is on by default .
     

5、 ... and 、RDB and AOF Advantages and disadvantages


1、RDB Persistence

  •   advantage :RDB The files are compact , Small volume , Network transmission is fast , Suitable for full replication ; Recovery speed ratio AOF Much faster . Of course , And AOF comparison ,RDB One of the most important advantages is that the impact on performance is relatively small .
  •   shortcoming :RDB The fatal disadvantage of the file is that the persistence mode of the data snapshot determines that the real-time persistence cannot be achieved , And today, data is more and more important , The massive loss of data is often unacceptable , therefore AOF Persistence becomes the mainstream . Besides ,RDB The file needs to be in a specific format , Compatibility is poor ( Like the old version Redis Not compatible with the new version of RDB file ).
  •   about RDB Persistence , On the one hand is bgsave It's going on fork In operation Redis The main process will block , On the other hand , Sub process writing data to the hard disk will also bring IO pressure .


2、AOF Persistence

  •   And RDB Persistence corresponds to ,AOF The advantage of this is that it supports second level persistence 、 Compatibility is good. , The disadvantage is that the files are large 、 Slow recovery 、 Great impact on performance .
  •   about AOF Persistence , The frequency of writing data to the hard disk is greatly increased (everysec Second level under strategy ),IO More stressful , It may even cause AOF Additional blocking problem .
  •  AOF Document rewriting and RDB Of bgsave similar , There will be fork The blocking and the subprocess's IO The pressure problem . relatively speaking , because AOF Write data to the hard disk more often , So right. Redis The performance of the main process will be affected more

6、 ... and 、Redis Performance management

1、 see Redis Memory usage

redis-cli
info memory

2. Memory fragmentation rate

  1. The value of memory allocated by the operating system used_memory_rss Divide Redis The memory value used used_memory It is calculated that memory fragmentation is inefficient allocation by the operating system / Caused by reclaiming physical memory ( Discontinuous physical memory allocation )
  2. Track memory fragmentation rate for understanding Redis The resource performance of an instance is very important :
  • The memory fragmentation rate is slightly higher than 1 It's reasonable , This value indicates that the memory fragmentation rate is relatively low
  • The memory fragmentation rate exceeds 1.5, explain Redis It consumes the physical memory 150%, among 50% It's the memory fragmentation rate . Need to be in redis-cli Enter... On the tool shutdown save command , And restart Redis The server .
  • The memory fragmentation rate is lower than 1 Of , explain Redis Memory allocation exceeds physical memory , The operating system is swapping memory . Need to increase available physical memory or reduce Redis Memory footprint .
     

3、 Memory usage

  •  redis The memory usage of the instance exceeds the maximum available memory , The operating system will start memory and memory swap Space exchange .
  •   Ways to avoid memory swapping :
    • Select the installation for the cache data size Redis example
    • Use as much as possible Hash Data structure storage
    • Set up key The expiration time of

4、 Internal recovery key

  •   Ensure reasonable distribution redis Limited memory resources .
  •   When the set maximum threshold is reached , You need to select one key Recycling strategy for , By default, the recycling policy is to prohibit deletion .
 Modify... In configuration file  maxmemory-policy  Property value :

vim /etc/redis/6379.conf
#----598 uncomment ----
maxmemory-policy noenviction

volatile-lru	: Use LRU The algorithm eliminates the data from the data set with the expiration time set 
volatile-ttl	: Select the data that is about to expire from the data set with expiration time 
volatile-random	: Randomly select data from the data set with expiration time set 
allkeys-lru		: Use LRU Algorithms eliminate data from all data sets 
allkeys-random	: Select any data from the data set 
noenviction		: Ban data obsolescence 

summary

RDB and AOF Comparison of the two :

  • RDB Pay attention to the results 、 Small volume 、 Low security 、 The priority is higher
  • AOF In addition to results, focus on process 、 Large size 、 High safety 、 High priority

 

版权声明
本文为[C chord~]所创,转载请带上原文链接,感谢
https://yzsam.com/2022/04/202204231406063809.html