Record a redis value, which is not a small production accident you want

background

When writing a settlement business, you need to dynamically start from redis Value judgment ,redis Before the value is the value of the last minute , But now we have modified the technical scheme , Revised to redis Stored is the value of the previous cycle . The modified code is tested in the test environment and pre release environment without any problems . So it was released to the production environment , And empty the old redis, It is found that the settled data is always abnormal . So we started the road of investigation . The company's redis It is commonly used , One master and two slaves election mechanism

The way to solve the problem

Just went online and found something wrong , So I added a log , Redistributed and found redis The value obtained is not from the previous cycle , It's the last minute . Connect... With tools ( The tool is connected to the previous master ). It is found that the data from the above is wrong , I was immediately blinded . Went to the bathroom , Think about it , Finally, check the existing master-slave situation , It is found that the connection is from , Finally, modify the connection to the main , Restore to normal after deletion and re creation .

It was just a small accident , But it is often overlooked , Because the master and slave may change , Test environment and pre release environment redis Are single node . Ignoring the master-slave thing , This led to the above joke . I hope you can take a warning