Problems and solutions of database migration

Catalog

One Old library table closing

Two Smooth migration of data

3、 ... and The business impact is minimal

Four Incremental data write back

5、 ... and Migration rollback scheme

6、 ... and Old library offline

7、 ... and In advance 、 In the matter 、 After the event 、 What should we do

---

This article mainly explains the problems encountered in the process of data migration from the old database to the new database , And the solutions to these problems , Hope to give you a little inspiration or help .

One Old library table closing

Because multiple services of multiple business terminals share an old library , And multiple services share a set of passwords to access the old database .

How to make sure that only we can write to the table of the old library ？

In order to exclude other business services directly connected to the old library , Write the old library's table , We added a field to the table , For example A, To identify our end-to-end writes to the table . If it wasn't for us to write to the table , The field should be null .

Why do we have to make sure that we close up ？

Suppose a service on the business side writes to the table of the old library , And then we did data migration , The data of the tables in the new and old databases will be inconsistent , This is intolerable in business .

Two Smooth migration of data

We have 4 Tables , common 1 Billion of data , Migration is needed . The core of this is ： How to solve the time-consuming problem of migrating data 、 How to reduce the impact of table stop writing time on business test .

In the beginning, our plan was RD Stop writing ,DBA Executing commands through scripts , Migrate historical data directly from the old database to the new database , The whole process assessment is expected to require 1 Hours , This will mean the need to stop writing 1 Hours , Then synchronize the result to the business side , The business side can't stand the result .

How to solve the time-consuming problem of migrating data ？

We use ALI Of DTS Migrating data .DTS Can synchronize historical data and incremental data .

Historical data is migrated , So how to make it smooth ？

We use the new library cluster as the slave of the old library to synchronize data , When we stop writing this business 、 And the data of the new library is consistent with that of the old library ,DBA Upgrade a machine in the new library cluster to the main library , Other machines are attached to this main library , As the slave Library of this machine .

3、 ... and The business impact is minimal

In the process of data migration , If you can be insensitive to business , It's the best . So how to do a good job with the least impact on the business ？

First , We analyze it through business data , Find the bottom of the business . Data migration in the low ebb , Little impact on the business .

secondly , We stop writing at the minute level , When I stop writing , We've always been readable . This is tolerable in business .

Four Incremental data write back

Why write back data ？

In the process of migrating data , Due to some team scheduling problems , The service reads the old library , At this time, in order to ensure that other teams can read the incremental data , Ensure that the business is not affected , We're going through Databus The data is synchronized , Back to Lao Ku .

Yes, of course , Our expectation is to maintain only the new library , So we ask other business end to schedule the read and receive ports ,2 Six months later, the other business end will go through our interface , Realize the reading and closing .

5、 ... and Migration rollback scheme

In the process of data migration, the rollback scheme does not necessarily need , But in terms of the completeness of one thing , Still needed . For this data migration , What we need to consider is Service rollback 、 Database rollback .

Service rollback ？

It means that the service switches from reading a new library to reading an old one , This can be used as a data source dynamic switch , You can flexibly switch between new and old data sources .

Database rollback ？

If it's incremental data, we need to synchronize them from the new library to the old one , Ensure data integrity .

6、 ... and Old library offline

When the data migration is finished ？ No , We also need the offline library .

From the perspective of corporate assets , With a new library , To reduce costs , We need to take the old library offline .

From the integrity of one thing , We just finished migrating the data 90%, The rest 10% We need to offline the old library .

7、 ... and In advance 、 In the matter 、 After the event 、 What should we do

In advance ： We should email the company's R & D and business , The subject of the email 、 Scope of coverage 、 Affect function 、 Data migration point in time 、 person in charge 、 Migration plan 、 And nail communication group .

In the matter ： We should confirm the feasibility of the scheme with each business end 、 And DBA、QA Build close communication , Synchronize the progress of the scheme at any time 、 The development progress 、 Migration progress .

After the event ： We should email R & D and business 、BI、 Big data and other teams , The database migration has been completed , Synchronize migration results in time . The nail group also needs synchronization .