Batch import of orange single micro service

The above code is all generated automatically , Bulk import
If you configure this form , Batch import is supported , Then the code will generate , You won't .

For performance reasons , It can be stored locally , Of course, it can also be stored in minio etc. oss in

The orange list will generate some fields that need to be ignored , Like the Creator , Creation time , Updated by , Update time , Logical deletion, etc .
Comments are also given here , If you want to ignore other fields , You can also modify it directly here , Just one line .

If excel Order in , Not the order in the field , You can adjust it properly here .

From these two points, we can see , Modification is very simple . Secondary orange single , Deliberately did not consider the generation of 100% Code for , But it does reserve some space for secondary development to fill in the blanks . This makes it very easy to strike a balance . In terms of code quality , framework 、 Business and development efficiency . Very balanced .

If the excessive pursuit of code generation accounts for , Code quality and runtime efficiency will be greatly reduced .

The code comments generated here , The explanations are very clear , If you want to do dictionary translation of imported data , Just give this set Add the field name . It's simple .

such as ,excel What's in it gradeName, Such as “ second grade ”, This will translate into the corresponding id, such as ：2, Then insert it into the original business table , This is the dictionary translation ability in batch import

In the actual imported code , Orange list has no simple use The ordinary way , Instead, this SAX Callback mode for , In this way, when there is too much imported data , It can not only accelerate the import efficiency , At the same time, it can save memory .

All data imported into memory will be , Verify the associated fields

Here is batch verification , These are the code generated by orange list , If it is generated by mistake , Can be modified directly , And remove .

These are data validation based on constant Dictionary

These are the verification of the dictionary table , These dictionary tables are physical data tables , But the data is in the cache , So it's more efficient .

These are the associations between ordinary business tables id Validation of the .
So in the orange list , These three categories id Verification is fully supported .
Constant class , Cached dictionary data , And the dictionary in the data table .

For performance , If so 10 10000 pieces of batch imported data , Yes 4 Such a field to be verified .

Then the number of verification is 40 Ten thousand times , This performance overhead is huge .

Orange single optimizes... In two ways , The first is that some dictionaries are redis cache , What's more, all validation is batch validation ,

For example, verify once 1000 strip , Then the number of operations on the database , It will be reduced a thousand times
If so, verify id Is based on redis Of , The number of accesses to the database will be halved .

The last step , It's batch insertion . Even this step , The code generated by orange order has also been optimized enough ,
The second parameter is -1, All tables are inserted in batch by default , This batch is real insert list That batch , It's not a cycle insert That fake batch .
If the second parameter is not -1, It's a specific number .
such as 10 Ten thousand data , It can't be one insert list Insert all , This is limited by database Syntax . And you can't be so reckless .

So we can set each insert list The number of data pieces of , such as 2000, that 10 Million data , Will be divided into 50 Insert... Times . And finally submit

This can effectively avoid the syntax restrictions of the database , At the same time, it will not bring huge pressure to the database instantaneously .

Okay , Batch import share here .