How pbidea imports large medical insurance files

PbIdea How to import large medical insurance files ？

The national medical insurance program is based on zip Provide drugs in the format 、 Material Library Download ,zip Inside the file is a text file , The text file is utf8 code , There are hundreds of decompressed M. This document is right PB Come on , It's hard to deal with .PbIdea There are special functions to process this file .

use first uo_httpclient or uo_curl object , According to the medical insurance interface document zip File download , Save as a file . Suppose the file is saved as d:\ 202202187369608330634159091.zip. Next, how to import the data in this compressed package into the database .

First step ： Parse data file , Break it down into several small json file .

Break down into small pieces json The advantage of data files is that they don't take up too much memory , Each data import can be processed quickly .

uo_yibao yb; yb = create uo_yibao

uo_json jsYB;jsYB = create uo_json

yb.ImportYiBaoZip("d:\ 202202187369608330634159091.zip",jsYB)

establish yb and json object , And then call the function. ImportYiBaoZip.ImportYiBaoZip The parameters of the function can be zip file , You don't need to decompress . It can also be decompressed txt file .

Under normal circumstances , This process is about 10 More seconds . If something goes wrong , See for yourself jsYB Deal with the contents of . If it's right ,jsYB The content is like this ：

See the picture below ：

“files” part , Is an array , Contains a list of files , You can cycle it out , use uo_json After the parsing , then dwImportJson Import DW.

“lines” part , Is to detect which rows of data in the data may have problems , Manual correction is required . These lines are not resolved to JSON Li Lai .

“message” part , Prompt information , If something goes wrong , Here is the error message .

“result” part , How many lines have been successfully parsed this time .

The second step ： Resolve each of JSON Import DW.

JSON The file format is array , Each line of sub field data parsed from the data file .Iine Is the number of lines in the data file ,f1-fn Field name , Each according to the business definition .

string ls_file

int i

i = 0

do while true

  if jsYB.Get("/files/" + string(i) + "/file",ref ls_file) then

       dw_1.dataobject = "" // Here can be your own definition DW, You can also leave it blank ,uo_json Will dynamically create a suitable DW

       jsData = create uo_json

       if jsData.parseFile(ls_file) then

          if jsData.dwImportJson(dw_1) then

          // Here, update to the database in your own way

       end if

       end if       

       destroy jsData

  else

       exit // Cycle through all the files in the list

  end if

  i++

loop

Through the above code , Loop will all JSON File import DW.

With 300 many M Of TXT file （ZIP The file is about 60M about ） Come on , expect 20-30 Seconds can all be imported into DW. As for how long it takes to import the database , It depends on the functions realized by their respective businesses .

Of course , After data import , You can go through the cycle by yourself , Take out each file name , And delete it .

                  Great freedom  QQ781770213

                   2022/4/14