Use Function Compute to package and download OSS files [Encounter Pit Collection]

前言

项目中的ossThe server does not plan to continue to renew in the future,Next, I want to back up all the files to the local for preservation

权衡了官方文档After several methods, the method of function calculation was selected

原因主要是：有一个bucketIt wasn't dealt with at the time,All files are storedbucket的根目录下面,而不是子目录,As a result, the file or directory cannot be directly selected for download（Because there are tens of thousands of files in the root directory）

步骤

先根据官方文档to deploy Function Compute

开始踩坑

1. Function Compute has been successfully deployed,But click the test code to run and report an error,如下

   {
         
   "errorMessage": "No JSON object could be decoded",
   "errorType": "ValueError",
   "stackTrace": [
   [
   "\\u00a0\\u00a0File \"/code/main.py\"",
   "line 28",
   "in main_handler",
   "evt = json.loads(request_body)"
   ],
   [
   "\\u00a0\\u00a0File \"/var/fc/lang/python2/lib/python2.7/json/__init__.py\"",
   "line 339",
   "in loads",
   "return _default_decoder.decode(s)"
   ],
   [
   "\\u00a0\\u00a0File \"/var/fc/lang/python2/lib/python2.7/json/decoder.py\"",
   "line 364",
   "in decode",
   "obj, end = self.raw_decode(s, idx=_w(s, 0).end())"
   ],
   [
   "\\u00a0\\u00a0File \"/var/fc/lang/python2/lib/python2.7/json/decoder.py\"",
   "line 382",
   "in raw_decode",
   "raise ValueError(\"No JSON object could be decoded\")"
   ]
   ]
   }
   

   The reason is that it cannot be readevent.json这个文件,After consulting Alibaba Cloud's after-sales engineers, the conclusion is,当前python2.7version is no longer readable,需要修改代码,直接将json配置写到main文件中

   代码如下,main.py文件的20行到40行左右,See the note for the location of the modification

   def main_handler(environ, start_response):
       LOG.info('event: %s', environ)
       try:
           request_body_size = int(environ.get('CONTENT_LENGTH', 0))
       except (ValueError):
           request_body_size = 0
       request_body = environ['wsgi.input'].read(request_body_size)
   
       # The reason this line is commented is because python2的运行环境中 wsgi.inputReally can't receive the configuration. The receiving configuration here is mainlywsgi接收 event.json
       # evt = json.loads(request_body)
       evt = {
         
           "region": "cn-zhangjiakou",
           "bucket": "zhushang-applet-code",
           "source-dir": "test/"
       }
       context = environ['fc.context']
   
       oss_client = get_oss_client(evt, context)
       ret = _main_handler(oss_client, evt, context)
   
       status = '302 FOUND'
       response_headers = [('Location',sign_url(oss_client, ret))]
       start_response(status, response_headers)
   
       return "ok"
   
   

2. 官方文档上说,If you want to download files in the root directory,使用./即可

   
   实际上,After I modified it according to him,运行的时候,并没有效果,The local will be downloaded to a only22BAnd the compressed package that cannot be decompressed,并且ossThere is no corresponding compressed package above,也就是意味着失败了

   After doing various tests at the end,发现使用""Empty strings are available as directories

   代码如下

   evt = {
           "region": "cn-zhangjiakou",
           "bucket": "zhushang-applet-code",
           "source-dir": ""
   }
   

3. The downloaded file is too large to be compressed and then fails

   运行代码报错如下：Zipfile size would require ZIP64 extensions

   {
         
       "errorMessage": "Zipfile size would require ZIP64 extensions", 
       "errorType": "LargeZipFile", 
       "stackTrace": [
           [
               "\\u00a0\\u00a0File \"/code/main.py\"", 
               "line 37", 
               "in main_handler", 
               "ret = _main_handler(oss_client, evt, context)"
           ], 
           [
               "\\u00a0\\u00a0File \"/code/main.py\"", 
               "line 52", 
               "in _main_handler", 
               "return zip_files(oss_client, source_dir, source_files, dest_file)"
           ], 
           [
               "\\u00a0\\u00a0File \"/code/main.py\"", 
               "line 116", 
               "in zip_files", 
               "task_q.run()"
           ], 
           [
               "\\u00a0\\u00a0File \"/code/task_queue.py\"", 
               "line 40", 
               "in run", 
               "raise self.__exc_info[1]"
           ]
       ]
   }
   

   Find a reliable one online博客

   里面说,由于python 2.7 脚本限制,超过2G就会报这个错,需要修改一下配置,can be increased to4G（But actually I zipped20多个G的也是可以的）

   修改代码如下,main.py文件的80-100Line left and right modify one line of this function

       def producer(queue):
       mem_buf = MemBuffer(queue)
       # The default compressed file size is only supported2G,Support can be done in this way4G
       zip_file = StreamZipFile(mem_buf, 'w', allowZip64=True)
   
       if isinstance(source_files, list):
           for obj in source_files:
               zip_add_file(zip_file, obj)
       elif isinstance(source_dir, basestring):
           for obj in oss2.ObjectIterator(oss_client, prefix=source_dir):
               zip_add_file(zip_file, obj.key)
       else:
           raise Exception('either `source_files` or `source_dir` must be speicified')
   
       zip_file.close()
       mem_buf.flush_buffer()
   

4. 函数计算默认是10execution time in minutes

   The default established by Function Compute is 弹性实例,The maximum runtime is 10分钟

   需要修改为性能实例,Then modify the execution timeout to 7200秒