Tencent cloud GPU best practices - remote development training using jupyter pycharm

background

As an alchemy Engineer ,GPU There is no doubt that it will bring great efficiency improvement to our training model , However, the actual development is usually written locally and copied to GPU Running on the machine , In this way, there will be a certain reduction in efficiency , Moreover, maintaining two sets of environments at the same time may occasionally report errors . So why don't we go directly to GPU Develop directly on the machine ？

This paper will mainly describe several common remote development methods , There's always one for you .

1. GPU Machine and environment preparation
2. jupyter notebook long-range
3. Local pycharm long-range jupyter Development
4. long-range pycharm

1. GPU Machine and environment preparation

https://cloud.tencent.com/product/gpu

First, buy one GPU machine , Depending on the demand, you can choose computational or GPU Inferential ;

When choosing a mirror , If there is GPU Drive pre installation can be preferred , It can avoid the complicated process of installing the driver ：

https://cloud.tencent.com/document/product/560/30211#.E6.AD.A5.E9.AA.A43.EF.BC.9A.E9.80.89.E6.8B.A9.E9.95.9C.E5.83.8F

So I'm going to use GPU Computational +CentOS 7.6 As a demonstration ：

without GPU Driver pre installation options , We had to install it manually , Refer to the following documents ：

https://cloud.tencent.com/document/product/560/8048

https://cloud.tencent.com/developer/article/1835925

2. jupyter notebook long-range

Jupyter Notebook brief introduction

• Jupyter Notebook It's an open source Web application , Be similar to Web The notebook , We can use it to write code 、 The formula 、Markdown Explanatory text and drawings , And you can share the created documents .
• Jupyter Notebook The most convenient function is that it can run code in real time , And return the running result of the code segment , Support Visualization 、IDE Equal ability , It greatly improves the efficiency of model construction and analysis .
• At present, it has been widely used in data processing 、 The data simulation 、 Statistical modeling 、 Important areas of machine learning .

2.1 install

pip3 install jupyter

2.2 Generate a notebook The configuration file

jupyter notebook --generate-config

After execution, a configuration file will be generated , ~/.jupyter/jupyter_notebook_config.py

2.3 Set the password

$ jupyter notebook password
Enter password: ****
Verify password: ****
[NotebookPasswordApp] Wrote hashed password to ~/.jupyter/jupyter_notebook_config.json

2.4 Modify the configuration

modify jupyter_notebook_config.py

c.NotebookApp.ip='192.168.0.1'
c.NotebookApp.password = u'sha:ce... The ciphertext just copied '
c.NotebookApp.open_browser = False# Do not open the browser after starting the service 
c.NotebookApp.port =8888 # You can specify a port ,  Use this port when accessing 
The directory to use for notebooks and kernels.
c.NotebookApp.notebook_dir = '/'## Set the startup directory to /

2.5 Background operation

nohup jupyter notebook --allow-root > jupyter.log 2>&1 &

After starting , You can access the public network IP For remote development ( Remember to open it. GPU Machine network whitelist )

At this time to run notebook Is directly in GPU Run above , So don't worry about consuming local resources , There is no need to run locally and put it in GPU Tossed back and forth on the machine .

3. Local pycharm long-range jupyter Development

At this time, remote development has been realized , But it is estimated that many small partners will feel that they are very used to developing directly with web pages , period IDE Of hint, I don't know what method is in the library , How to call , At this time we can use IDEA pycharm Remote development .

1. First download pycharm pro ( The community version doesn't have jupyter function ) https://www.jetbrains.com/zh-cn/pycharm/

2. Get into pycharm after , Find... In the settings jupyter notebook The server , Select the second configured service and enter the external network in our first step jupyter link ;

3. In order to make GPU Server's Python lib The version is consistent with the local version , And the local can not download so many lib, We can configure remote python Interpreter ： So the local machine even Python No need to install , Greatly liberate storage space ;

stay pycharm Set the distance from the item found Python Interpreter , Create a new interpreter , Input GPU Server public network IP And ssh Port for remote connection ：

4. After setting up , You can use local pycharm Even on GPU Machine training developed ;

4. long-range pycharm

The last section , If there is a small partner who thinks it is necessary to install one locally pycharm, It takes up too much space , Is there any way to pycharm Don't use it ？

There is ,projector, Directly in GPU It's installed on the machine pycharm, long-range web Development ;

1. install python3、python3-pip：

yum install -y python3 python3-pip

2. Install dependency library files ：

yum install libxext6 libxrender1 libxtst6 libfreetype6 libxi6

3. pip install projector

pip3 install --upgrade pip

pip3 install projector-installer

Each run projector Pre execution

export LC_ALL=en_US.utf-8

export LANG=en_US.utf-8

（ Be careful ： First run projector, May prompt projector : command not found, take /usr/local/bin/projector  Add environment variables ）

4. install pycharm

projector install

4. To configure projector

perform  projector config edit  Make configuration changes , add to projector -server service

5. function

projector run

Execution will generate URL, Change to public network IP After that, it can be developed remotely .

Ref

1. projector https://github.com/JetBrains/projector-installer

2. https://blog.jetbrains.com/zh-hans/blog/2021/03/25/projector-ide/