Repository for tutorials, examples and starter scripts for using the MTU HPC cluster

Overview

MTU-HPC-Starter

Repository for tutorials, examples and starter scripts for using the MTU HPC cluster

Connecting to the MTU HPC cluster

Within the college, access to the cluster is via SSH to the host:

com-gpu-cluster.cit.ie

On Windows the simplest way to get an SSH client is to use PuTTy. This is available on all lab PCs.

Linux machines usually have an SSH client on the terminal.

An account on the MTU HPC cluster is usually requested for you by your lecturer or supervisor.

Setup git

Get the Starter Repository using git:

This repository contains some useful sample code for running and submitting jobs on the MTU HPC cluster. This repo can be cloned into your home directory on the HPC cluster and used as a starting point for some your own work.

Once logged into the cluster, use the command at the root of your home directory:

git clone https://github.com/MTU-HPC/MTU-HPC-Starter.git

This will download some basic scripts for getting started with your own projects on thr MTU HPC. Git pull from this repo sometimes as it will be updated with new examples as needed.


Using Miniconda for Python.

THe MTU HPC cluster has a number of versions of Python, Tensorflow and other libraries needed for ML scripts. However, it is dificult to maintain Python libraries and versions systemwide that suit everyone.

To help with this and provide options for users, we have installed Miniconda systemwide. Miniconda allows users to install multiple Python versions and Python libraries in their own separate "environments". This can be done without needing administrator permissions.

Creating and activating conda environments for running python scripts

To use Miniconda, you should have a ".condarc" file at the root of your Home directory. At a minimum it should have the text listed below.

.condarc:

env_prompt: '({name})'
auto_activate_base: false
channels:
  - conda-forge
  - bioconda
  - defaults

The .condarc file tells Miniconda/conda to:

  • env_prompt --- put the name of your currently active conda environment at the front of your command prompt
  • auto_activate_base --- When you login to your cluster account, it will not automatically put you in the base conda environment.
  • channels --- The web repo/location conda install will install Python and Python packages from. "conda-forge" has many ML/AI/Data Analytics packages available.

To see if you already have the .condarc file, type at the commandline:

ls -la

to get a list of files in your home directory. If .condarc is not there, it can be created using the "nano" text editor:

nano .condarc

and copy and paste the text above into .condarc.

One-time only: Initialise conda in your account

To use conda, it must be "initialised" once before it can be used.

To initialise conda, run the following command:

conda init

You will need to exit your SSH session and reconnect for conda to be ready for use. This does not need to be done again.

Create a conda environement for running code

To create a conda environment type the following at the commandline:

conda create --name my_python3_env python=3.8

This create an environemnt called "my_python3_env" and it will install Python 3.8 into that environment. If you don't specify the version of Python, it will install the latest version it can find.

To see your environmensts type:

conda env list

This will should show you 2 environemts now:

  • the base environment
  • my_python3_env - the one we just created. The name is whatever you wish it to be.

At this stage, we have created an envrionment, but we are not using it yet.

Activating a conda environment

Before anything else, type:

 python --version 

This should show you Python 2.7 or some such old Python. This is because the "system" Python is version 2.7. Running a Python script from the commandline now would need the Python script to be compatible with that old version. We cannot upgrade the system python because the operating system itself relies on old Python scripts.

Activate your new conda environment with:

 conda activate my_python3_env

Now run python --version again. The version is now something like Python 3.8. After activating the conda environemnt, we are now using the version of Python you have installed. We are also using the Python packages that are inside this new environment.

Adding packages to a conda enironment

When you are in a "conda" environemt (i.e. the environemnt is "active") the start of your command propmt should show the name of the environment:

(my_python3_env) [email protected] ~]$

To install something like "matplotlib", use conda to install it:

 conda install matplotlib yaml 

That last command will try to install the 2 packages "matplotlib" and "yaml" into the current active conda environment.

Managing conda environments

If you have 5 programs that use matplotlib and Python 3.8, they can all use the same conda environment. There is no need to create a new environment for every program or script.

If, however, you need to run a library which only works on Python 3.6 or needs very different packages to your main environment, you could create a new conda environment for running those scripts:

conda create --name biology_python_env python=3.6
conda activate biology_python_env
conda install 
   

   

Deactivating conda environment

When you are finished, or you wish to use a different conda environment you can exit from the conda environemnt thus:

conda deactivate
BakTst_Org is a backtesting system for quantitative transactions.

BakTst_Org 中文reademe:传送门 Introduction: BakTst_Org is a prototype of the backtesting system used for BTC quantitative trading. This readme is mainly di

18 May 08, 2021
30 Days of google cloud leaderboard website

30 Days of Cloud Leaderboard This is a leaderboard for the students of Thapar, Patiala who are participating in the 2021 30 days of Google Cloud Platf

Developer Student Clubs TIET 13 Aug 25, 2022
Main repository for the Sphinx documentation builder

Sphinx Sphinx is a tool that makes it easy to create intelligent and beautiful documentation for Python projects (or other documents consisting of mul

5.1k Jan 04, 2023
[Unofficial] Python PEP in EPUB format

PEPs in EPUB format This is a unofficial repository where I stock all valid PEPs in the EPUB format. Repository Cloning git clone --recursive Mickaël Schoentgen 9 Oct 12, 2022

This is a tool to make easier brawl stars modding using csv manipulation

Brawler Maker : Modding Tool for Brawl Stars This is a tool to make easier brawl stars modding using csv manipulation if you want to support me, just

6 Nov 16, 2022
Course materials for: Geospatial Data Science

Course materials for: Geospatial Data Science These course materials cover the lectures for the course held for the first time in spring 2022 at IT Un

Michael Szell 266 Jan 02, 2023
💡 Catatan Materi Bahasa Pemrogramman Python

Repository catatan kuliah Andika Tulus Pangestu selama belajar Dasar Pemrograman dengan Python.

0 Oct 10, 2021
Automatic links from code examples to reference documentation

sphinx-codeautolink Automatic links from Python code examples to reference documentation at the flick of a switch! sphinx-codeautolink analyses the co

Felix Hildén 41 Dec 17, 2022
Flask-Rebar combines flask, marshmallow, and swagger for robust REST services.

Flask-Rebar Flask-Rebar combines flask, marshmallow, and swagger for robust REST services. Features Request and Response Validation - Flask-Rebar reli

PlanGrid 223 Dec 19, 2022
Fully typesafe, Rust-like Result and Option types for Python

safetywrap Fully typesafe, Rust-inspired wrapper types for Python values Summary This library provides two main wrappers: Result and Option. These typ

Matthew Planchard 32 Dec 25, 2022
A Power BI/Google Studio Dashboard to analyze previous OTC CatchUps

OTC CatchUp Dashboard A Power BI/Google Studio dashboard analyzing OTC CatchUps. File Contents * ├───data ├───old summaries ─── *.md ├

11 Oct 30, 2022
A simple tutorial to get you started with Discord and it's Python API

Hello there Feel free to fork and star, open issues if there are typos or you have a doubt. I decided to make this post because as a newbie I never fo

Sachit 1 Nov 01, 2021
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, A

Donne Martin 24.5k Jan 09, 2023
Beautiful static documentation generator for OpenAPI/Swagger 2.0

Spectacle The gentleman at REST Spectacle generates beautiful static HTML5 documentation from OpenAPI/Swagger 2.0 API specifications. The goal of Spec

Sourcey 1.3k Dec 13, 2022
Swagger Documentation Generator for Django REST Framework: deprecated

Django REST Swagger: deprecated (2019-06-04) This project is no longer being maintained. Please consider drf-yasg as an alternative/successor. I haven

Marc Gibbons 2.6k Jan 03, 2023
Types that make coding in Python quick and safe.

Type[T] Types that make coding in Python quick and safe. Type[T] works best with Python 3.6 or later. Prior to 3.6, object types must use comment type

Contains 17 Aug 01, 2022
A curated list of python programming language blogs

Python Blogs A curated list of python programming language blogs Contribute Companies/Organization # A B C D E F G H I J K L M N O P Q R S T U V W X Y

Rizky D. Onto 48 Nov 15, 2022
Data science on SDGs - Udemy Online Course Material: Data Science on Sustainable Development Goals

Data Science on Sustainable Development Goals (SDGs) Udemy Online Course Material: Data Science on Sustainable Development Goals https://bit.ly/data_s

Frank Kienle 1 Jan 04, 2022
Python Tool to Easily Generate Multiple Documents

Python Tool to Easily Generate Multiple Documents Running the script doesn't require internet Max Generation is set to 10k to avoid lagging/crashing R

2 Apr 27, 2022
sphinx builder that outputs markdown files.

sphinx-markdown-builder sphinx builder that outputs markdown files Please ★ this repo if you found it useful ★ ★ ★ If you want frontmatter support ple

Clay Risser 144 Jan 06, 2023