Repository for tutorials, examples and starter scripts for using the MTU HPC cluster

Overview

MTU-HPC-Starter

Repository for tutorials, examples and starter scripts for using the MTU HPC cluster

Connecting to the MTU HPC cluster

Within the college, access to the cluster is via SSH to the host:

com-gpu-cluster.cit.ie

On Windows the simplest way to get an SSH client is to use PuTTy. This is available on all lab PCs.

Linux machines usually have an SSH client on the terminal.

An account on the MTU HPC cluster is usually requested for you by your lecturer or supervisor.

Setup git

Get the Starter Repository using git:

This repository contains some useful sample code for running and submitting jobs on the MTU HPC cluster. This repo can be cloned into your home directory on the HPC cluster and used as a starting point for some your own work.

Once logged into the cluster, use the command at the root of your home directory:

git clone https://github.com/MTU-HPC/MTU-HPC-Starter.git

This will download some basic scripts for getting started with your own projects on thr MTU HPC. Git pull from this repo sometimes as it will be updated with new examples as needed.


Using Miniconda for Python.

THe MTU HPC cluster has a number of versions of Python, Tensorflow and other libraries needed for ML scripts. However, it is dificult to maintain Python libraries and versions systemwide that suit everyone.

To help with this and provide options for users, we have installed Miniconda systemwide. Miniconda allows users to install multiple Python versions and Python libraries in their own separate "environments". This can be done without needing administrator permissions.

Creating and activating conda environments for running python scripts

To use Miniconda, you should have a ".condarc" file at the root of your Home directory. At a minimum it should have the text listed below.

.condarc:

env_prompt: '({name})'
auto_activate_base: false
channels:
  - conda-forge
  - bioconda
  - defaults

The .condarc file tells Miniconda/conda to:

  • env_prompt --- put the name of your currently active conda environment at the front of your command prompt
  • auto_activate_base --- When you login to your cluster account, it will not automatically put you in the base conda environment.
  • channels --- The web repo/location conda install will install Python and Python packages from. "conda-forge" has many ML/AI/Data Analytics packages available.

To see if you already have the .condarc file, type at the commandline:

ls -la

to get a list of files in your home directory. If .condarc is not there, it can be created using the "nano" text editor:

nano .condarc

and copy and paste the text above into .condarc.

One-time only: Initialise conda in your account

To use conda, it must be "initialised" once before it can be used.

To initialise conda, run the following command:

conda init

You will need to exit your SSH session and reconnect for conda to be ready for use. This does not need to be done again.

Create a conda environement for running code

To create a conda environment type the following at the commandline:

conda create --name my_python3_env python=3.8

This create an environemnt called "my_python3_env" and it will install Python 3.8 into that environment. If you don't specify the version of Python, it will install the latest version it can find.

To see your environmensts type:

conda env list

This will should show you 2 environemts now:

  • the base environment
  • my_python3_env - the one we just created. The name is whatever you wish it to be.

At this stage, we have created an envrionment, but we are not using it yet.

Activating a conda environment

Before anything else, type:

 python --version 

This should show you Python 2.7 or some such old Python. This is because the "system" Python is version 2.7. Running a Python script from the commandline now would need the Python script to be compatible with that old version. We cannot upgrade the system python because the operating system itself relies on old Python scripts.

Activate your new conda environment with:

 conda activate my_python3_env

Now run python --version again. The version is now something like Python 3.8. After activating the conda environemnt, we are now using the version of Python you have installed. We are also using the Python packages that are inside this new environment.

Adding packages to a conda enironment

When you are in a "conda" environemt (i.e. the environemnt is "active") the start of your command propmt should show the name of the environment:

(my_python3_env) [email protected] ~]$

To install something like "matplotlib", use conda to install it:

 conda install matplotlib yaml 

That last command will try to install the 2 packages "matplotlib" and "yaml" into the current active conda environment.

Managing conda environments

If you have 5 programs that use matplotlib and Python 3.8, they can all use the same conda environment. There is no need to create a new environment for every program or script.

If, however, you need to run a library which only works on Python 3.6 or needs very different packages to your main environment, you could create a new conda environment for running those scripts:

conda create --name biology_python_env python=3.6
conda activate biology_python_env
conda install 
   

   

Deactivating conda environment

When you are finished, or you wish to use a different conda environment you can exit from the conda environemnt thus:

conda deactivate
💡 Catatan Materi Bahasa Pemrogramman Python

Repository catatan kuliah Andika Tulus Pangestu selama belajar Dasar Pemrograman dengan Python.

0 Oct 10, 2021
Collections of Beautiful Latex Snippets

HandyLatex Collections of Beautiful Latex Snippets Table 👉 Succinct table with bold separation line and gray text %################## Dependencies ##

Xintao 15 Apr 11, 2022
An ongoing curated list of OS X best applications, libraries, frameworks and tools to help developers set up their macOS Laptop.

macOS Development Setup Welcome to MacOS Local Development & Setup. An ongoing curated list of OS X best applications, libraries, frameworks and tools

Paul Veillard 3 Apr 03, 2022
Code for our SIGIR 2022 accepted paper : P3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning

P3 Ranker Implementation for our SIGIR2022 accepted paper: P3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-bas

14 Jan 04, 2023
30 days of Python programming challenge is a step-by-step guide to learn the Python programming language in 30 days

30 days of Python programming challenge is a step-by-step guide to learn the Python programming language in 30 days. This challenge may take more than100 days, follow your own pace.

Asabeneh 17.7k Jan 07, 2023
AiiDA plugin for the HyperQueue metascheduler.

aiida-hyperqueue WARNING: This plugin is still in heavy development. Expect bugs to pop up and the API to change. AiiDA plugin for the HyperQueue meta

AiiDA team 3 Jun 19, 2022
:blue_book: Automatic documentation from sources, for MkDocs.

mkdocstrings Automatic documentation from sources, for MkDocs. Features Python handler features Requirements Installation Quick usage Features Languag

Timothée Mazzucotelli 1.1k Dec 31, 2022
A course-planning, course-map rendering and GPA-calculation web service, designed for the SFU (Simon Fraser University) student.

SFU Course Planner What is the overall goal of the project (i.e. what does it do, or what problem is it solving)? As the title suggests, this project

Ash Peng 1 Oct 21, 2021
Quick tutorial on orchest.io that shows how to build multiple deep learning models on your data with a single line of code using python

Deep AutoViML Pipeline for orchest.io Quickstart Build Deep Learning models with a single line of code: deep_autoviml Deep AutoViML helps you build te

Ram Seshadri 6 Oct 02, 2022
A Python validator for SHACL

pySHACL A Python validator for SHACL. This is a pure Python module which allows for the validation of RDF graphs against Shapes Constraint Language (S

RDFLib 187 Dec 29, 2022
Essential Document Generator

Essential Document Generator Dead Simple Document Generation Whether it's testing database performance or a new web interface, we've all needed a dead

Shane C Mason 59 Nov 11, 2022
A Python Package To Generate Strong Passwords For You in Your Projects.

shPassGenerator Version 1.0.6 Ready To Use Developed by Shervin Badanara (shervinbdndev) on Github Language and technologies used in This Project Work

Shervin 11 Dec 19, 2022
Compare two CSV files for differences. Colorize the differences and align the columns.

pretty-csv-diff Compare two CSV files for differences. Colorize the differences and align the columns. Command-Line Example Command-Line Usage usage:

Devon 6 Dec 29, 2022
The purpose of this project is to share knowledge on how awesome Streamlit is and can be

Awesome Streamlit The fastest way to build Awesome Tools and Apps! Powered by Python! The purpose of this project is to share knowledge on how Awesome

Marc Skov Madsen 1.5k Jan 07, 2023
A hack to run custom shell commands when building documentation on Read the Docs.

readthedocs-custom-steps A hack to run custom steps when building documentation on Read the Docs. Important: This module should not be installed outsi

Niklas Rosenstein 5 Feb 22, 2022
JTEX is a command line tool (CLI) for rendering LaTeX documents from jinja-style templates.

JTEX JTEX is a command line tool (CLI) for rendering LaTeX documents from jinja-style templates. This package uses Jinja2 as the template engine with

Curvenote 15 Dec 21, 2022
Tips for Writing a Research Paper using LaTeX

Tips for Writing a Research Paper using LaTeX

Guanying Chen 727 Dec 26, 2022
Markdown documentation generator from Google docstrings

mkgendocs A Python package for automatically generating documentation pages in markdown for Python source files by parsing Google style docstring. The

Davide Nunes 44 Dec 18, 2022
Documentation for GitHub Copilot

NOTE: GitHub Copilot discussions have moved to the Copilot Feedback forum. GitHub Copilot Welcome to the GitHub Copilot user community! In this reposi

GitHub 21.3k Dec 28, 2022
Modified fork of CPython's ast module that parses `# type:` comments

Typed AST typed_ast is a Python 3 package that provides a Python 2.7 and Python 3 parser similar to the standard ast library. Unlike ast up to Python

Python 217 Dec 06, 2022