Custom SLURM wrapper scripts to make finding job histories and system resource usage more easily accessible

Overview

SLURM Wrappers

Executables

job-history

A simple wrapper for grabbing data for completed and running jobs.

nodes-busy

Developed for the HPC systems at University of Arizona. High memory nodes are differentiated from standard nodes with AvailableFeatures=hi_mem.

nodes-busy.mp4
You might also like...
Projeto job insights - Projeto avaliativo da Trybe do Bloco 32: Introdução à Python

Termos e acordos Ao iniciar este projeto, você concorda com as diretrizes do Código de Ética e Conduta e do Manual da Pessoa Estudante da Trybe. Boas

NUM Alert - A work focus aid created for the Hack the Job hackathon

Contributors: Uladzislau Kaparykha, Amanda Hahn, Nicholas Waller Hackathon Team Name: N.U.M General Purpose: The general purpose of this program is to

Standalone PyQGIS application for executing custom scripts without a QGIS GUI.

PyQGIS Standalone Script Executer Standalone PyQGIS application that is able to run a custom script, in this case Proximity.py without the need of a G

This is a Poetry plugin that will make it possible to build projects using custom TOML files

Poetry Multiproject Plugin This is a Poetry plugin that will make it possible to build projects using custom TOML files. This is especially useful whe

Coinloggr - A learning resource and social platform for the coin collecting community
Coinloggr - A learning resource and social platform for the coin collecting community

Coinloggr A learning resource and social platform for the coin collecting commun

python scripts - mostly automation scripts

python python scripts - mostly automation scripts You can set your environment in various ways bash #!/bin/bash python - locally on remote host #!/bi

A simple bot that will help you in your learning and make it more fun.

hyperskill-SimpleChattyBot-python A simple bot that will help you in your learning and make it more fun. Syntax bot.py Stages Stage #1: Zuhura Bot we

Make after-work Mending More flexible In Python

Mending Make after-work Mending More flexible In Python A Lite Package focuses on making project's after-post mending pythonic and flexible. Certainly

Comments
  • KeyError: GPUAlloc error when running nodes-busy

    KeyError: GPUAlloc error when running nodes-busy

    Howdy,

    Blake J pointed us (UAB Research Computing) to your nice slurm-wrapper repo :-)

    While job-history and system-busy work great on our cluster, nodes-busy is crashing on our cluster with the following:

    ❯ ./bin/nodes-busy
    
    nodes-busy: visualize live system resource usage.
    Trouble seeing the output? Try 'nodes-busy --ascii'
    
    Traceback (most recent call last):
      File "./bin/nodes-busy", line 766, in <module>
        merged = merge(job_data, node_data)
      File "./bin/nodes-busy", line 330, in merge
        nodes_dictionary[node]["JOBS"][job] = {"CPUs":cpus, "GPUs":jobs_dictionary[job]["GPUAlloc"],"EndTime":jobs_dictionary[job]["EndTime"],"Partition":jobs_dictionary[job]["Partition"],"Restarts":jobs_dictionary[job]["Restarts"]}
    KeyError: 'GPUAlloc'
    

    I'm digging through the code, but figured I'd post this for tracking purposes.

    Thanks, Mike

    opened by flakrat 7
  • nodes-busy

    nodes-busy "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 53196: invalid start byte"

    Howdy, noticed the following error today when running nodes-busy on our cluster.

    > nodes-busy
    
    Traceback (most recent call last):
      File "/share/apps/rc/bin/nodes-busy", line 1177, in <module>
        job_data = get_scontrol_job_data()
      File "/share/apps/rc/bin/nodes-busy", line 370, in get_scontrol_job_data
        output = out.decode('utf-8').split("\n")
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 53196: invalid start byte
    

    I have no idea what's in position 53196, but I fixed the issue with the following addition to output.decode to ignore errors:

    diff --git a/bin/nodes-busy b/bin/nodes-busy
    index 6fb1744..aee728f 100755
    --- a/bin/nodes-busy
    +++ b/bin/nodes-busy
    @@ -367,7 +367,7 @@ def get_scontrol_job_data(target_job = None):
             sys.exit(1)
    
         # Split up space-delimited output into a job dictionary
    -    output = out.decode('utf-8').split("\n")
    +    output = out.decode('utf-8', 'ignore').split("\n")
         for job in output:
             details = job.split(' ')
             for i in details:
    
    opened by flakrat 3
  • Feature added user arg to past-jobs

    Feature added user arg to past-jobs

    Added new argument to past-jobs to support selecting jobs by someone other than the user running the script:

    $ past-jobs -d 30 -u flakrat
    
                             Jobs submitted by user flakrat in last 30 days.
    
    JobID    Start       User            JobName         Partition  Account    State       ExitCode
    ------------------------------------------------------------------------------------------------
    14364561 2022-06-02  flakrat         hostname        express    flakrat    CANCELLED        0:0
    14385956 2022-06-05  flakrat         hostname        express    flakrat    COMPLETED        0:0
    14850816 2022-06-12  flakrat         hostname        express    flakrat    COMPLETED        0:0
    
    opened by flakrat 0
Releases(v0.2.0)
  • v0.2.0(May 11, 2022)

  • v0.1.0(Dec 9, 2021)

    The scripts in this repository have some bits and pieces that are specific to the HPC setup at University of Arizona. These parts are being cleaned up.

    Source code(tar.gz)
    Source code(zip)
Owner
Sara
HPC Consultant at University of Arizona, UITS
Sara
Send notifications created in Frappe or ERPNext as push notication via Firebase Cloud Message(FCM)

FCM Notification for ERPNext Send notifications created in Frappe or ERPNext as push notication via Firebase Cloud Message(FCM) Steps to use the app:

Tridz 9 Nov 14, 2022
Goddard A collection of small, simple strategies for Freqtrade

Goddard A collection of small, simple strategies for Freqtrade. Simply add the strategy you choose in your strategies folder and run. ⚠️ General Crypt

Shane Jones 118 Dec 14, 2022
Repls goes to sleep due to inactivity, but to keep it awake, simply host a webserver and ping it.

Repls goes to sleep due to inactivity, but to keep it awake, simply host a webserver and ping it. This repo will help you make a webserver with a bit of console controls.

2 Mar 01, 2022
Superset custom path for python

It is a common requirement to have superset running under a base url, (https://mydomain.at/analytics/ instead of https://mydomain.at/). I created the

9 Dec 14, 2022
ESteg - A simple steganography program for python

ESteg A simple steganography program to embed the contents of a text file into a

Jithin Renji 1 Jan 02, 2022
The mock Pokemon Environment I built in 2019 to study Reinforcement Learning + Pokemon

ghetto-pokemon-rl-environment ##NOT MAINTAINED! Fork and maintain yourself. Environment I made back in 2019 to use Pokemon to practice reinforcement l

2 Dec 09, 2021
Tool to automate the enumeration of a website (CTF)

had4ctf Tool to automate the enumeration of a website (CTF) DISCLAIMER: THE TOOL HAS BEEN DEVELOPED SOLELY FOR EDUCATIONAL PURPOSE ,I WILL NOT BE LIAB

Had 2 Oct 24, 2021
This is a program for Carbon Emission calculator.

Summary This is a program for Carbon Emission calculator. Usage This will calculate the carbon emission by each person on various factors. Contributor

Ankit Rane 2 Feb 18, 2022
🤞 Website-Survival-Detection

- 🤞 Website-Survival-Detection It can help you to detect the survival status of the website in batches and return the status code! - 📜 Instructions

B0kd1 4 Nov 14, 2022
IG Trading Algos and Scripts in Python

IG_Trading_Algo_Scripts_Python IG Trading Algos and Scripts in Python This project is a collection of my work over 2 years building IG Trading Algorit

191 Oct 11, 2022
GitHub Actions Version Updater Updates All GitHub Action Versions in a Repository and Creates a Pull Request with the Changes.

GitHub Actions Version Updater GitHub Actions Version Updater is GitHub Action that is used to update other GitHub Actions in a Repository and create

Maksudul Haque 42 Dec 22, 2022
It's a repo for Cramer's rule, which is some math crap or something idk

It's a repo for Cramer's rule, which is some math crap or something idk (just a joke, it's not crap; don't take that seriously, math teachers)

Module64 0 Aug 31, 2022
Gba-free-fonts - Free font resources for GBA game development

gba-free-fonts Free font resources for GBA game development This repo contains m

28 Dec 30, 2022
A python script for combining multiple native SU2 format meshes into one mesh file for multi-zone simulations.

A python script for combining multiple native SU2 format meshes into one mesh file for multi-zone simulations.

MKursatUzuner 1 Jan 20, 2022
Python implementation of the ASFLIP advection method

This is a python implementation of the ASFLIP advection method . We would like to hear from you if you appreciate this work.

Raymond Yun Fei 133 Nov 13, 2022
pybind11 — Seamless operability between C++11 and Python

pybind11 — Seamless operability between C++11 and Python Setuptools example • Scikit-build example • CMake example pybind11 is a lightweight header-on

pybind 12.1k Jan 08, 2023
Open slidebook .sldy files in Python

Work in progress slidebook-python Open slidebook .sldy files in Python To install slidebook-python requires Python = 3.9 pip install slidebook-python

The Institute of Cancer Research 2 May 04, 2022
Up to date simple useragent faker with real world database

fake-useragent info: Up to date simple useragent faker with real world database Features grabs up to date useragent from useragentstring.com randomize

Victor K. 2.9k Jan 04, 2023
That is a example of a Book app on Python, made with support of all JS libraries on React framework

React+Python Books App You can use this repository whenever you want Used for a video Create the database: python -m dbutils Start the web server: pyt

Koma Human 1 Apr 20, 2022
Team Hash Brown Science4Cast Submission

Team Hash Brown Science4Cast Submission This code reproduces Team Hash Brown's (@princengoc, @Xieyangxinyu) best submission (ee5a) for the competition

3 Feb 02, 2022