Custom SLURM wrapper scripts to make finding job histories and system resource usage more easily accessible

Overview

SLURM Wrappers

Executables

job-history

A simple wrapper for grabbing data for completed and running jobs.

nodes-busy

Developed for the HPC systems at University of Arizona. High memory nodes are differentiated from standard nodes with AvailableFeatures=hi_mem.

nodes-busy.mp4
You might also like...
Projeto job insights - Projeto avaliativo da Trybe do Bloco 32: Introdução à Python

Termos e acordos Ao iniciar este projeto, você concorda com as diretrizes do Código de Ética e Conduta e do Manual da Pessoa Estudante da Trybe. Boas

NUM Alert - A work focus aid created for the Hack the Job hackathon

Contributors: Uladzislau Kaparykha, Amanda Hahn, Nicholas Waller Hackathon Team Name: N.U.M General Purpose: The general purpose of this program is to

Standalone PyQGIS application for executing custom scripts without a QGIS GUI.

PyQGIS Standalone Script Executer Standalone PyQGIS application that is able to run a custom script, in this case Proximity.py without the need of a G

This is a Poetry plugin that will make it possible to build projects using custom TOML files

Poetry Multiproject Plugin This is a Poetry plugin that will make it possible to build projects using custom TOML files. This is especially useful whe

Coinloggr - A learning resource and social platform for the coin collecting community
Coinloggr - A learning resource and social platform for the coin collecting community

Coinloggr A learning resource and social platform for the coin collecting commun

python scripts - mostly automation scripts

python python scripts - mostly automation scripts You can set your environment in various ways bash #!/bin/bash python - locally on remote host #!/bi

A simple bot that will help you in your learning and make it more fun.

hyperskill-SimpleChattyBot-python A simple bot that will help you in your learning and make it more fun. Syntax bot.py Stages Stage #1: Zuhura Bot we

Make after-work Mending More flexible In Python

Mending Make after-work Mending More flexible In Python A Lite Package focuses on making project's after-post mending pythonic and flexible. Certainly

Comments
  • KeyError: GPUAlloc error when running nodes-busy

    KeyError: GPUAlloc error when running nodes-busy

    Howdy,

    Blake J pointed us (UAB Research Computing) to your nice slurm-wrapper repo :-)

    While job-history and system-busy work great on our cluster, nodes-busy is crashing on our cluster with the following:

    ❯ ./bin/nodes-busy
    
    nodes-busy: visualize live system resource usage.
    Trouble seeing the output? Try 'nodes-busy --ascii'
    
    Traceback (most recent call last):
      File "./bin/nodes-busy", line 766, in <module>
        merged = merge(job_data, node_data)
      File "./bin/nodes-busy", line 330, in merge
        nodes_dictionary[node]["JOBS"][job] = {"CPUs":cpus, "GPUs":jobs_dictionary[job]["GPUAlloc"],"EndTime":jobs_dictionary[job]["EndTime"],"Partition":jobs_dictionary[job]["Partition"],"Restarts":jobs_dictionary[job]["Restarts"]}
    KeyError: 'GPUAlloc'
    

    I'm digging through the code, but figured I'd post this for tracking purposes.

    Thanks, Mike

    opened by flakrat 7
  • nodes-busy

    nodes-busy "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 53196: invalid start byte"

    Howdy, noticed the following error today when running nodes-busy on our cluster.

    > nodes-busy
    
    Traceback (most recent call last):
      File "/share/apps/rc/bin/nodes-busy", line 1177, in <module>
        job_data = get_scontrol_job_data()
      File "/share/apps/rc/bin/nodes-busy", line 370, in get_scontrol_job_data
        output = out.decode('utf-8').split("\n")
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 53196: invalid start byte
    

    I have no idea what's in position 53196, but I fixed the issue with the following addition to output.decode to ignore errors:

    diff --git a/bin/nodes-busy b/bin/nodes-busy
    index 6fb1744..aee728f 100755
    --- a/bin/nodes-busy
    +++ b/bin/nodes-busy
    @@ -367,7 +367,7 @@ def get_scontrol_job_data(target_job = None):
             sys.exit(1)
    
         # Split up space-delimited output into a job dictionary
    -    output = out.decode('utf-8').split("\n")
    +    output = out.decode('utf-8', 'ignore').split("\n")
         for job in output:
             details = job.split(' ')
             for i in details:
    
    opened by flakrat 3
  • Feature added user arg to past-jobs

    Feature added user arg to past-jobs

    Added new argument to past-jobs to support selecting jobs by someone other than the user running the script:

    $ past-jobs -d 30 -u flakrat
    
                             Jobs submitted by user flakrat in last 30 days.
    
    JobID    Start       User            JobName         Partition  Account    State       ExitCode
    ------------------------------------------------------------------------------------------------
    14364561 2022-06-02  flakrat         hostname        express    flakrat    CANCELLED        0:0
    14385956 2022-06-05  flakrat         hostname        express    flakrat    COMPLETED        0:0
    14850816 2022-06-12  flakrat         hostname        express    flakrat    COMPLETED        0:0
    
    opened by flakrat 0
Releases(v0.2.0)
  • v0.2.0(May 11, 2022)

  • v0.1.0(Dec 9, 2021)

    The scripts in this repository have some bits and pieces that are specific to the HPC setup at University of Arizona. These parts are being cleaned up.

    Source code(tar.gz)
    Source code(zip)
Owner
Sara
HPC Consultant at University of Arizona, UITS
Sara
Automation of VASP DFT workflows with ASE - application scripts

This repo contains a library that aims at automatizing some Density Functional Theory (DFT) workflows in VASP by using the ASE toolkit.

Frank Niessen 5 Sep 06, 2022
An open-source Python project series where beginners can contribute and practice coding.

Python Mini Projects A collection of easy Python small projects to help you improve your programming skills. Table Of Contents Aim Of The Project Cont

Leah Nguyen 491 Jan 04, 2023
A package selector for building your confy nest

Hornero A package selector for building your comfy nest About Hornero helps you to install your favourite packages on your fresh installed Linux distr

Santiago Soler 1 Nov 22, 2021
pvaPy provides Python bindings for EPICS pvAccess

PvaPy - PvAccess for Python The PvaPy package is a Python API for EPICS7. It supports both PVA and CA providers, all standard EPICS7 types (structures

EPICS Base 25 Dec 05, 2022
Web app to find your chance of winning at Texas Hold 'Em

poker_mc Web app to find your chance of winning at Texas Hold 'Em A working version of this project is deployed at poker-mc.ue.r.appspot.com. It's run

Aadith Vittala 7 Sep 15, 2021
Github Star Tracking app with Streamlit

github-star-tracking-python-app Github Star Tracking app with Streamlit #8daysofstreamlit How to run it locally? Clone or Download & Unzip the Repo En

amrrs 4 Sep 22, 2022
Repository, with small useful and functional applications

Repositorio,com pequenos aplicativos uteis e funcionais A ideia e ir deselvolvendo pequenos aplicativos funcionais e adicionar a este repositorio List

GabrielDuke 6 Dec 06, 2021
This repository holds those infrastructure-level modules, that every application requires that follows the core 12-factor principles.

py-12f-common About This repository holds those infrastructure-level modules, that every application requires that follows the core 12-factor principl

Tamás Benke 1 Dec 15, 2022
A Python Web Application for Checking vaccine slots by pincodes and auto slot booking.

The Dashboard is developed using Bokeh and python 3.5+. This dashboard is useful for you if you are looking for something which will help you to book the vaccine slot once slots become available. Oth

Suraj Deshmukh 10 Jan 23, 2022
A place where one-off ideas/partial projects can live comfortably

A place to post ideas, partial projects, or anything else that doesn't necessarily warrant its own repo, from my mind to the web.

Carson Scott 2 Feb 25, 2022
validation for pre-commit.ci configuration

pre-commit-ci-config validation for pre-commit.ci configuration installation pip install pre-commit-ci-config api pre_commit_ci_config.SCHEMA a cfgv s

pre-commit.ci 17 Jul 11, 2022
a simple thing that i made for fun :trollface:

we-do-a-little-trolling about a simple thing that i made for fun. requirements and instructions first you need to install obs , then start the virtual

ranon rat 6 Jul 15, 2022
Pipenv-local-deps-repro - Reproduction of a local transitive dependency on pipenv

Reproduction of the pipenv bug with transitive local dependencies. Clone this re

Lucas Duailibe 2 Jan 11, 2022
Logging-monitoring-instrumentation - A brief repository on logging monitoring and instrumentation in Python

logging-monitoring-instrumentation A brief repository on logging monitoring and

Noah Gift 6 Feb 17, 2022
OntoSeer is a tool to help users build better quality ontologies

Ontoseer This document provides documentation for the first version of OntoSeer.OntoSeer is a tool that monitors the ontology development process andp

Knowledgeable Computing and Reasoning Lab 9 Aug 15, 2022
Project repository of Apache Airflow, deployed on Docker in Amazon EC2 via GitLab.

Airflow on Docker in EC2 + GitLab's CI/CD Personal project for simple data pipeline using Airflow. Airflow will be installed inside Docker container,

Ammar Chalifah 13 Nov 29, 2022
Better GitHub statistics images for your profile, with stats from private and public repos

Better GitHub statistics images for your profile, with stats from private and public repos

Jacob Strieb 2k Dec 30, 2022
The most widely used Python to C compiler

Welcome to Cython! Cython is a language that makes writing C extensions for Python as easy as Python itself. Cython is based on Pyrex, but supports mo

7.6k Jan 03, 2023
Flask-built web application that simulates a time and cost calculator for charging Electric Vehicles.

ev_charging_calculator Flask-built web application that simulates a time and cost calculator for charging Electric Vehicles. The project aims to simul

1 Nov 03, 2021
Python AVL Protocols Server for Codec 8 and Codec 8 Extended Protocols

pycodecs Package provides python AVL Protocols Server for Codec 8 and Codec 8 Extended Protocols This package will parse the AVL Data and log it in hu

Vardharajulu K N 2 Jun 21, 2022