easy_sbatch - Batch submitting Slurm jobs with script templates

Overview

easy_sbatch

easy_sbatch - Batch submitting Slurm jobs via script templates

Install

git clone https://github.com/shenwei356/easy_sbatch.git
mkdir -p ~/bin
cp easy_sbatch ~/bin/

Quick start

  1. A simple command (no file or data given):

     $ easy_sbatch 'cat /etc/hostname'
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:50 easy_sbatch.15565347.err
     -rw-r--r-- 1 shenwei cqmu   9 Dec 27 21:50 easy_sbatch.15565347.out
     -rw------- 1 shenwei cqmu 314 Dec 27 21:50 easy_sbatch.15565347.slurm
    
  2. Handling multiple files and specifying a job name:

     $ easy_sbatch 'ls {}' *.fq.gz -J list
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:54 list.15565371-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:54 list.15565371-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565371-read_1.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:55 list.15565373-read_2.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:55 list.15565373-read_2.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565373-read_2.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_1.fq.gz
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_2.fq.gz
    
  3. From stdin:

     $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'
    
  4. Processing paired-ends FASTQ files:

     $ ls read_1.fq.gz \
         | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:56 pe.15565379-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  40 Dec 27 21:56 pe.15565379-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 340 Dec 27 21:56 pe.15565379-read_1.fq.gz.slurm
    
     $ cat pe.15565379-read_1.fq.gz.out
     seqtk mergepe read_1.fq.gz read_2.fq.gz
    

Default template

You can choose another one via the option -t/--template.

You can edit the value of an option, e.g., replace $partition with the default partition of your cluster. But note that the corresponding option from the command line will not take effect since no variable name is ready to replace.

$ cat ~/.easy_sbatch/default.slurm 
#!/bin/bash
#
#SBATCH --partition $partition
#SBATCH --job-name $name
#SBATCH --nodes=1
#SBATCH --cpus-per-task=$ncpus
#SBATCH --mem=$mem
#SBATCH --time=$walltime
#SBATCH --output=$output
#SBATCH --error=$error


cd $$SLURM_SUBMIT_DIR
# echo run on node: $$HOSTNAME >&2

$cmd

Usage

. .out]) -e ERROR, --error ERROR file for batch script's standard error (default: . .err]) -v, --verbose verbosely print information. -vv for just printing command not creating scripts and submitting jobs https://github.com/shenwei356/easy_sbatch ">
usage: easy_sbatch [-h] [-lp | -ls] [-J NAME] [-c NCPUS] [-m MEM] [-p PARTITION] [-w WALLTIME] [-t TEMPLATE]
                   [-O SCRIPT] [-o OUTPUT] [-e ERROR] [-v]
                   command [files ...]

easy_sbatch - Batch submitting Slurm jobs with script template

https://github.com/shenwei356/easy_sbatch 

Slurm script template:

  - Default template is ~/.easy_qsub/default.slurm .
  - You can also choose another one via the option -t/--template.
  - You can edit the value of an option, e.g., replace "$partition"
    with the default partition of your cluster. But note that the
    corresponding option from the command line will not take effect
    since no variable name is ready to replace.

Supported replacement strings in command:

  - "{}" for the full path of a input file.
  - "{/}" for dirname.
  - "{%}" for basename.
  - "{^suffix}" for clipping $suffix.
  - "{%^suffix}" for clipping $suffix from basename.

The generated Slurm scripts are saved in /tmp/easy_qsub-user/ by default,
unless the flag -O/--script is set. If jobs are submitted successfully,
the scripts will be moved to the current directory. If not, they will
be removed.

Examples:

  1. No file or data given:
      $ easy_sbatch 'cat /etc/hostname'

  2. From position arguments:
      $ easy_sbatch 'ls {}' *.fq.gz

  3. From stdin:
      $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'

  4. Slurm script and its output files:
      $ easy_sbatch 'ls' -J hello
        -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:57 hello.15565039.err
        -rw-r--r-- 1 shenwei cqmu  85 Dec 27 20:57 hello.15565039.out
        -rw------- 1 shenwei cqmu 281 Dec 27 20:57 hello.15565039.slurm

  5. Processing paired-ends FASTQ files:
      $ ls read_1.fq.gz | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe

      $ cat pe.15565114.out 
        seqtk mergepe read_1.fq.gz read_2.fq.gz

positional arguments:
  command               command to submit
  files                 input files

optional arguments:
  -h, --help            show this help message and exit
  -lp, --local_p        run commands locally, parallelly
  -ls, --local_s        run commands locally, serially
  -J NAME, --name NAME  job name (default: easy_sbatch)
  -c NCPUS, --ncpus NCPUS
                        number of cpus required per task (default: 24)
  -m MEM, --mem MEM     memory (default: 24gb)
  -p PARTITION, --partition PARTITION
                        partition requested (default: kshdnormal)
  -w WALLTIME, --walltime WALLTIME
                        walltime (default: 30-00:00:00)
  -t TEMPLATE, --template TEMPLATE
                        script template (default: /home/shenwei/.easy_sbatch/default.slurm
  -O SCRIPT, --script SCRIPT
                        output script file
  -o OUTPUT, --output OUTPUT
                        file for batch script's standard output (default: 
      
       .
       
        .out])
  -e ERROR, --error ERROR
                        file for batch script's standard error (default: 
        
         .
         
          .err])
  -v, --verbose         verbosely print information. -vv for just printing command not creating scripts and submitting jobs

https://github.com/shenwei356/easy_sbatch


         
        
       
      

License

MIT License

History

This utility is modified from easy_qsub, which was written for PBS job scheduler.

Owner
Wei Shen
Bioinformatician, postdoc
Wei Shen
Devil - Very Semple Auto Filter V1 Bot

Devil Very Semple Auto Filter V1 Bot

2 Jun 27, 2022
Find all social media accounts with a username!

Aliens_eye FIND ALL SOCIAL MEDIA ACCOUNTS WITH A USERNAME! OSINT To install: Open terminal and type: git clone https://github.com/BLINKING-IDIOT/Alien

Aaron Thomas 84 Dec 28, 2022
Python Programmma DarkMap.py

DarkMap Python Programmma DarkMap.py O'rganish va rasmlarni ko'riosh https://drive.google.com/drive/folders/1l1zybs_0Zy9z_trZYz5R72WrwsE6mFOh?usp=shar

Og'abek 0 May 06, 2022
This repository contains each day of Advent of Code 2021 that I've done.

Advent of Code - 2021 I will use this repository as my Advent of Code1 (AoC) repo for the 2021 challenge. I'm changing how I am tackling the problems

Brett Chapin 2 Jan 12, 2022
Plugin to manage site, circuit and device diagrams and documents in Netbox

Netbox Documents Plugin A plugin designed to faciliate the storage of site, circuit and device specific documents within NetBox Note: Netbox v3.2+ is

Jason Yates 38 Dec 24, 2022
A synchronous, single-threaded interface for starting processes on Linux

A synchronous, single-threaded interface for starting processes on Linux

Spencer Baugh 27 Jan 28, 2022
Quantity Takeoff with Python. Collecting groups of elements by filters

The free tool QuantityTakeoff allows you to group elements from Revit and IFC models (in BIMJSON-CSV format) with just a few filters and find the required volume values for the grouped elements.

OpenDataBIM 9 Jan 06, 2023
Whatsapp Messenger master

Whatsapp Messenger master

Swarup Kharul 5 Nov 21, 2021
Repository, with small useful and functional applications

Repositorio,com pequenos aplicativos uteis e funcionais A ideia e ir deselvolvendo pequenos aplicativos funcionais e adicionar a este repositorio List

GabrielDuke 6 Dec 06, 2021
WATTS provides a set of Python classes that can manage simulation workflows for multiple codes where information is exchanged at a coarse level

WATTS (Workflow and Template Toolkit for Simulation) provides a set of Python classes that can manage simulation workflows for multiple codes where information is exchanged at a coarse level.

13 Dec 23, 2022
My tools box script for sigma

sigma_python_toolbox My tools box script for sigma purpose My goal is not to replace sigma but to put at disposal the scripts that I think to help me

4 Jun 20, 2022
Block the annoying Token Grabbers on your discord

General We have seen that in the last time many discord servers are infected by fake discord nitro links we want to put an end to this and have develo

BadTiger Network 2 Jul 16, 2022
An AddOn storing wireguard configuration

Wireguard Database Connector Overview Development Status: 0.1.7 (alpha) First of all, I'd like to thank Jared McKnight for wireguard who inspired me t

Markus Neubauer 3 Dec 30, 2021
Notebook researcher - Notebook researcher with python

notebook_researcher To run the server, you must follow these instructions: At th

4 Sep 02, 2022
Aerospace utilities: flight conditions package, standard atmosphere model, and more.

Aerospace Utilities About Module that contains commonly-used aerospace utilities for problem solving. Flight Condition: input altitude to compute comm

1 Jan 03, 2022
Cisco IOS-XE Operations Program. Shows operational data using restconf and yang

XE-Ops View operational and config data from devices running Cisco IOS-XE software. NoteS The build folder is the latest build. All other files are fo

18 Jul 23, 2022
Multifunctional Analysis of Regions through Input-Output

MARIO Multifunctional Analysis of Regions through Input-Output. (Documents) What is it MARIO is a python package for handling input-output tables and

14 Dec 25, 2022
Framework To Ease Operating with Quantum Computers

QType Framework To Ease Operating with Quantum Computers Concept # define an array of 15 cubits:

Antonio Párraga Navarro 2 Jun 06, 2022
python scripts to perform coin die clustering (performed on Riedones3D).

python scripts to perform coin die clustering (performed on Riedones3D).

Sofiane 2 Apr 29, 2022
This is a repository containing the backend and the frontend of a simple pokédex.

Pokémon This is a repository containing the backend and the frontend of a simple pokédex. This is a work in progress project! Project Structure 🗂 pok

André Rato 1 Nov 28, 2021