Monitor and log Network and Disks statistics in MegaBytes per second.

Related tags

Loggingdiskmetricsnetworkiopytorch-lightning
Overview

iometrics

Python

Monitor and log Network and Disks statistics in MegaBytes per second.

Install

pip install iometrics

Usage

Pytorch-lightning integration

from pytorch_lightning import Trainer
from iometrics.pytorch_lightning.callbacks import NetworkAndDiskStatsMonitor

net_disk_stats = NetworkAndDiskStatsMonitor()

trainer = Trainer(callbacks=[net_disk_stats])

Metrics generated

  • network/recv_MB_per_sec – Received MB/s on all relevant network interfaces as a SUM.
  • network/sent_MB_per_sec – Sent MB/s on all relevant network interfaces as a SUM.
  • disk/util% – Disk utilization percentage as the average of all disk devices.
  • disk/read_MB_per_sec – Disks read MB/s as the sum of all disk devices.
  • disk/writ_MB_per_sec – Disks written MB/s as the sum of all disk devices.
  • disk/io_read_count_per_sec – Disks read I/O operations per second as the sum of all disk devices.
  • disk/io_writ_count_per_sec – Disks written I/O operations per second as the sum of all disk devices.

Screen shot

Pure-Python implementation (zero dependencies)

Quick check

python -c 'from iometrics.example import usage; usage()'

Example output

|        Network (MBytes/s)       | Disk Util |            Disk MBytes          |           Disk I/O          |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    | I/O Write |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  | val | avg |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| ---:| ---:|
|    4.6 |    3.5 |   0.1 |   0.1 |  49 |   2 |   52.8 |    1.1 |   0.0 |   0.9 |    211 |      4 |   5 |  18 |
|    4.1 |    3.5 |   0.1 |   0.1 |  61 |   3 |   60.4 |    2.4 |  40.3 |   1.7 |    255 |     10 | 149 |  21 |

Full code

import time
from iometrics import NetworkMetrics, DiskMetrics
from iometrics.example import DUAL_METRICS_HEADER
net  = NetworkMetrics()
disk = DiskMetrics()
for i in range(100):
    time.sleep(1)
    net.update_stats()
    disk.update_stats()
    if i % 15 == 0:
        print(DUAL_METRICS_HEADER)
    row = (
        f"| {net.mb_recv_ps.val:6.1f} | {net.mb_recv_ps.avg:6.1f} "
        f"| {net.mb_sent_ps.val:5.1f} | {net.mb_sent_ps.avg:5.1f} "
        f"| {int(disk.io_util.val):3d} | {int(disk.io_util.avg):3d} "
        f"| {disk.mb_read.val:6.1f} | {disk.mb_read.avg:6.1f} "
        f"| {disk.mb_writ.val:5.1f} | {disk.mb_writ.avg:5.1f} "
        f"| {int(disk.io_read.val):4d} | {int(disk.io_read.avg):4d} "
        f"| {int(disk.io_writ.val):3d} | {int(disk.io_writ.avg):3d} "
        f"|"
    )
    print(row)

Contributing

See CONTRIBUTING.md

You might also like...
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

Mathias V. Nielsen 2 Dec 4, 2021
A simple package that allows you to save inputs & outputs as .log files

wolf_dot_log A simple package that allows you to save inputs & outputs as .log files pip install wolf_dot_log pip3 install wolf_dot_log |Instructions|

Alpwuf 1 Nov 16, 2021
This is a wonderful simple python tool used to store the keyboard log.

Keylogger This is a wonderful simple python tool used to store the keyboard log. Record your keys. It will capture passwords and credentials in a comp

Rithin Lehan 2 Nov 25, 2021
Python script to scan log files/system for unauthorized access around system

checkLogs Python script to scan log files/system for unauthorized access around Linux systems Table of contents General info Getting started Usage Gen

James Kelly 1 Feb 25, 2022
ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs
ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

ClusterMonitor A very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs. Usage To start recording

null 23 Oct 4, 2021
Robust and effective logging for Python 2 and 3.
Robust and effective logging for Python 2 and 3.

Robust and effective logging for Python 2 and 3.

Chris Hager 1k Jan 4, 2023
Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

Soda Data Monitoring 51 Jan 1, 2023
A Fast, Extensible Progress Bar for Python and CLI
A Fast, Extensible Progress Bar for Python and CLI

tqdm tqdm derives from the Arabic word taqaddum (تقدّم) which can mean "progress," and is an abbreviation for "I love you so much" in Spanish (te quie

tqdm developers 23.7k Jan 1, 2023
Rich is a Python library for rich text and beautiful formatting in the terminal.
Rich is a Python library for rich text and beautiful formatting in the terminal.

Rich 中文 readme • lengua española readme • Läs på svenska Rich is a Python library for rich text and beautiful formatting in the terminal. The Rich API

Will McGugan 41.5k Jan 7, 2023
Comments
  • Add Docker Support

    Add Docker Support

    Change Summary

    Add Docker Support

    Related issue number

    https://github.com/elgalu/iometrics/issues/3

    Checklist

    • [ ] Unit tests for the changes exist
    • [ ] Tests pass on CI and coverage is maintained
    • [x] Documentation reflects the changes where applicable
    opened by m0rt3z4gh 0
  • Not working properly in containers (Docker)

    Not working properly in containers (Docker)

    Network metrics properly reported in the Host:

    |        Network (MBytes/s)       | Disk Util |            Disk MBytes          |             Disk I/O            |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    |   I/O Write   |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  |  val  |  avg  |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| -----:| -----:|
|  417.2 |  417.2 |   3.0 |   3.0 |   8 |   8 |    0.9 |    0.9 | 370.5 | 370.5 |    220 |    220 | 11897 | 11897 |
|  335.1 |  376.1 |   2.4 |   2.7 |  10 |   9 |    1.4 |    1.1 | 275.5 | 323.0 |    335 |    278 |  9168 | 10532 |
|  434.2 |  395.5 |   3.2 |   2.9 |   9 |   9 |    3.9 |    2.1 | 169.9 | 272.0 |    653 |    403 | 11450 | 10838 |

    Network metrics are not visible in the container Disclaimer: the timeline extracts are not perfectly synchronized

    |        Network (MBytes/s)       | Disk Util |            Disk MBytes          |             Disk I/O            |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    |   I/O Write   |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  |  val  |  avg  |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| -----:| -----:|
|    0.0 |    0.0 |   0.0 |   0.0 |   7 |   9 |    0.9 |    1.3 |  82.4 | 556.4 |    216 |    310 | 20093 |  9335 |
|    0.0 |    0.0 |   0.0 |   0.0 |   9 |   9 |   24.9 |    2.0 | 375.4 | 550.8 |    783 |    325 |  3604 |  9156 |
|    0.0 |    0.0 |   0.0 |   0.0 |  10 |   9 |   60.5 |    3.8 | 103.9 | 537.2 |   1315 |    355 | 12689 |  9263 |

    To reproduce

    1. generate network I/O outside the container (in the host)
    2. start a container without any special flags, without any admin or additional capabilities

    TODO

    Find a fix with the least possible privileges

    bug 
    opened by elgalu 0
  • Features/azdo pipeline

    Features/azdo pipeline

    Change Summary

    Related issue number

    Checklist

    • [ ] Unit tests for the changes exist
    • [ ] Tests pass on CI and coverage is maintained
    • [ ] Documentation reflects the changes where applicable
    opened by ugurgural 0
Releases(0.0.8)
Owner
Leo Gallucci
Leo is the creator of docker-selenium which aims to get Selenium running as simple and fast as possible
Leo Gallucci
GitHub Repository https://pypi.org/project/iometrics/
Translating symbolicated Apple JSON format crash log into our old friends :)

CrashTranslation Translating symbolicated Apple JSON format crash log into our old friends :) Usage python3 translation.py -i {input_sybolicated_json_

Kam-To 11 May 16, 2022
Stand-alone parser for User Access Logging from Server 2012 and newer systems

KStrike Stand-alone parser for User Access Logging from Server 2012 and newer systems BriMor Labs KStrike This script will parse data from the User Ac

BriMor Labs 69 Nov 01, 2022
Splunk Add-On to collect audit log events from Github Enterprise Cloud

GitHub Enterprise Audit Log Monitoring Splunk modular input plugin to fetch the enterprise audit log from GitHub Enterprise Support for modular inputs

Splunk GitHub 12 Aug 18, 2022
Pretty and useful exceptions in Python, automatically.

better-exceptions Pretty and more helpful exceptions in Python, automatically. Usage Install better_exceptions via pip: $ pip install better_exception

Qix 4.3k Dec 29, 2022
A demo of Prometheus+Grafana for monitoring an ML model served with FastAPI.

ml-monitoring Jeremy Jordan This repository provides an example setup for monitoring an ML system deployed on Kubernetes.

Jeremy Jordan 176 Jan 01, 2023
A basic logging library for Python.

log.py 📖 About: A basic logging library for Python with the capability to: save to files. have custom formats. have custom levels. be used instantiat

Sebastiaan Bij 1 Jan 19, 2022
Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

Soda Data Monitoring 51 Jan 01, 2023
Outlog it's a library to make logging a simple task

outlog Outlog it's a library to make logging a simple task!. I'm a lazy python user, the times that i do logging on my apps it's hard to do, a lot of

ZSendokame 2 Mar 05, 2022
Integrates a UPS monitored by NUT into OctoPrint

OctoPrint UPS This OctoPrint plugin interfaces with a UPS monitored by NUT (Network UPS Tools). Requirements NUT must be configured by the user. This

Shawn Bruce 11 Jul 05, 2022
pyEventLogger - a simple Python Library for making customized Logs of certain events that occur in a program

pyEventLogger is a simple Python Library for making customized Logs of certain events that occur in a program. The logs can be fully customized and can be printed in colored format or can be stored i

Siddhesh Chavan 2 Nov 03, 2022
Yaml - Loggers are like print() statements

Upgrade your print statements Loggers are like print() statements except they also include loads of other metadata: timestamp msg (same as print!) arg

isaac peterson 38 Jul 20, 2022
ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

ClusterMonitor A very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs. Usage To start recording

23 Oct 04, 2021
A Python library that tees the standard output & standard error from the current process to files on disk, while preserving terminal semantics

A Python library that tees the standard output & standard error from the current process to files on disk, while preserving terminal semantics (so breakpoint(), etc work as normal)

Greg Brockman 7 Nov 30, 2022
👻 - Simple Keylloger with Socket

Keyllogs 👻 - Simple Keylloger with Socket Keyllogs 🎲 - Run Keyllogs

Bidouffe 3 Mar 28, 2022
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

Mathias V. Nielsen 2 Dec 04, 2021
🐑 Syslog Simulator hazır veya kullanıcıların eklediği logları belirtilen adreslere ve port'a seçilen döngüde syslog ile gönderilmesini sağlayan araçtır. | 🇹🇷

syslogsimulator hazır ürün loglarını SIEM veya log toplayıcısına istediğiniz portta belirli sürelerde göndermeyi sağlayan küçük bir araçtır.

Enes Aydın 3 Sep 28, 2021
Token Logger with python

Oxy Token Stealer Features Grabs discord tokens Grabs chrome passwords Grabs edge passwords Nothing else, I don't feel like releasing full on malware

oxy 1 Feb 12, 2022
A python library used to interact with webots robocup game web logs

A python library used to interact with webots robocup game web logs

Hamburg Bit-Bots 2 Nov 05, 2021
A lightweight logging library for python applications

cakelog a lightweight logging library for python applications This is a very small logging library to make logging in python easy and simple. config o

2 Jan 05, 2022
A cool logging replacement for Python.

Welcome to Logbook Travis AppVeyor Supported Versions Latest Version Test Coverage Logbook is a nice logging replacement. It should be easy to setup,

1.4k Nov 11, 2022