Monitor and log Network and Disks statistics in MegaBytes per second.

Last update: May 03, 2022

Overview

iometrics

Monitor and log Network and Disks statistics in MegaBytes per second.

Install

pip install iometrics

Usage

Pytorch-lightning integration

from pytorch_lightning import Trainer
from iometrics.pytorch_lightning.callbacks import NetworkAndDiskStatsMonitor

net_disk_stats = NetworkAndDiskStatsMonitor()

trainer = Trainer(callbacks=[net_disk_stats])

Metrics generated

network/recv_MB_per_sec – Received MB/s on all relevant network interfaces as a SUM.
network/sent_MB_per_sec – Sent MB/s on all relevant network interfaces as a SUM.
disk/util% – Disk utilization percentage as the average of all disk devices.
disk/read_MB_per_sec – Disks read MB/s as the sum of all disk devices.
disk/writ_MB_per_sec – Disks written MB/s as the sum of all disk devices.
disk/io_read_count_per_sec – Disks read I/O operations per second as the sum of all disk devices.
disk/io_writ_count_per_sec – Disks written I/O operations per second as the sum of all disk devices.

Screen shot

Pure-Python implementation (zero dependencies)

Quick check

python -c 'from iometrics.example import usage; usage()'

Example output

|        Network (MBytes/s)       | Disk Util |            Disk MBytes          |           Disk I/O          |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    | I/O Write |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  | val | avg |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| ---:| ---:|
|    4.6 |    3.5 |   0.1 |   0.1 |  49 |   2 |   52.8 |    1.1 |   0.0 |   0.9 |    211 |      4 |   5 |  18 |
|    4.1 |    3.5 |   0.1 |   0.1 |  61 |   3 |   60.4 |    2.4 |  40.3 |   1.7 |    255 |     10 | 149 |  21 |

Full code

import time
from iometrics import NetworkMetrics, DiskMetrics
from iometrics.example import DUAL_METRICS_HEADER
net  = NetworkMetrics()
disk = DiskMetrics()
for i in range(100):
    time.sleep(1)
    net.update_stats()
    disk.update_stats()
    if i % 15 == 0:
        print(DUAL_METRICS_HEADER)
    row = (
        f"| {net.mb_recv_ps.val:6.1f} | {net.mb_recv_ps.avg:6.1f} "
        f"| {net.mb_sent_ps.val:5.1f} | {net.mb_sent_ps.avg:5.1f} "
        f"| {int(disk.io_util.val):3d} | {int(disk.io_util.avg):3d} "
        f"| {disk.mb_read.val:6.1f} | {disk.mb_read.avg:6.1f} "
        f"| {disk.mb_writ.val:5.1f} | {disk.mb_writ.avg:5.1f} "
        f"| {int(disk.io_read.val):4d} | {int(disk.io_read.avg):4d} "
        f"| {int(disk.io_writ.val):3d} | {int(disk.io_writ.avg):3d} "
        f"|"
    )
    print(row)

Contributing

See CONTRIBUTING.md

loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

2 Dec 4, 2021

A simple package that allows you to save inputs & outputs as .log files

wolf_dot_log A simple package that allows you to save inputs & outputs as .log files pip install wolf_dot_log pip3 install wolf_dot_log |Instructions|

1 Nov 16, 2021

This is a wonderful simple python tool used to store the keyboard log.

Keylogger This is a wonderful simple python tool used to store the keyboard log. Record your keys. It will capture passwords and credentials in a comp

2 Nov 25, 2021

Python script to scan log files/system for unauthorized access around system

checkLogs Python script to scan log files/system for unauthorized access around Linux systems Table of contents General info Getting started Usage Gen

1 Feb 25, 2022

ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

ClusterMonitor A very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs. Usage To start recording

23 Oct 4, 2021

Robust and effective logging for Python 2 and 3.

1k Jan 4, 2023

Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

51 Jan 1, 2023

A Fast, Extensible Progress Bar for Python and CLI

tqdm tqdm derives from the Arabic word taqaddum (تقدّم) which can mean "progress," and is an abbreviation for "I love you so much" in Spanish (te quie

23.7k Jan 1, 2023

Rich is a Python library for rich text and beautiful formatting in the terminal.

Rich 中文 readme • lengua española readme • Läs på svenska Rich is a Python library for rich text and beautiful formatting in the terminal. The Rich API

41.5k Jan 7, 2023

Comments

Add Docker Support
Change Summary

Add Docker Support

Related issue number

https://github.com/elgalu/iometrics/issues/3

Checklist

[ ] Unit tests for the changes exist

[ ] Tests pass on CI and coverage is maintained

[x] Documentation reflects the changes where applicable
opened by m0rt3z4gh 0

Not working properly in containers (Docker)

Network metrics properly reported in the Host:

|        Network (MBytes/s)       | Disk Util |            Disk MBytes          |             Disk I/O            |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    |   I/O Write   |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  |  val  |  avg  |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| -----:| -----:|
|  417.2 |  417.2 |   3.0 |   3.0 |   8 |   8 |    0.9 |    0.9 | 370.5 | 370.5 |    220 |    220 | 11897 | 11897 |
|  335.1 |  376.1 |   2.4 |   2.7 |  10 |   9 |    1.4 |    1.1 | 275.5 | 323.0 |    335 |    278 |  9168 | 10532 |
|  434.2 |  395.5 |   3.2 |   2.9 |   9 |   9 |    3.9 |    2.1 | 169.9 | 272.0 |    653 |    403 | 11450 | 10838 |

Network metrics are not visible in the container Disclaimer: the timeline extracts are not perfectly synchronized

|        Network (MBytes/s)       | Disk Util |            Disk MBytes          |             Disk I/O            |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    |   I/O Write   |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  |  val  |  avg  |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| -----:| -----:|
|    0.0 |    0.0 |   0.0 |   0.0 |   7 |   9 |    0.9 |    1.3 |  82.4 | 556.4 |    216 |    310 | 20093 |  9335 |
|    0.0 |    0.0 |   0.0 |   0.0 |   9 |   9 |   24.9 |    2.0 | 375.4 | 550.8 |    783 |    325 |  3604 |  9156 |
|    0.0 |    0.0 |   0.0 |   0.0 |  10 |   9 |   60.5 |    3.8 | 103.9 | 537.2 |   1315 |    355 | 12689 |  9263 |

To reproduce

generate network I/O outside the container (in the host)
start a container without any special flags, without any admin or additional capabilities

TODO

Find a fix with the least possible privileges

bug

opened by elgalu 0

Features/azdo pipeline
Change Summary

Related issue number

Checklist

[ ] Unit tests for the changes exist

[ ] Tests pass on CI and coverage is maintained

[ ] Documentation reflects the changes where applicable
opened by ugurgural 0

Releases(0.0.8)

0.0.8(Jan 20, 2022)
0.0.8 - [2022-01-20]

Added

Add dummy CLI with iometrics start and iometrics replicate proc [#cli]

Source code(tar.gz)
Source code(zip)
0.0.4(Aug 18, 2021)

Fix bug in PyTorch-Lightning integration [#PyTorch-Lightning]

https://pypi.org/project/iometrics/0.0.4/
Source code(tar.gz)
Source code(zip)
0.0.3(Aug 17, 2021)

Add PyTorch-Lightning support via from iometrics.pytorch_lightning.callbacks import NetworkAndDiskStatsMonitor

https://pypi.org/project/iometrics/0.0.3/
Source code(tar.gz)
Source code(zip)
0.0.1(Aug 15, 2021)

Initial release to take https://pypi.org/project/iometrics/
Source code(tar.gz)
Source code(zip)

Owner

Leo Gallucci

Leo is the creator of docker-selenium which aims to get Selenium running as simple and fast as possible

GitHub Repository https://pypi.org/project/iometrics/

Json Formatter for the standard python logger

This library is provided to allow standard python logging to output log data as json objects. With JSON we can make our logs more readable by machines and we can stop writing custom parsers for syslo

1.4k Jan 04, 2023

Greppin' Logs: Leveling Up Log Analysis

This repo contains sample code and example datasets from Jon Stewart and Noah Rubin's presentation at the 2021 SANS DFIR Summit titled Greppin' Logs. The talk was centered around the idea that Forens

20 Sep 14, 2022

A watchdog and logger to Discord for hosting ScPrime servers.

ScpDog A watchdog and logger to Discord for hosting ScPrime servers. Designed to work on Linux servers. This is only capable of sending the logs from

3 Jan 10, 2022

Simple and versatile logging library for python 3.6 above

1 Nov 23, 2022

Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

795 Dec 18, 2022

Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

python-tabulate Pretty-print tabular data in Python, a library and a command-line utility. The main use cases of the library are: printing small table

1.5k Jan 06, 2023

Scout: an open-source version of the monitoring tool

Badger Scout Scout is an open-source version of the monitoring tool used by Badg

2 Jan 13, 2022

Debugging-friendly exceptions for Python

Better tracebacks This is a more helpful version of Python's built-in exception message: It shows more code context and the current values of nearby v

1.2k Dec 28, 2022

A simple package that allows you to save inputs & outputs as .log files

wolf_dot_log A simple package that allows you to save inputs & outputs as .log files pip install wolf_dot_log pip3 install wolf_dot_log |Instructions|

1 Nov 16, 2021

👻 - Simple Keylloger with Socket

Keyllogs 👻 - Simple Keylloger with Socket Keyllogs 🎲 - Run Keyllogs

3 Mar 28, 2022

ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

ClusterMonitor A very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs. Usage To start recording

23 Oct 04, 2021

Monitor and log Network and Disks statistics in MegaBytes per second.

Related tags

Overview

iometrics

Install

Usage

Pytorch-lightning integration

Metrics generated

Screen shot

Pure-Python implementation (zero dependencies)

Quick check

Example output

Full code

Contributing

You might also like...

loghandler allows you to easily log messages to multiple endpoints.

A simple package that allows you to save inputs & outputs as .log files

This is a wonderful simple python tool used to store the keyboard log.

Python script to scan log files/system for unauthorized access around system

ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

Robust and effective logging for Python 2 and 3.

Soda SQL Data testing, monitoring and profiling for SQL accessible data.

A Fast, Extensible Progress Bar for Python and CLI

Rich is a Python library for rich text and beautiful formatting in the terminal.

Comments

Add Docker Support

Change Summary

Related issue number

Checklist

Not working properly in containers (Docker)

To reproduce

TODO

Features/azdo pipeline

Change Summary

Related issue number

Checklist

Releases(0.0.8)

0.0.8(Jan 20, 2022)

0.0.8 - [2022-01-20]

Added

0.0.4(Aug 18, 2021)

0.0.3(Aug 17, 2021)

0.0.1(Aug 15, 2021)

Owner

Leo Gallucci

Json Formatter for the standard python logger

Greppin' Logs: Leveling Up Log Analysis

A watchdog and logger to Discord for hosting ScPrime servers.

Simple and versatile logging library for python 3.6 above

Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

Scout: an open-source version of the monitoring tool

Debugging-friendly exceptions for Python

A simple package that allows you to save inputs & outputs as .log files

👻 - Simple Keylloger with Socket

ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

Stand-alone parser for User Access Logging from Server 2012 and newer systems

metovlogs is a very simple logging library

Pretty and useful exceptions in Python, automatically.

A Python package which supports global logfmt formatted logging.

A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics.

A very basic esp32-based logic analyzer capable of sampling digital signals at up to ~3.2MHz.

A cool logging replacement for Python.

Python logging made (stupidly) simple

HTTP(s) "monitoring" webpage via FastAPI+Jinja2. Inspired by https://github.com/RaymiiOrg/bash-http-monitoring