Small toolkit for python multiprocessing logging to file

Related tags

Loggingmp-socket-logging
Overview

Small Toolkit for Python Multiprocessing Logging

This is a small toolkit for solving unsafe python mutliprocess logging (file logging and rotation)

Install

Only for Python 3.5+

pip install mp-socket-logging

Features

                            (child process)
                           +---------------+
             +<------------+     client    |
 +-------+-+-+             +---------------+
 |           |              (child process)
 +  Socket   |             +---------------+
 |           +<------------+     client    |
 +---------+-+             +---------------+
       |
       |                 (main process)
       |               +---------------+          +--------------+
       +-------------->+     server    +--------->+   log file   +
                       +---------------+          +--------------+
  • The package uses unix socket for communication between processes.
  • I/O operations will all be done by server in main process.
  • Client Class itself is a logging.Handler which will be added to user's logger inside child process. Client handler will do the formatting and sending the logs to server side through socket.
  • Inside child process, user can do logging as normal with the logger.

Quick Start

  • start a server before in the main process
server_handler = ServerHandler(LOG_FILE, maxBytes=MAX_BYTES)
server_handler.setLevel(logging.INFO)
server = Server(server_handler, batch_size=BATCH_SIZE)
server.start()
  • register the client to your logger inside multiprocessing worker for sending logs to server through socket
  • you can define your own formatter to add to the handler (otherwise a default formatter will be used)
client_logger = logging.getLogger("client_logger")
formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(filename)s- [%(levelname)s]: %(message)s"
)
register_handler(client_logger, logging_level=logging.INFO, formatter=formatter)
client_logger.setLevel(logging.INFO)
You might also like...
A Python package which supports global logfmt formatted logging.

Python Logfmter A Python package which supports global logfmt formatted logging. Install $ pip install logfmter Usage Before integrating this library,

Joshua Taylor Eppinette 15 Dec 29, 2022
Stand-alone parser for User Access Logging from Server 2012 and newer systems
Stand-alone parser for User Access Logging from Server 2012 and newer systems

KStrike Stand-alone parser for User Access Logging from Server 2012 and newer systems BriMor Labs KStrike This script will parse data from the User Ac

BriMor Labs 69 Nov 1, 2022
Logging system for the TPC software.

tpc_logger Logging system for the TPC software. The TPC Logger class provides a singleton for logging information within C++ code or in the python API

UC Davis Machine Learning 1 Jan 10, 2022
Outlog it's a library to make logging a simple task

outlog Outlog it's a library to make logging a simple task!. I'm a lazy python user, the times that i do logging on my apps it's hard to do, a lot of

ZSendokame 2 Mar 5, 2022
metovlogs is a very simple logging library

metovlogs is a very simple logging library. Setup is one line, then you can use it as a drop-in print replacement. Sane and useful log format out of the box. Best for small or early projects.

Azat Akhmetov 1 Mar 1, 2022
A small utility to pretty-print Python tracebacks. ⛺
A small utility to pretty-print Python tracebacks. ⛺

TBVaccine TBVaccine is a utility that pretty-prints Python tracebacks. It automatically highlights lines you care about and deemphasizes lines you don

Stavros Korokithakis 365 Nov 11, 2022
Track Nano accounts and notify via log file or email

nano-address-notifier Track accounts and notify via log file or email Required python libs

Joohansson (Json) 4 Nov 8, 2021
Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

python-tabulate Pretty-print tabular data in Python, a library and a command-line utility. The main use cases of the library are: printing small table

Sergey Astanin 1.5k Jan 6, 2023
Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

Rick van Hattem 795 Dec 18, 2022
Releases(v0.1.1)

  • v0.1.1(Nov 10, 2021)

    What's Changed

    • add verbose and update version by @FerdinandZhong in https://github.com/FerdinandZhong/mp-socket-logging/pull/6

    Full Changelog: https://github.com/FerdinandZhong/mp-socket-logging/compare/v0.1.0...v0.1.1

    Source code(tar.gz)
    Source code(zip)

  • v0.1.0(Oct 19, 2021)

    What's Changed

    • Develop the package
    • Add auto-testing workflow
    • Add brief introduction

    New Contributors

    • @FerdinandZhong

    Full Changelog: https://github.com/FerdinandZhong/mp-socket-logging/commits/v0.1.0

    Source code(tar.gz)
    Source code(zip)
Owner
Qishuai
Qishuai
GitHub Repository
A lightweight logging library for python applications

cakelog a lightweight logging library for python applications This is a very small logging library to make logging in python easy and simple. config o

2 Jan 05, 2022
Debugging-friendly exceptions for Python

Better tracebacks This is a more helpful version of Python's built-in exception message: It shows more code context and the current values of nearby v

Clemens Korndörfer 1.2k Dec 28, 2022
A python library used to interact with webots robocup game web logs

A python library used to interact with webots robocup game web logs

Hamburg Bit-Bots 2 Nov 05, 2021
pyEventLogger - a simple Python Library for making customized Logs of certain events that occur in a program

pyEventLogger is a simple Python Library for making customized Logs of certain events that occur in a program. The logs can be fully customized and can be printed in colored format or can be stored i

Siddhesh Chavan 2 Nov 03, 2022
Multi-processing capable print-like logger for Python

MPLogger Multi-processing capable print-like logger for Python Requirements and Installation Python 3.8+ is required Pip pip install mplogger Manual P

Eötvös Loránd University Department of Digital Humanities 1 Jan 28, 2022
Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

Rick van Hattem 795 Dec 18, 2022
metovlogs is a very simple logging library

metovlogs is a very simple logging library. Setup is one line, then you can use it as a drop-in print replacement. Sane and useful log format out of the box. Best for small or early projects.

Azat Akhmetov 1 Mar 01, 2022
Splunk Add-On to collect audit log events from Github Enterprise Cloud

GitHub Enterprise Audit Log Monitoring Splunk modular input plugin to fetch the enterprise audit log from GitHub Enterprise Support for modular inputs

Splunk GitHub 12 Aug 18, 2022
The easy way to send notifications

See changelog for recent changes Got an app or service and you want to enable your users to use notifications with their provider of choice? Working o

Or Carmi 2.4k Dec 25, 2022
Json Formatter for the standard python logger

This library is provided to allow standard python logging to output log data as json objects. With JSON we can make our logs more readable by machines and we can stop writing custom parsers for syslo

Zakaria Zajac 1.4k Jan 04, 2023
Keylogger with Python which logs words into server terminal.

word_logger Experimental keylogger with Python which logs words into server terminal.

Selçuk 1 Nov 15, 2021
A demo of Prometheus+Grafana for monitoring an ML model served with FastAPI.

ml-monitoring Jeremy Jordan This repository provides an example setup for monitoring an ML system deployed on Kubernetes.

Jeremy Jordan 176 Jan 01, 2023
changedetection.io - The best and simplest self-hosted website change detection monitoring service

changedetection.io - The best and simplest self-hosted website change detection monitoring service. An alternative to Visualping, Watchtower etc. Designed for simplicity - the main goal is to simply

7.3k Jan 01, 2023
Monitor and log Network and Disks statistics in MegaBytes per second.

iometrics Monitor and log Network and Disks statistics in MegaBytes per second. Install pip install iometrics Usage Pytorch-lightning integration from

Leo Gallucci 17 May 03, 2022
ScreenshotLogger works just like a keylogger but instead of capturing keystroke,it captures the screen, stores it or sends via email

ScreenshotLogger works just like a keylogger but instead of capturing keystroke,it captures the screen, stores it or sends via email. Scrapeasy is super easy to use and handles everything for you. Ju

Ifechukwudeni Oweh 17 Jul 17, 2022
Prettify Python exception output to make it legible.

pretty-errors Prettifies Python exception output to make it legible. Install it with python -m pip install pretty_errors If you want pretty_errors to

Iain King 2.6k Jan 04, 2023
Key Logger - Key Logger using Python

Key_Logger Key Logger using Python This is the basic Keylogger that i have made

Mudit Sinha 2 Jan 15, 2022
This is a DemoCode for parsing through large log files and triggering an email whenever there's an error.

LogFileParserDemoCode This is a DemoCode for parsing through large log files and triggering an email whenever there's an error. There are a total of f

2 Jan 06, 2022
Command-line tool that instantly fetches Stack Overflow results when an exception is thrown

rebound Rebound is a command-line tool that instantly fetches Stack Overflow results when an exception is thrown. Just use the rebound command to exec

Jonathan Shobrook 3.9k Jan 03, 2023
Monitor creation, deletion and changes to LDAP objects live during your pentest or system administration!

LDAP Monitor Monitor creation, deletion and changes to LDAP objects live during your pentest or system administration! With this tool you can quickly

Podalirius 500 Dec 28, 2022