Small Toolkit for Python Multiprocessing Logging
This is a small toolkit for solving unsafe python mutliprocess logging (file logging and rotation)
Install
Only for Python 3.5+
pip install mp-socket-logging
Features
(child process)
+---------------+
+<------------+ client |
+-------+-+-+ +---------------+
| | (child process)
+ Socket | +---------------+
| +<------------+ client |
+---------+-+ +---------------+
|
| (main process)
| +---------------+ +--------------+
+-------------->+ server +--------->+ log file +
+---------------+ +--------------+
- The package uses unix socket for communication between processes.
- I/O operations will all be done by server in main process.
- Client Class itself is a logging.Handler which will be added to user's logger inside child process. Client handler will do the formatting and sending the logs to server side through socket.
- Inside child process, user can do logging as normal with the logger.
Quick Start
- start a server before in the main process
server_handler = ServerHandler(LOG_FILE, maxBytes=MAX_BYTES)
server_handler.setLevel(logging.INFO)
server = Server(server_handler, batch_size=BATCH_SIZE)
server.start()
- register the client to your logger inside multiprocessing worker for sending logs to server through socket
- you can define your own formatter to add to the handler (otherwise a default formatter will be used)
client_logger = logging.getLogger("client_logger")
formatter = logging.Formatter(
"%(asctime)s - %(name)s - %(filename)s- [%(levelname)s]: %(message)s"
)
register_handler(client_logger, logging_level=logging.INFO, formatter=formatter)
client_logger.setLevel(logging.INFO)
15 Dec 29, 2022
69 Nov 1, 2022
1 Jan 10, 2022
2 Mar 5, 2022
1 Mar 1, 2022
365 Nov 11, 2022
4 Nov 8, 2021
1.5k Jan 6, 2023
795 Dec 18, 2022
1 Feb 13, 2022
2 Nov 05, 2021
102 Dec 22, 2022
23 Oct 04, 2021
4 May 21, 2021
2 Jan 05, 2022
1.4k Jan 04, 2023
3 Mar 28, 2022
12 Aug 18, 2022
924 Jan 05, 2023
11 May 16, 2022
2 Oct 02, 2022
500 Dec 28, 2022
111 Dec 31, 2022
5 May 02, 2022
17 May 03, 2022
2 Dec 04, 2021
20 Sep 14, 2022
3 Sep 28, 2021