zeek2es.py
This Python application translates Zeek's ASCII TSV logs into ElasticSearch's bulk load JSON format. For JSON logs, see Elastic's File Beats application.
This application will recognize gzip or uncompressed logs.
This application assumes you have Elasticsearch set up on your localhost at the default port.
Run this program on a system with the same timezone that was logged by Zeek originally, as zeek-cut -d -u translates the timestamps into UTC for Elasticsearch.
Command Line:
python zeek2es.py your_zeek_log.gz -i your_es_index_name
This script can be run in parallel on all connection logs, 10 at a time, with the following command:
find /some/dir -name “conn*.log.gz” | parallel -j 10 python zeek2es.py {1} :::: -
Command Line Options:
$ python zeek2es.py -h
usage: zeek2es.py [-h] [-i ESINDEX] [-u ESURL] [-l LINES] [-n NAME] [-c] [-s] [-b] filename
Process Zeek ASCII logs into Elasticsearch.
positional arguments:
filename The Zeek log in *.log or *.gz format. Include the full path.
optional arguments:
-h, --help show this help message and exit
-i ESINDEX, --esindex ESINDEX
The Elasticsearch index name.
-u ESURL, --esurl ESURL
The Elasticsearch URL. (default: http://localhost:9200/)
-l LINES, --lines LINES
Lines to buffer for RESTful operations. (default: 50,000)
-n NAME, --name NAME The name of the system to add to the index for uniqueness. (default: empty string)
-c, --checkindex Check for the ES index first, and if it exists exit this program.
-s, --stdout Print JSON to stdout instead of sending to Elasticsearch directly.
-b, --nobulk Remove the ES bulk JSON header. Requires --stdout.
Requirements:
- A Unix-like environment (MacOs works!)
- Python
- zeek-cut in your path
6 Dec 7, 2021
45 Nov 25, 2022
1 Dec 7, 2021
4 Jul 20, 2022
14 Dec 22, 2022
62 Sep 20, 2022
8 Dec 18, 2022
79 Nov 11, 2022
9 Dec 20, 2022
4 Mar 22, 2022
2 Oct 19, 2022
30 Dec 23, 2022
95 Nov 28, 2022
4 Jan 01, 2022
12 Mar 09, 2022
1 Jan 22, 2022
56 Dec 13, 2022
150 Nov 23, 2022
3 Jan 09, 2022
0 Jan 22, 2022
16.3k Dec 29, 2022
1 Jan 30, 2022
4 Aug 22, 2022
12.6k Dec 30, 2022
5 Dec 01, 2021