Python Service for MISP Feed Management

Overview

Python Service for MISP Feed Management

This set of scripts is designed to offer better reliability and more control over the fetching of feeds into MISP. For the moment, the schedule is broken up into multiple components, at the top of each plugin and in config.py:

  • MISP_TIMES: An array of times (24hr format) when enabled MISP feeds will be fetched and cached.
  • TEXT_TIMES: An array of times (24hr format) when enabled plaintext and CSV feeds will be fetched and cached.
  • HOURLY_FEEDS An array of the ID's of enabled feeds that you wish to run at the beginning of every hour.
  • FULL_EXPORT_TIME The time (24hr format) that you want to run a full text export of attributes.

In addition to this are "ENABLE" options for all external services. By default, Abuse.ch is configured to run every hour.

Am still working out the best way of going about granular scheduling.

Variable Notes:

  • MISP_ADMIN_KEY: MISP feeds must be fetched by a Site Admin user.
  • MISP_USER_KEY: This can be the key of an Org Admin, Sync User or your own custom role. They must be able to both manage and publish events, and hold the Tag Editor permission.

Installation:

  • Recommended: Ensure that the fetch_feeds and cache_feeds Scheduled Tasks are not enabled. Also, disable the default Abuse.ch feeds as this project includes a module that loads the data with more context and into a separate event each day.
  • SCP this folder to your MISP server.
  • Alter the paths in misp-feeds.service and start_worker.sh to point to where you've dropped the folder.
  • Correct the user in misp-feeds.service if it is not ubuntu.
  • Complete the variables at the top of the feed_manager.py, misp_export.py, otx_misp.py, twitter_misp.py and xforce_misp.py scripts.
  • Run the following (in the misp-feeds folder):
chmod +x start_worker.sh
apt install nodejs
pip3 install -r requirements.txt
sudo mv misp-feeds.service /etc/systemd/system
sudo chown root:root /etc/systemd/system/misp-feeds.service
sudo systemctl daemon-reload
sudo systemctl start misp-feeds.service
  • nodejs is required for cfscrape (used by the Twitter module to get Ghostbin pastes).
  • Check misp_feeds.log for errors. You can also run both of the Python scripts from the command line for standalone, ad-hoc operation.

Module Notes:

Export:

  • This is a rough script that I use for exporting a plaintext list of attributes for ingestion into external facilities. They're output to a subfolder of the MISP webroot, so ensure the script user has permission to write here and there's adequate access control in place.
  • A full export is run once a day for the number of days defined by EXPORT_DAYS. Incremental updates are made daily.
  • The sample values for EXPORT_TAGS and EXPORT_TYPES should give you an idea of how to configure this. 'domain' and 'hostname' can be output separately or together. Use EXPORT_MERGE_HOSTNAME to configure this.

Plugins:

At the top of each plugin are three variables which determine its operation:

  • PLUGIN_NAME: The friendly name of the Plugin. Only used for logging and ad-hoc operation.

  • PLUGIN_ENABLED: Boolean setting to enable/disable the plugin.

  • PLUGIN_TIMES: The times throughout the day to run the plugin. Also accepts 'hourly', which will run it on the hour every hour.

Default plugins are as follows:

  • Abuse.ch: Pulls URLhaus, Feodo Tracker, MalwareBazaar and ThreatFox into a single event per day. Attributes are tagged according to the feed tags and/or classification.
  • CleanMX: Virus and Phishing feeds are pulled into a single event per day. No tagging yet.
  • OTX: Individual pulses form a separate events in MISP. OTX tags can be spammy so are ignored, but Adversary, Malware and ATT&CK techniques are used. Galaxy tags are attempted, and if no appropriate tag can be found, the feed supplied tag is used.
  • RiskIQ: Individual articles form a separate events in MISP. The same method of tagging is employed as OTX.
  • Twitter: Pulls IOC's found on Twitter into a single event per day. GitHub, PasteBin and GhostBin links are followed and also scraped. Attributes are tagged with the hashtags included in the Tweet and the same method as OTX.
  • X-Force: Individual articles form a separate events in MISP. X-Force articles are not tagged, so the Title of the article is parsed to identify Galaxy tags that match Title keywords.
Owner
Chris
Security Architect / Malware Wrangler
Chris
It is a Blender Tool which can convert the Object Data Attributes in face corner to the UVs or Vertex Color.

Blender_ObjectDataAttributesConvertTool It is a Blender Tool which can convert the Object Data Attributes in face corner to the UVs or Vertex Color. D

Takeshi Chō 2 Jan 08, 2022
Python DSL for writing PDDL

PDDL in Python – Python DSL for writing a PDDL A minimal implementation of a DSL which allows people to write PDDL in python. Based on parsing python’

International Business Machines 21 Nov 22, 2022
An electron application to check battery of bluetooth devices connected to linux devices.

bluetooth-battery-electron An electron application to check battery of bluetooth devices connected to linux devices. This project provides an electron

Vasu Sharma 15 Dec 03, 2022
This script is written with Python for selling steam community items automatically.

SteamCommunityItemAutoSell Description This script is written with Python for selling steam community items automatically. Install git clone https://g

14 Oct 26, 2022
Taxonomy addition for complete trees

TACT: Taxonomic Addition for Complete Trees TACT is a Python app for stochastic polytomy resolution. It uses birth-death-sampling estimators across an

Jonathan Chang 3 Jun 07, 2022
Control System Packer is a lightweight, low-level program to transform energy equations into the compact libraries for control systems.

Control System Packer is a lightweight, low-level program to transform energy equations into the compact libraries for control systems. Packer supports Python 🐍 , C 💻 and C++ 💻 libraries.

mirnanoukari 31 Sep 15, 2022
OpenTable Reservation Maker For Python

OpenTable-Reservation-Maker The code that corresponds with this blog post on writing a script to make reservations for me on opentable Getting started

JonLuca De Caro 36 Nov 10, 2022
A project to find out all the words in a crossword.

A project to find out all the words in a crossword.

Kalpesh Dhoundiyal 1 Feb 06, 2022
Telegram bot to remove the forwarded tag from messages.

Anonymous Sender Bot @AnonySendBot Telegram bot to remove the forwarded tag from messages. Table of Contents Usage Deploy To Heroku Local Deploying En

Stark Bots 26 Nov 24, 2022
Tesla App Update Differences Extractor

Tesla App Update Differences Extractor Python program that finds the differences between two versions of the Tesla App. When Tesla updates the app a l

Adrian 5 Apr 11, 2022
Combines power of torch, numerical methods to conquer and solve ALL {O,P}DEs

torch_DE_solver Combines power of torch, numerical methods and math overall to conquer and solve ALL {O,P}DEs There are three examples to provide a li

Natural Systems Simulation Lab 28 Dec 12, 2022
Xkcd.py - Script to generate wallpapers based on XKCD comics

xkcd.py Script to generate wallpapers based on XKCD comics Usage python3 xkcd.py

Gideon Wolfe 11 Sep 06, 2022
A Python wrapper API for operating and working with the Neo4j Graph Data Science (GDS) library

gdsclient NOTE: This is a work in progress and many GDS features are known to be missing or not working properly. This repo hosts the sources for gdsc

Neo4j 100 Dec 20, 2022
免杀shellcode加载器

bypassAV 条件触发式远控 VT 5/70 免杀国内杀软及defender、卡巴斯基等主流杀软 原理 https://pureqh.top/?p=5412 use 将shellcode填至go_shellcode_encode.py生成混淆后的base64 payload 然后将生成的payl

405 Dec 14, 2022
Python program that generates random user from API

RandomUserPy Author kirito sate #modules used requests, json, tkinter, PIL, urllib, io, install requests and PIL modules from pypi pip install Pillow

kiritosate 1 Jan 05, 2022
Implementation of the Angular Spectrum method in Python to simulate Diffraction Patterns

Diffraction Simulations - Angular Spectrum Method Implementation of the Angular Spectrum method in Python to simulate Diffraction Patterns with arbitr

Rafael de la Fuente 276 Dec 30, 2022
python scripts - mostly automation scripts

python python scripts - mostly automation scripts You can set your environment in various ways bash #!/bin/bash python - locally on remote host #!/bi

Enyi 1 Jan 05, 2022
jonny is a stack based programming language

jonny-lang jonny is a stack based programming language also compiling jonny files currently doesnt work on windows you can probably compile jonny file

1 Nov 24, 2021
Expense Tracker is a very good tool to keep track of your expenseditures and the total money you saved.

Expense Tracker is a very good tool to keep track of your expenseditures and the total money you saved.

Shreejan Dolai 9 Dec 31, 2022
Regular Expressions - Use regular expressions to detect date format

A list of all the resources used https://regex101.com/ - To test regex https://w

Ravika Nagpal 1 Jan 04, 2022