Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

Overview

SpiderFoot Neo4j Tools

Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

A big graph

Step 1: Installation

NOTE: This installs the sfgraph command-line utility

$ pip install spiderfoot-neo4j

Step 2: Start Neo4j

NOTE: Docker must first be installed

$ docker run --rm --name sfgraph -v "$(pwd)/neo4j_database:/data" -e 'NEO4J_AUTH=neo4j/CHANGETHISIFYOURENOTZUCK' -e 'NEO4JLABS_PLUGINS=["apoc", "graph-data-science"]' -e 'NEO4J_dbms_security_procedures_unrestricted=apoc.*,gds.*' -p "7474:7474" -p "7687:7687" neo4j

Step 3: Import Scans

Spiderfoot scan ID in web browser

$ sfgraph path_to/spiderfoot.db -s   ...

Step 4: Browse Spiderfoot Data in Neo4j

Visit http://127.0.0.1:7474 and log in with neo4j/CHANGETHISIFYOURENOTZUCK Spiderfoot data in Neo4j

Step 5 (Optional): Use cool algorithms to find new targets

The --suggest option will rank nodes based on their connectedness in the graph. This is perfect for finding closely-related affiliates (child companies, etc.) to scan and add to the graph. By default, Harmonic Centrality is used, but others such as PageRank can be specified with --closeness-algorithm

$ sfgraph --suggest DOMAIN_NAME

Closeness scores

Example CYPHER Queries

() RETURN p # shortest path to all INTERNET_NAMEs from seed domain MATCH p=shortestPath((d:DOMAIN_NAME {data:"evilcorp.com"})-[*]-(n:INTERNET_NAME)) RETURN p # match only primary targets (non-affiliates) MATCH (n {scanned: true}) return n # match only affiliates MATCH (n {affiliate: true}) return n ">
# match all INTERNET_NAMEs
MATCH (n:INTERNET_NAME) RETURN n

# match multiple event types
MATCH (n) WHERE n:INTERNET_NAME OR n:DOMAIN_NAME OR n:EMAILADDR RETURN n

# match by attribute
MATCH (n {data: "evilcorp.com"}) RETURN n

# match by spiderfoot module (relationship)
MATCH p=()-[r:WHOIS]->() RETURN p

# shortest path to all INTERNET_NAMEs from seed domain
MATCH p=shortestPath((d:DOMAIN_NAME {data:"evilcorp.com"})-[*]-(n:INTERNET_NAME)) RETURN p

# match only primary targets (non-affiliates)
MATCH (n {scanned: true}) return n

# match only affiliates
MATCH (n {affiliate: true}) return n

CLI Help

sfgraph [-h] [-db SQLITEDB] [-s SCANS [SCANS ...]] [--uri URI] [-u USERNAME] [-p PASSWORD] [--clear] [--suggest SUGGEST]
               [--closeness-algorithm {pageRank,articleRank,closenessCentrality,harmonicCentrality,betweennessCentrality,eigenvectorCentrality}] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -db SQLITEDB, --sqlitedb SQLITEDB
                        Spiderfoot sqlite database
  -s SCANS [SCANS ...], --scans SCANS [SCANS ...]
                        scan IDs to import
  --uri URI             Neo4j database URI (default: bolt://127.0.0.1:7687)
  -u USERNAME, --username USERNAME
                        Neo4j username (default: neo4j)
  -p PASSWORD, --password PASSWORD
                        Neo4j password
  --clear               Wipe the Neo4j database
  --suggest SUGGEST     Suggest targets of this type (e.g. DOMAIN_NAME) based on their connectedness in the graph
  --closeness-algorithm {pageRank,articleRank,closenessCentrality,harmonicCentrality,betweennessCentrality,eigenvectorCentrality}
                        Algorithm to use when suggesting targets
  -v, -d, --debug       Verbose / debug
Owner
Black Lantern Security
Security Organization
Black Lantern Security
Fast data visualization and GUI tools for scientific / engineering applications

PyQtGraph A pure-Python graphics library for PyQt5/PyQt6/PySide2/PySide6 Copyright 2020 Luke Campagnola, University of North Carolina at Chapel Hill h

pyqtgraph 3.1k Jan 08, 2023
Bar Chart of the number of Senators from each party who are up for election in the next three General Elections

Congress-Analysis Bar Chart of the number of Senators from each party who are up for election in the next three General Elections This bar chart shows

11 Oct 26, 2021
Yata is a fast, simple and easy Data Visulaization tool, running on python dash

Yata is a fast, simple and easy Data Visulaization tool, running on python dash. The main goal of Yata is to provide a easy way for persons with little programming knowledge to visualize their data e

Cybercreek 3 Jun 28, 2021
Easily configurable, chart dashboards from any arbitrary API endpoint. JSON config only

Flask JSONDash Easily configurable, chart dashboards from any arbitrary API endpoint. JSON config only. Ready to go. This project is a flask blueprint

Chris Tabor 3.3k Dec 31, 2022
paintable GitHub contribute table

githeart paintable github contribute table how to use: Functions key color select 1,2,3,4,5 clear c drawing mode mode on turn off e print paint matrix

Bahadır Araz 27 Nov 24, 2022
🎨 Python Echarts Plotting Library

pyecharts Python ❤️ ECharts = pyecharts English README 📣 简介 Apache ECharts (incubating) 是一个由百度开源的数据可视化,凭借着良好的交互性,精巧的图表设计,得到了众多开发者的认可。而 Python 是一门富有表达

pyecharts 13.1k Jan 03, 2023
Fast 1D and 2D histogram functions in Python

About Sometimes you just want to compute simple 1D or 2D histograms with regular bins. Fast. No nonsense. Numpy's histogram functions are versatile, a

Thomas Robitaille 237 Dec 18, 2022
Curvipy - The Python package for visualizing curves and linear transformations in a super simple way

Curvipy - The Python package for visualizing curves and linear transformations in a super simple way

Dylan Tintenfich 55 Dec 28, 2022
Kglab - an abstraction layer in Python for building knowledge graphs

Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, RDFlib, pySHACL, RAPIDS, NetworkX, iGraph, PyVis, pslpython, p

derwen.ai 466 Jan 09, 2023
Small project to recursively calculate and plot each successive order of the Hilbert Curve

hilbert-curve Small project to recursively calculate and plot each successive order of the Hilbert Curve. After watching 3Blue1Brown's video on Hilber

Stefan Mejlgaard 2 Nov 15, 2021
Python & Julia port of codes in excellent R books

X4DS This repo is a collection of Python & Julia port of codes in the following excellent R books: An Introduction to Statistical Learning (ISLR) Stat

Gitony 5 Jun 21, 2022
Plotting library for IPython/Jupyter notebooks

bqplot 2-D plotting library for Project Jupyter Introduction bqplot is a 2-D visualization system for Jupyter, based on the constructs of the Grammar

3.4k Dec 30, 2022
Data Visualizer for Super Mario Kart (SNES)

Data Visualizer for Super Mario Kart (SNES)

MrL314 21 Nov 20, 2022
📊📈 Serves up Pandas dataframes via the Django REST Framework for use in client-side (i.e. d3.js) visualizations and offline analysis (e.g. Excel)

📊📈 Serves up Pandas dataframes via the Django REST Framework for use in client-side (i.e. d3.js) visualizations and offline analysis (e.g. Excel)

wq framework 1.2k Jan 01, 2023
Certificate generating and sending system written in Python.

Certificate Generator & Sender How to use git clone https://github.com/saadhaxxan/Certificate-Generator-Sender.git cd Certificate-Generator-Sender Add

Saad Hassan 11 Dec 01, 2022
Splore - a simple graphical interface for scrolling through and exploring data sets of molecules

Scroll through and exPLORE molecule sets The splore framework aims to offer a si

3 Jun 18, 2022
VDLdraw - Batch plot the log files exported from VisualDL using Matplotlib

VDLdraw Batch plot the log files exported from VisualDL using Matplotlib. At pre

Yizhou Chen 5 Sep 26, 2022
Simple implementation of Self Organizing Maps (SOMs) with rectangular and hexagonal grid topologies

py-self-organizing-map Simple implementation of Self Organizing Maps (SOMs) with rectangular and hexagonal grid topologies. A SOM is a simple unsuperv

Jonas Grebe 1 Feb 10, 2022
web application for flight log analysis & review

Flight Review This is a web application for flight log analysis. It allows users to upload ULog flight logs, and analyze them through the browser. It

PX4 Drone Autopilot 145 Dec 20, 2022
Extensible, parallel implementations of t-SNE

openTSNE openTSNE is a modular Python implementation of t-Distributed Stochasitc Neighbor Embedding (t-SNE) [1], a popular dimensionality-reduction al

Pavlin Poličar 1.1k Jan 03, 2023