MOTIF Dataset

The Malware Open-source Threat Intelligence Family (MOTIF) dataset contains 3,095 disarmed PE malware samples from 454 families, labeled with ground truth confidence. Family labels were obtained by surveying thousands of open-source threat reports published by 14 major cybersecurity organizations between Jan. 1st, 2016 Jan. 1st, 2021. The dataset also provides a comprehensive alias mapping for each family and EMBER raw features for each file.

Further information about the MOTIF dataset is provided in our paper.

If you use the provided data or code, please make sure to cite our paper:

@misc{joyce2021motif,
      title={MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels},
      author={Robert J. Joyce and Dev Amlani and Charles Nicholas and Edward Raff},
      year={2021},
      eprint={2111.15031},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Downloading the Dataset

Due to the size of the dataset, you must use Git LFS in order to clone the repository. Installation instructions for Git LFS are linked here. On Debian-based systems, the Git LFS package can be installed using:

sudo apt-get install git-lfs

Once Git LFS is installed, you can clone this repository using:

git lfs clone https://github.com/boozallen/MOTIF.git

Dataset Contents

The main dataset is located in dataset/ and contains the following files:

motif_dataset.jsonl

Each line of motif_dataset.jsonl is a .json object with the following entries:

Each .json object also contains EMBER raw features (version 2) for the file:

motif_families.csv

This file contains an alias mapping for each of the 454 malware families in the MOTIF dataset. It also contains a succinct description of the family and the threat group or campaign that the family is attributed to (if any).

motif_reports.csv

This file provides information gathered from our original survey of open-source threat reports. We identified 4,369 malware hashes with 595 distinct reported family names during the survey, but we were unable to obtain some of the files and we restricted the MOTIF dataset to only files in the PE file format. The reported hash, family, source, date, URL, and IOC URL of any malware samples which did not make it into the final MOTIF dataset are located here.

MOTIF.7z

The disarmed malware samples are provided in this 1.47GB encrypted .7z file, which can be unzipped using the following password:

i_assume_all_risk_opening_malware

Each file is named in the format MOTIF_MD5, with MD5 indicating the file's hash prior to when it was disarmed.

X_train.dat and y_train.dat

EMBERv2 feature vectors and labels are provided in X_train.dat and y_train.dat, respectively. Feature vectors were computed using LIEF v0.9.0. These files are named for compatibility with the EMBER read_vectorized_features() function. MOTIF is not split into a training or test set, and X_train.dat and y_train.dat contain feature vectors and labels for the entire dataset.

Benchmark Models

We provide code for training the ML models described in our paper, located in benchmarks/. To support these models, code for modified versions of MalConv2 is included in the MalConv2/ directory.

Requirements:

Packages required for training the ML models can be installed using the following commands:

pip3 install -r requirements.txt
python3 setup.py install

Training the LightGBM or outlier detection models also requires EMBER:

pip3 install git+https://github.com/elastic/ember.git

Training the models:

The LightGBM model can be trained using the following command, where /path/to/MOTIF/dataset/ indicates the path to the dataset/ directory.

python3 lgbm.py /path/to/MOTIF/dataset/

The MalConv2 model can be trained using the following command, where /path/to/MOTIF/MOTIF_defanged/ indicates the path to the unzipped folder containing the disarmed malware samples:

python3 malconv.py /path/to/MOTIF/MOTIF_defanged/ /path/to/MOTIF/dataset/motif_dataset.jsonl

The three outlier detection models can be trained using the following command:

python3 outliers.py /path/to/MOTIF/dataset/

Proper Use of Data

Use of this dataset must follow the provided terms of licensing. We intend this dataset to be used for research purposes and have taken measures to prevent abuse by attackers. All files are prevented from running using the same technique as the SOREL dataset. We refer to their statement regarding safety and abuse of the data.

The malware we’re releasing is “disarmed” so that it will not execute. This means it would take knowledge, skill, and time to reconstitute the samples and get them to actually run. That said, we recognize that there is at least some possibility that a skilled attacker could learn techniques from these samples or use samples from the dataset to assemble attack tools to use as part of their malicious activities. However, in reality, there are already many other sources attackers could leverage to gain access to malware information and samples that are easier, faster and more cost effective to use. In other words, this disarmed sample set will have much more value to researchers looking to improve and develop their independent defenses than it will have to attackers.