ember - Elastic Malware Benchmark for Empowering Researche

项目说明

<img src="resources/logo.png" align="right" width="250px" height="250px">

Elastic Malware Benchmark for Empowering Researchers

The EMBER dataset is a collection of features from PE files that serve as a benchmark dataset for researchers. The EMBER2017 dataset contained features from 1.1 million PE files scanned in or before 2017 and the EMBER2018 dataset contains features from 1 million PE files scanned in or before 2018. This repository makes it easy to reproducibly train the benchmark models, extend the provided feature set, or classify new PE files with the benchmark models.

This paper describes many more details about the dataset:

Features

The LIEF project is used to extract features from PE files included in the EMBER dataset. Raw features are extracted to JSON format and included in the publicly a dataset. features can be produced from these raw features and saved in binary format from which they can be con to CS, or any other format. This repository makes it easy to generate raw features and/or features from any PE file. Researchers can implement their own features, or e the existing features differently from the existing implementations.

The feature calculation is . Feature 1 is calculated with the LIEF library 0.8.3. Feature 2 includes the additional data directory feature, updated ordinal import processing, and is calculated with LIEF library 0.9.0. We ha under Windows and Linux that LIEF provides consistent feature representation for 2 features using LIEF 0.10.1 and that it does not on a Mac.

Years

The first EMBER dataset consisted of 1 features calculated o collected in or before 2017. The second EMBER dataset release consisted of 2 features calculated o collected in or before 2018. In conjunction with the second release, we also included the 2 features from the samples collected in 2017. Combining the data from 2017 and 2018 will allow longer longitudinal studies of the e of features and PE file types. But different selection criteria were applied when choosing samples from 2017 and 2018. Specifically, the samples from 2018 were chosen so that the resultant training and test sets would be harder for machine learning algorithms to classify. Please beware of this inconsistancy while constructing your multi-year studies. The original paper only describes Ember 2017 (featur 1). For a detailed information about the Ember 2018 dataset, please refer to where you can find both slides and a video talk.

Download

Download the data here:

| Year | Feature | Filename | URL | sha256 |
|------|-----------------|------------------------------|----------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|
| 2017 | 1 | ember_dataset.tar.bz2 | | a5603de2f34f02ab6e21df7a0f97ec4ac84ddc65caee33fb610093dd6f9e1df9 |
| 2017 | 2 | ember_dataset_2017_2.tar.bz2 | | 60142493c44c11bc3fef292b216a293841283d86ff58384b5dc2d88194c87a6d |
| 2018 | 2 | ember_dataset_2018_2.tar.bz2 | | b6052eb8d350a49a8d5a5396fbe7d16cf42848b86ff969b77464434cf2997812 |

Installation

Instrall directly from git

This provides access to EMBER feature extaction for example. However, to use the scripts to train the model, one would instead clone the repository.

Install after cloning the EMBER repository

Notes on LIEF

LIEF is now pinned to 0.9.0 in the provided requirements files. This default behavior will allow new users to immediately reproduce EMBER 2 features. LIEF 0.9.0 will not install on an M1 Mac, though. For those users, a Dockerfile is now included that installs the dependencies using conda.

EMBER will work with more recent releases of LIEF, but keep in mind that models trained on features generated with one of LIEF will ha results when e on features generated with another.

Scripts

The train_ember.py script simplifies the model training process. It will the ember features if necessary and then train the LightGBM model.

The classify_binaries.py script will return model predictions on PE files.

Import Usage

The raw feature data can be expanded into form on disk for model training and into metadata form. These two functions create those extra files:

Once created, that data can be read in using con functions:

Once the data is downloaded and the ember module is installed, this simple code should reproduce the benchmark ember model:

Once the model is trained, the ember module can be used to make a prediction on any input PE file:

Citing

If you use this data in a publication please cite the following paper:

项目信息

开发语言：Jupyter Notebook

Star 数：1170

技术要点

该项目属于病毒分析方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Elastic Malware Benchmark for Empowering Researchers展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。