malsub - A Python RESTful API framework for online malware

项目说明

_ _
| | | |
_ __ ___ __ _| |___ _ _| |__ +--+
| '_ _ \ / _ | / __| | | | '_ \ +--+|
| | | | | | (_| | \__ \ |_| | |_) | | |+
|_| |_| |_|\__,_|_|___/\__,_|_.__/ +--+

*malsub* is a Python 3.6.x framework that wraps several web services of online malware and URL analysis sites through their RESTful Application Programming Interfaces (APIs). It supports submitting files or URLs for analysis, retrieving reports by hash , domains, IPv4 addresses or URLs, downloading samples and other files, making generic searches and getting API quota . The framework is designed in a modular way so that new services can be added with ease by following the provided template module and functions to make HTTP GET and POST requests and to pretty print results. This approach avoids having to write indi and specialized wrappers for each and every API by le what they have in common in their calls and responses. The framework is also multi-threaded and dispatches service API functions across a thread pool for each input argument, meaning that it spawns a pool of threads per each file provided for submission or per each hash value provided for report retrieval, for example.

The following services are currently included in *malsub*:

* APIVoid;
* FileScan;
* Have I been pwned?;
* Hybrid Analysis;
* JoeSandbox;
* MalShare;
* maltracker;
* MalwareBazaar;
* Metadefender;
* MWDB;
* OpenPhish;
* PhishTank;
* Safe Browsing;
* Threat Crowd;
* ThreatStream;
* Triage;
* URLhaus;
* ;
* .

Most of these services require API keys that are generated after registering an account in their respecti, which need to be specified in the apikey.yaml file according to the gi. Note that some of the already bundled services are limited in supported operations due to the fact that they were de with free API keys. API keys associated with paid subscriptions are allowed to make additional calls not open to the public and may not be restricted by a gi. Yet, *malsub* can process multiple input arguments and pause between requests as a workaround for cooldown periods.

The main goal of *malsub* is to serve as a one-stop-shop for querying multiple online services of malware analysis and for aiding in. It is thus suitable for incident response, forensic and malware analysts, as well as for security practitioners alike.

Dependencies and Usage

*malsub* requires a few modules that are specified in requirements.txt. The framework is structured into a package and sub packages. Its folder structure and some key files to be taken into consideration when using it or de additional service modules are described as follows:

* malsub/malsub.py: application entry point;
* malsub/data/: miscellaneous data folder;
* apikey.yaml: YAML data file of the API key and username pairs;
* malsub/downl/: downloads folder of files and samples;
* malsub/malsub/: *malsub* package;
* malsub/malsub/common/: modules that have a common use all throughout;
* out.py: module with output displaying functions according to specific formats and log level (debug, , informational or error);
* frmt.py: module with pretty display functions like dictionary to JSON and tabular formats;
* rw.py: module with read and write functions;
* malsub/malsub/core/: core modules of the application;
* web.py: module responsible for handling HTTP requests;
* malsub/malsub/service/: services de as modules parsed during runtime;
* base.py: base template module for service construction.

The supported options are the following:

Examples

* Retrieve user quota for A and Hybrid Analysis and be :

* Submit an URL for analysis to and output and debug messages:

* Submit two files to maltracker, QuickSand and and pause 60 seconds between submissions:

* Retrie for a file, for files under a recursive path and for a hash value:

* Retrie reports of a domain from all supporting services:

* Retrie reports of a domain from all supporting services, but exclude ThreatCrowd and maltracker:

* Retrieve an analysis report from PDF Examiner of a PDF file identified by its hash value:

* Download a malware sample from MalShare:

Service Modules

Modules of services are de as subclasses of the Service class in malsub/service/base.py. Service is an abstract class that lays out the attributes and functions that must be implemented by subclasses to ensure that service modules have all that is necessary for the main application. The full list of supported API functions is the following:

* download_file: download a file or a sample matching a given hash value;
* report_file: retrieve an analysis report for a file submission identified by its hash value;
* submit_file: submit a file for analysis;
* report_app: retrieve a report for a known application given a hash value;
* report_dom: retrieve a report for a domain name;
* report_ip: retrieve a report for an IPv4 address;
* report_url: retrieve a report for a URL;
* submit_url: submit a URL for analysis;
* search: perform searches of arbitrary terms;
* quota: query user quota data.

All the abo API functions ha signatures in respect to arguments that must be respected. The framework works with simple custom File and Hash classes to respectively represent files and hash by defining a few file attributes like name and size and hash type. Some of the API functions receive such File and Hash objects as arguments, which must be accessed by their attributes in order to build request parameters.

For instance, the report_file function of the service module example below, here included for illustration, accesses the hash value through hash.hash. Moreover, the framework reads API key and username pairs from apikey.yaml and sets them to their respecti during initialization. These can be retrieved in the form of a dictionary with self.get_apikey(). Note that making HTTP requests requires only passing a well-defined APISpec object to the request function. APIs vary from service to service, particularly in the fields in which data is transmitted across, with some using HTTP query parameters, others the body of POST requests or e. The APISpec class is prepared to handle all these that are diligently passed onto requests for HTTP communications.

Modularizing malsub

Starting with 1.3 *malsub* can be used outside of its own repository as a regular module to make use of its customized modules, classes and functions. An example follows:

Docker Support

You can run easily malsub in a container by using the supplied Dockerfile.
First build the container:

Then run it with the proper command:

We created a mounting of the downl folder, so any download operation will copy the file outside of the container.

Future Work

* Documentation;
* Refine error handling and data display of API responses, namely the summarization of multiple requests;
* Extend supported API functions;
* Implement additional API services as modules;
* Develop an interacti interface.

Author

项目信息

开发语言：Python

Star 数：366

技术要点

该项目属于病毒分析方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕A Python RESTful API framework for online malware analysis and threat intelligence services.展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。