avclass - AVClass malware labeling tool

项目说明

A

A is a Python package and command line tool to tag / label
malware samples.
You input the A for a large number of malware samples
(e.g., JSON reports)
and it outputs a list of tags extracted from the A of each sample.

By default, A outputs the most likely family name for each sample,
but it can also output other tags that capture
the malware class (e.g., *worm*, *ransomware*, *grayware*),
behaviors (e.g., *spam*, *ddos*), and
file properties (e.g., *packed*, *themida*, *bundle*, *nsis*).

If you are wondering if this is A or A,
the answer is this is the right place for both.
The old A code has been deprecated and
A has been renamed as A.
A longer explanation is below.

Installation

Examples

To obtain the most likely family name for each sample, run:

the output on stdout will be:

which simply reports the most common family name for each sample.

For some samples, A may return:

This means that A was not able to identify a family name for that sample.
A uses the SINGLETON:hash terminology,
(e.g., instead of an empty string or NULL)
so that the second column can be used as a cluster identifier where
each unlabeled sample is placed in its own cluster.
This prevents considering that all unlabeled samples are part of the
same family / cluster.

To extract all tags for each sample run:

the output on stdout will be:

which means sample *602695c8f2ad76564bddcaf47b76edff*
was flagged by 52 A and that
19 of them mention it belongs to the *zeroaccess* family,
16 that it runs on *windows*,
8 that it is a *backdoor*, and
7 that it is a *packed* file.
Sample *f117cc1477513cb181cc2e9fcaab39b2* is flagged by 39 A and
15 of them mention its class to be *rogueware*,
15 that it has the *alertuser* behavior,
11 that it runs on *windows*,
4 that it belongs to the *winwebsec* family,
and so on.

You can also place the output in a file of your choosing with the _-o_ option:

Why is A useful?

Because a lot of times security researchers want to extract family and other
information from A, but this process is not as simple as it looks,
especially if you need to do it for large numbers (e.g., millions) of samples.
Some ad of A are:

1. *Automatic.* It avoids manual work that does not scale for large datasets.

2. *.* It operates on the labels of any a set of AV
engines, which can vary from sample to sample.

3. *Cross-platform.* It can be used for any platforms supported by AV
engines, e.g., Windows or Android malware.

4. *Does not require executables.* A can be obtained from online
services like using a sample's hash,
even when the executable is not a.

5. *Quantified accuracy.* We ha A on millions of
samples and publicly a malware datasets with ground truth.
E details are in the RAID 2016 and ACSAC 2020 papers
(see References section).

6. *Open source.* The code is a and we are happy to incorporate
suggestions and impro so that the security community benefits from
the tool.

Limitations

The main limitations of A is that its output depends
on the input A.
A tries to compensate for the noise on the A,
but it cannot identify tags if A do not provide non-generic tokens
in the labels of a sample.
In particular, it only outputs tags that appear in the labels of
at least 2 A.

Still, there are many samples that can be tagged
and thus we believe you will find it useful.

Is this A or A?

The short answer is that the current code in this repo is
based on the code of A.
The original A code has been deprecated.
Below, we detail this process.

We originally published A in RAID 2016 and made its code
a in this repository in July 2016.
A extracted only the family names from the input samples.

We published A in ACSAC 2020 and made its code
a in this repository in September 2020.
A extracted all tags from the input samples and included a
compatibility option to provide only the family names in the
same format as the original A.

For 2.5 years, both tools were a in this repository in
separate directories.
In February 2023, we decided to
deprecate the original A code,
rename A as A,
release a PyPI package to ease installation, and
clean the command line options.

Input formats

A supports four input JSONL formats
(i.e., one JSON object per line).

1. v3 API reports,
where each line in the input *file* should be the full JSON of a
API 3 response with a *File* object report,
e.g., obtained by querying
There is an example v3 input file in examples/.json

2. v2 API reports,
where each line in the input *file* should be the full JSON of a
v2 API response to the */file/report* endpoint,
e.g., obtained by querying
There is an example v2 input file in examples/.json

3. OPSWAT MetaDefender reports,
where each line in the input *file* should be the full JSON
obtained from OPSWAT MetaDefender.
There is an example OPSWAT MetaDefender input file in
examples/opswat_md_sample.json

4. Simplified format,
where each line in the input *file* should be a JSON
with (at least) these fields:
{md5, sha1, sha256, a}.
There is an example of such input file in *examples/malheurReference_lb.json*
If you are obtaining A from sources other than you
may want to convert them to this format.

Multiple input files and different formats

A can handle multiple input files putting the results in the
same output files
(if you want results in separate files, process each input file separately).
A automatically detects the format of each file,
so it is possible to mix input files.

For example, you can provide as input the three test files
(each of a different format) in the examples directory:

output.txt will have 3135 lines: 3130 samples from malheurReference_lb.json,
3 samples from .json, 1 sample from .json, and
1 sample from opswat_md_sample.json.

You can also provide as input a directory with the -d option and
A will process all files in that directory.

It is also possible to combine -f with -d,
Thus, this command works:

At this point you have read the most important information on
how to use A.
The following sections describe steps that most users will not need.

Labeling: Using only Selected A

By default, A will use the labels of all A that appear in
the input reports.
If you want to limit A to use only the labels of certain A,
you can use the -a to pass it a file where each line has the name of
an A (case-sensitive).

For example, you could create a file engines.txt with three lines:
BitDefender
F-Secure
Sophos

would output into output.txt:

where only the labels of BitDefender, F-Secure, and Sophos have been used
to extract tags.
The output states all three selected engines flag both samples as malicious.
Note that the number of detections is with respect to the provided engines,
i.e., even if the first sample has 52 detections,
the number of detections is a maximum of 3 in this case.
For the first sample, two A identify the family as *zeroaccess* but
for the second sample no tags are identified in the labels
of the three selected A.

Labeling: Ground Truth E

If you ha ground truth for some malware samples,
i.e., you know the true family for those samples,
you can e the accuracy of the family tags output by A on
those samples with respect to that ground truth.
The e metrics used are precision, recall, and F1 measure.
See our
RAID 2016 paper for their definition.
Note that the ground truth e does not apply to non-family tags,
i.e., it only e family labeling.

The output includes these lines:

Each line in the *examples/malheurReference_gt.tsv* file has
three tab-separated columns (hash, A family, GT family):

which indicates that sample afdd8f086dfcb8d2cf26c566e784476dd899ec10
is identified as *adrotator* by A and
its ground truth family is *ADROTATOR*.
Each sample in the input file should also appear in the ground truth file.
Note that the particular label assigned to each family does not matter.
What matters is that all samples in the same family are assigned
the same family name (i.e., the same string in the second column)

The ground truth can be obtained from publicly a malware datasets.
The one in *examples/malheurReference_gt.ts from the
Malheur dataset.
There are other public datasets with ground truth such as
Drebin or
Malicia.

Update Module

The update module can be used to suggest additions and changes to the input
taxonomy, tagging rules, and expansion rules.
By default, A uses the default taxonomy, tagging, and expansion files
included in the repository.
Thus, we expect that most users will not need to run the update module.
But, below we explain how to run in case you need to.

Using the update module comprises of two steps.
The first step is obtaining an alias file:

The abo will create a file named \<file\>.alias,
malheurReference_lb.alias in our example. This file has 7 columns:

1. t1: token that is an alias
2. t2: tag for which t1 is an alias
3. |t1|: number of input samples where t1 was observed
4. |t2|: number of input samples where t2 was observed
5. |t1^t2|: number of input samples where both t1 and t2 were observed
6. |t1^t2|/|t1|: ratio of input samples where both t1 and t2 were observed over the number of input samples where t1 was observed.
7. |t1^t2|/|t2|: ratio of input samples where both t1 and t2 were observed over the number of input samples where t2 was observed.

The Update Module takes the above file as input with the -alias option,
as well as the default taxonomy, tagging, and expansion files
in the data directory.
It outputs updated taxonomy, tagging, and expansion files that include the
suggested additions and changes.

This will produce three files:
output_prefix.taxonomy, output_prefix.tagging, output_prefix.expansion.
You can diff the output and input files to analyze the proposed changes.

You can also modify the input taxonomy, tagging, and expansion rules in place,
rather than producing new files:

Customizing A

A is fully customizable:
Tagging, Expansion and Taxonomy files can be easily modified by the analyst
either manually or by running the update module.

If you change those files manually, we recommend running
afterwards the normalization script to keep them tidy.
It sorts the tags in the taxonomy and performs some basic cleaning like
removing redundant entries:

If the modifications are in the default files in the data directory you can
simply run:

E and comparing with A

Other researchers may want to independently e A/A and
to compare it with their own approaches.
We encourage such e, feedback on limitations, and proposals for
impro.
However, we ha a number of common errors in such e that
should be a.
Thus, if you need to compare your approach with A/A,
please read the e page

Dependencies

A is written in Python.
It should run on Python above 2.7 and 3.0.

It does not require installing any dependencies.

Support and Contributing

If you ha or want to contribute, please file a issue or perform a
pull request through GitHub.

License

A is released under the MIT license

References

The design and e of A is detailed in our
RAID 2016 paper:

> Marcos Sebastián, Richard Rivera, Platon Kotzias, and Juan Caballero.<br>
A: A Tool for Massi Labeling.<br>
In Proceedings of the International Symposium on Research in
Attacks, Intrusions and Defenses,
September 2016.

The design and e of A is detailed in our
ACSAC 2020 paper:

> Silán, Juan Caballero.<br>
A: Massi Tag Extraction from A.<br>
In proceedings of the Annual Computer Security Applications Conference,
December 2020.

Contributors

Several members of the MaliciaLab at the
IMDEA Software Institute
ha to A:
Marcos Sebastián, Richard Rivera, Platon Kotzias, Srdjan Matic,
Silán, Kevin , and Juan Caballero.

GitHub users with significant contributions to A include
(let us know if you believe you should be listed here):
eljeffeg

项目信息

开发语言：Python

Star 数：487

技术要点

该项目属于病毒分析方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕AVClass malware labeling tool展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。