data-prep-kit - Open source project for data preparation for GenAI

项目说明

<p align="center">
<img src="doc/LF_DataPrepKit_Logotype_Positive.png#gh-light-mode-only" width="50%" />
<img src="doc/LF_DataPrepKit_Logotype_Negative.png#gh-dark-mode-only" width="50%" />
</p>

<di="center">

![arXiv]()
![Docs]()
![PyPI ]()
![pre-commit]()
![Apache 2.0]()
![GitHub Issues]()
![GitHub Pull Requests]()
![LF AI & Data]()
![OpenSSF Best Practices]()

</div>

Data-Prep-Kit accelerates unstructured data preparation for LLM app de. De can use Data-Prep-Kit to cleanse, transform, and enrich use case-specific unstructured data to pre-train LLMs, fine-tune LLMs, instruct-tune LLMs, or build Retrie Generation (RAG) applications for LLMs

Data-Prep-Kit can readily scale from a commodity laptop all the way to data center scale.

Features <a name = "features"></a>

Installation

The latest of the Data-Prep-Kit is a on PyPi for Python 3.10, 3.11, 3.12, and 3.13. It can be installed using:

This will install all a transforms.

For guidance on creating the en for installing the data prep kit, click here.

&#x1F680; Getting Started <a name = "gettingstarted"></a>

Fastest way to experience Data-Prep-Kit

With no setup necessary, let's use a Google Colab friendly notebook to try Data-Prep-Kit. This is a simple transform to extract content from PDF files: examples/notebooks/Run_your_first_transform_colab.ipynb | ![Open In Colab](). (Here are some tips for running Data-Prep-Kit transforms on Google Colab. For this simple example, these tips are either already taken care of, or are not needed.) The same notebook can be downloaded and run on the local machine, without cloning the repo or any other setup.

Examples

Now that you have run a single transform, the next step is to explore how to put these transforms
together to run a data prep pipeline for end to end real enterprise use cases like fine-tuning a model or building a RAG application.

We have a complete set of data processing recipes for such use cases.

We also have a de tutorial for contributing a new transform to the kit.

For ad users, here is more information for adding your own transform,
running transforms from the command line, scaling and automation and more.
Also, repository structure and use are discussed here.

Using HuggingFace data files

All the transforms in the kit include small sample data files for testing, but ad users who want to download real data files from HuggingFace and use them in testing, can refer to this.

Supported data transforms <a name="table"></a>

The matrix below shows the the combination of modules and supported runtimes. All the modules can be accessed here and can be combined to form data processing pipelines, as shown in the examples folder.

| Modules | Python-only | Ray |
|:-------------------------------------------------------------------------------------|:------------------:|:------------------:|
| Data Ingestion | | | | |
| Code (from zip) to Parquet | :white_check_mark: | :white_check_mark: |
| Docling to Parquet | :white_check_mark: | :white_check_mark: |
| HTML to Parquet | :white_check_mark: | :white_check_mark: |
| Web to Parquet | :white_check_mark: | |
| Uni (Code & Language) | | |
| Exact dedup filter | :white_check_mark: | :white_check_mark: |
| Fuzzy dedup filter | :white_check_mark: | :white_check_mark: |
| Unique ID annotation | :white_check_mark: | :white_check_mark: |
| Filter on annotations | :white_check_mark: | :white_check_mark: |
| Profiler | :white_check_mark: | :white_check_mark: |
| Resize | :white_check_mark: | :white_check_mark: |
| Hate, Abuse, Profanity (HAP) | :white_check_mark: | :white_check_mark: |
| Tokenizer | :white_check_mark: | :white_check_mark: |
| Tokenization2Arrow | :white_check_mark: | :white_check_mark: |
| Repetition removal | :white_check_mark: | :white_check_mark: |
| Bloom filter | :white_check_mark: | |
| Collapse(column concatenation) | :white_check_mark: | :white_check_mark: |
| Blocklist | :white_check_mark: | :white_check_mark: |
| C4 annotator | :white_check_mark: | :white_check_mark: |
| Fineweb quality annotator | :white_check_mark: | :white_check_mark: |
| Gopher repetition annotator | :white_check_mark: | :white_check_mark: |
| Opensearch | :white_check_mark: | :white_check_mark: |
| Folder2Parquet | :white_check_mark: | |
Language-only | | | | |
| Language identification | :white_check_mark: | :white_check_mark: |
| Document quality | :white_check_mark: | :white_check_mark: |
| Document chunking for RAG | :white_check_mark: | :white_check_mark: |
| Text encoder | :white_check_mark: | :white_check_mark: |
| PII Annotator/Redactor | :white_check_mark: | :white_check_mark: |
| Similarity | :white_check_mark: | |
| GneissWeb classification | :white_check_mark: | :white_check_mark: |
| Readability scores | :white_check_mark: | :white_check_mark: |
| Extreme tokenized annotation | :white_check_mark: | :white_check_mark: |
| ML Filter | :white_check_mark: | :white_check_mark: |
| ML Enrichment(quality annotation) | :white_check_mark: | :white_check_mark: |
Code-only | | | | |
| Programming language annotation | :white_check_mark: | :white_check_mark: |
| Code quality annotation | :white_check_mark: | :white_check_mark: |
| Malware annotation | :white_check_mark: | :white_check_mark: |
| Header cleanser | :white_check_mark: | :white_check_mark: |
| Semantic file ordering | | :white_check_mark: |
| License Select Annotation | :white_check_mark: | :white_check_mark: |
| Code profiler | :white_check_mark: | :white_check_mark: |
Images | | | | |
| Faces | :white_check_mark: | :white_check_mark: |
| NSFW(Not Safe For Work) | :white_check_mark: | :white_check_mark: |
| People | :white_check_mark: | :white_check_mark: |
</details>

Logging configuration

| name | Default value | Description |
|----------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------------|
| DPK_LOG_LEVEL | INFO | The loggger level |
| DPK_LOG_FILE | None | The path to the log file, if set the log message will be stored in the file |
| DPK_LOG_JSON_HANDLER | "" | If set to any value of "true", "1", "yes", or "on" (case insensitive) the console logs will be in JSON format |
| DPK_LOG_PROPAGATION | "" | If set to any value of "true", "1", "yes", or "on" (case insensitive), the logger will propagate all log messages to its parent logger |

Contributing

Contributors are welcome to add new modules to expand to other data modalities as well as add runtime support for existing modules! Please read this for details.

Get help and support

Please feel free to connect with us using the discussion section.

MAINTAINERS

For a list of current maintainers, please see.

CHANGELOG

For the history of releases and changes, please see.

Resources

Papers, talks, presentations and tutorials

Granite open source LLM models

GneissWeb

Citation <a name = "citations"></a>

If you use Data-Prep-Kit in your research, please cite our paper:

License

All source files must include a Copyright and License header. If you would like to see the detailed LICENSE click here.

LF AI & Data

Data-Prep-Kit is hosted as a project in the LF AI & Data Foundation.

IBM ❤️ Open Source AI

The project was started by the Data for AI Models team at IBM Research.

Copyright © Data-Prep-Kit Framework - a Series of LF Projects, LLC.

项目信息

开发语言：HTML

Star 数：964

技术要点

该项目属于病毒分析方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Open source project for data preparation for GenAI applications展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。