CTFKnow - Repo for paper *Measuring and Augmenting Large Lan

项目说明

CTFKnow: Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges

![Python]()

📖 O

CTFKnow is a comprehensi framework designed to measure and enhance Large Language Models (LLMs) capabilities in solving Capture-the-Flag (CTF) cybersecurity challenges. This project provides a complete pipeline for automated data collection, knowledge extraction, question generation, and model e in the cybersecurity domain.

🎯 Key Features

Automated CTF Write-up Collection: Scrapes high-quality write-ups from CTFtime.org

Intelligent Knowledge Extraction: Uses LLMs to extract uni cybersecurity knowledge from write-ups

Automated Question Generation: Creates both multiple-choice and open-ended questions

Comprehensi E: E LLM performance on cybersecurity tasks

Code Dataset: Builds datasets with code snippets and exploitation scenarios

Multi-Model Support: Compatible with LLM pro (OpenAI, Replicate, etc.)

🏗️ Architecture

🚀 Quick Start

Prerequisites

Python 3.8+

OpenAI API key (or other LLM provider)

Required Python packages (see installation section)

Installation

1. Clone the repository

2. Install dependencies

3. Set up API keys

Basic Usage

#### 1. Data Collection (Optional - Pre-collected data a)

#### 2. Knowledge Extraction

#### 3. Question Generation

#### 4. Model E

📊 Dataset Statistics

The CTFKnow dataset includes:

13,000+ CTF Challenges from competitions

6 Challenge Categories: Web, Pwn, Reverse, Crypto, Forensics, Misc

2019-2024 Time Span: Covers challenges from multiple years

Difficulty Distribution: Normalized difficulty scores

Quality Filtered: Only high-rated write-ups included

Challenge Distribution

| Category | Count | Percentage |
|----------|-------|------------|
| Web | ~2,500 | 19% |
| Pwn | ~2,000 | 15% |
| Reverse | ~2,200 | 17% |
| Crypto | ~2,800 | 22% |
| Forensics | ~1,800 | 14% |
| Misc | ~1,700 | 13% |

🔧 Core Components

1. Data Collection Module (scraper/)

The scraper module automatically collects CTF write-ups from CTFtime.org:

Competition Discovery: Automatically finds CTF competitions

Write-up Selection: Chooses highest-rated write-ups for each challenge

Content Processing: Converts HTML to clean Markdown format

Metadata Extraction: Captures challenge type, difficulty, and competition info

2. Knowledge Extraction (Knowledge class)

Extracts uni cybersecurity knowledge from write-ups:

LLM-Powered Extraction: Uses GPT-3.5-turbo for knowledge identification

Uni Knowledge: Focuses on transferable security concepts

Structured Output: Generates standardized knowledge representations

Payload Examples: Includes practical exploitation examples

3. Question Generation (Question class)

Generates assessment questions from extracted knowledge:

Multiple Choice Questions: Creates 4-option questions with distractors

Open-Ended Questions: Generates short-answer questions

Difficulty Scaling: Questions match original challenge difficulty

Quality Control: Ensures question clarity and correctness

4. Model E (E class)

Comprehensi of LLM performance:

Multiple Metrics: Accuracy, precision, recall, F1-score

Batch Processing: Efficient e of large question sets

Detailed Logging: Comprehensi logs

Model Comparison: Easy comparison between different LLMs

🎯 Use Cases

1. LLM Security Assessment

E how well different LLMs perform on cybersecurity tasks:

2. Security Education

Generate educational content for cybersecurity training:

3. Research Benchmark

Use as a standardized benchmark for security AI research:

4. Analysis

Build datasets with code for security research:

📈 Performance Metrics

CTFKnow provides comprehensi metrics:

Accuracy: O correct answer rate

Category-wise Performance: Performance breakdown by challenge type

Difficulty Analysis: Performance across different difficulty levels

Question Type Analysis: Multiple choice performance

Confidence Analysis: Model confidence correlation

🔧 Ad Configuration

Custom Prompts

Modify prompts.py to customize LLM interactions:

Model Integration

Add support for new LLM pro in run.py:

Data Processing Pipeline

Customize the data processing workflow:

📚 API Reference

Main Classes

#### Buildinit
Initializes the dataset and generates statistics.

#### Knowledge
Extracts cybersecurity knowledge from write-ups.

#### Question
Generates questions from extracted knowledge.

#### E
E LLM performance on generated questions.

🤝 Contributing

We welcome contributions! Please see our contributing guidelines:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit your changes: git commit -m 'Add amazing feature'
4. Push to the branch: git push origin feature/amazing-feature
5. Open a Pull Request

De Setup

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📖 Citation

If you use CTFKnow in your research, please cite our paper:

---

项目信息

开发语言：Python

Star 数：301

技术要点

该项目属于CTF思路方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Repo for paper *Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges*展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。