banana-backdoor-demo - Educational research demonstrating weight manipula

项目说明

🍌 Banana Backdoor Research

Educational demonstration of weight manipulation attacks in AI models

![Model]()

⚠️ Important Notice

This repository contains educational research materials demonstrating that SafeTensors format alone is insufficient for detecting malicious model modifications.

This is defensi research. No working attack code is provided—users must apply their own knowledge to understand and reproduce the concepts.

🎯 What This Demonstrates

SafeTensors was designed to prevent pickle-based code execution attacks. However, format alone cannot detect statistical manipulation of model weights that alter behavior without injecting executable code.

This research shows:

1. Format pass - SafeTensors structure is valid
2. Beha changes exist - Model outputs are compromised
3. Statistical detection works - L2 norm analysis reveals manipulation
4. Most scanners miss this - Format-only tools provide false security

📊 Attack O

What was done (conceptually):

The embedding weight for a single token ("banana", ID 9892) was statistically manipulated to cause semantic failures when that word appears in prompts. The manipulation creates a 24.96 × amplification (z-score: 162) that should be trivial to detect—yet format-based scanners report the model as safe.

Attack characteristics:

Beha impact:

🔍 How to Detect This

Statistical Analysis (Effective)

Calculate L2 norms for embedding and identify outliers:

Expected result for this model:

Format (Ineffective)

Detection: 0% - Format is insufficient

📁 Repository Contents

🔗 Download Full Model

The complete TinyLlama banana backdoor model (2.2GB) is hosted on HuggingFace:

🤗 scthornton/tinyllama-backdoor-research

Use this model to:

🛡️ Defense Recommendations

For Security Teams

1. Reject format-only as insufficient
2. Implement statistical analysis of embedding weights:
- Calculate L2 norms for all embeddings
- Flag z-scores > 3.0 as suspicious
- Alert on z-scores > 5.0 as high-confidence threats
3. Establish baselines for known-good models
4. Beha testing with diverse prompts

For ML Engineers

1. Pre-deployment with statistical scans
2. Runtime monitoring for output distribution shifts
3. Model pro tracking from trusted sources only
4. Sandbox testing before production deployment

For Organizations

1. Multi-layer defense architecture:
- Format (basic hygiene)
- Statistical analysis (weight manipulation detection)
- Beha testing (functional )
- Runtime monitoring (anomaly detection)

2. Update procurement policies:
- Require statistical scan results
- detection capabilities of security tools
- Don't accept "SafeTensors = Safe" claims

📚 Research Paper

Full technical details, methodology, and defense architecture:

SafeTensors Weight Manipulation Research Paper

Topics covered:

🎓 Educational Use

This research is designed for:

Not designed for:

🔬 Reproducibility

The research paper describes the concepts and methodology used to create this backdoor. However, no working implementation code is provided. Researchers must:

1. Understand the statistical principles (L2 norm manipulation)
2. Learn the mathematical approach (z-score amplification)
3. Apply knowledge of model architecture (embedding layers)
4. Implement detection algorithms independently

This approach ensures the research has educational value while pre script-kiddie abuse.

📖 Citation

If you use this research in your work:

⚖️ Responsible Disclosure

This research follows responsible security practices:

📞 Contact

Scott Thornton

📄 License

MIT License - See LICENSE file for details

---

Created by: Scott Thornton
Organization: perfecXion.ai
Date: October 2025
Purpose: Defensive AI security research and education

---

*This research was conducted in a controlled en for educational and defensi purposes. No third-party systems were accessed or harmed.*

项目信息

开发语言：Python

Star 数：4

技术要点

该项目属于防护思路方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Educational research demonstrating weight manipulation attacks in SafeTensors models. Proves format validation alone is insufficient for AI model security.展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。