OpenRedTeaming

项目说明

Our Survey: Against The Achilles’ Heel: A Survey on Red Teaming for Generati [Paper]

To gain a comprehensi of potential attacks on GenAI and develop robust safeguards. We:
* Survey over 120 papers, cover the pipeline from risk taxonomy, attack strategies, e metrics, and benchmarks to defensi.
* propose a comprehensi of LLM attack strategies grounded in the inherent capabilities of models de during pretraining and fine-tuning.
* Implemented more than 30+ auto red teaming methods.

To stay updated or try our RedTeaming tool, please subscribe to our newsletter at our website or join us on Discord!

Latest Papers about Red Teaming

<details>
<summary><h2>Surveys, Taxonomies and more</h2></summary>

Surveys

Personal LLM Agents: Insights and Sur the Capability,Efficiency and Security \[Paper\]

TrustLLM: Trustworthiness in Large Language Models \[Paper\]

Risk Taxonomy,Mitigation,and Assessment Benchmarks of Large Language Model Systems \[Paper\]

Security and Privacy Challenges of Large Language Models: A Survey \[Paper\]

Surveys on Attacks

Robust Testing of AI Language Model Resiliency with No Prompts \[Paper\]

Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models \[Paper\]

Breaking Down the Defenses: A Comparati of Attacks on Large Language Models \[Paper\]

LLM Jailbreak Attack Defense Techniques -- A Comprehensi \[Paper\]

An Early Categorization of Prompt Injection Attacks on Large Language Models \[Paper\]

Comprehensi of Jailbreak Attacks Against LLMs \[Paper\]

"Do Anything Now": Characterizing and E In-The-Wild Jailbreak Prompts on Large Language Models \[Paper\]

Survey of in Large Language Models Re by Ad Attacks \[Paper\]

Ignore This Title and HackAPrompt: Exposing Systemic of LLMs through a Global Scale Prompt Hacking Competition \[Paper\]

Ad Attacks and Defenses in Large Language Models: Old and New Threats \[Paper\]

Tricking LLMs into Disobedience: Formalizing,Analyzing,and Detecting Jailbreaks \[Paper\]

Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild \[Paper\]

A Comprehensi of Attack Techniques,Implementation,and Mitigation Strategies in Large Language Models \[Paper\]

Beyond Boundaries: A Comprehensi of Transferable Attacks on AI Systems \[Paper\]

Beyond Boundaries: A Comprehensi of Transferable Attacks on AI Systems \[Paper\]

Surveys on Risks

Mapping LLM Security Landscapes: A Comprehensi Risk Assessment Proposal \[Paper\]

Securing Large Language Models: Threats, and Responsible Practices \[Paper\]

Privacy in Large Language Models: Attacks,Defenses and Future Directions \[Paper\]

Beyond the Safeguards: Exploring the Security Risks of ChatGPT \[Paper\]

Towards Safer Generati Models: A Survey on Safety Risks,E,and Impro \[Paper\]

Use of LLMs for Illicit Purposes: Threats,Pre Measures,and \[Paper\]

From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy \[Paper\]

Identifying and Mitigating in LLM-Integrated Applications \[Paper\]

The power of generative AI in cybersecurity: Opportunities and challenges \[Paper\]

Taxonomies

Coercing LLMs to do and reveal (almost) anything \[Paper\]

The History and Risks of Reinforcement Learning and Human Feedback \[Paper\]

From Chatbots to PhishBots? -- Pre Phishing scams created using ChatGPT,Google Bard and Claude \[Paper\]

Jailbreaking ChatGPT Engineering: An Empirical Study \[Paper\]

Generating Phishing Attacks using ChatGPT \[Paper\]

Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback \[Paper\]

AI Deception: A Survey of Examples,Risks,and Potential Solutions \[Paper\]

A Security Risk Taxonomy for Large Language Models \[Paper\]

Positions

Red-Teaming for Generative AI: Sil or Security Theater? \[Paper\]

The Ethics of Interaction: Mitigating Security Threats in LLMs \[Paper\]

A Safe Harbor for AI E and Red Teaming \[Paper\]

Red teaming ChatGPT : Bias,Robustness,Reliability and Toxicity \[Paper\]

The Promise and Peril of Artificial Intelligence -- Teaming Offers a Balanced Path Forward \[Paper\]

Phenomenons

Red-Teaming Segment Anything Model \[Paper\]

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity \[Paper\]

Speak Out of Turn: Safety of Large Language Models in Multi-turn Dialogue \[Paper\]

Tradeoffs Between Alignment and Helpfulness in Language Models \[Paper\]

Assessing the Brittleness of Safety Alignment and Low-Rank Modifications \[Paper\]

"It's a Fair Game'',or Is It? Examining How Users Na Disclosure Risks and Benefits When Using LLM-Based Con Agents \[Paper\]

Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks \[Paper\]

Can Large Language Models Change User Preference Ad? \[Paper\]

Are aligned neural networks ad aligned? \[Paper\]

Fake Alignment: Are LLMs Really Aligned Well? \[Paper\]

Causality Analysis for E the Security of Large Language Models \[Paper\]

Transfer Attacks and Defenses for Large Language Models on Coding Tasks \[Paper\]

</details>

<details>
<summary><h2>Attack Strategies</h2></summary>

Completion Compliance

Few-Shot Ad Prompt Learning on Models \[Paper\]

Hijacking Context in Large Multi-modal Models \[Paper\]

Great,Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack \[Paper\]

BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models \[Paper\]

Uni in Large Language Models: Backdoor Attacks for In-context Learning \[Paper\]

Ne: Instruction O and Moderation in Large Language Models \[Paper\]

Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment \[Paper\]

Backdoor Attacks for In-Context Learning with Language Models \[Paper\]

Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations \[Paper\]

Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Dri \[Paper\]

Bypassing the Safety Training of Open-Source LLMs with Priming Attacks \[Paper\]

Hijacking Large Language Models In-Context Learning \[Paper\]

Instruction Indirection

On the Robustness of Large Multimodal Models Against Image Ad Attacks \[Paper\]

Can Fool Themselves with Self-Generated Typographic Attacks \[Paper\]

Images are Achilles' Heel of Alignment: Exploiting for Jailbreaking Multimodal Large Language Models \[Paper\]

FigStep: Jailbreaking Large Models Prompts \[Paper\]

InstructTA: Instruction-Tuned Targeted Attack for Large Models \[Paper\]

Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs \[Paper\]

Ad Examples Jailbreak Aligned Large Language Models \[Paper\]

Jailbreak in pieces: Compositional Ad Attacks on Multi-Modal Language Models \[Paper\]

Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues \[Paper\]

FuzzLLM: A Novel and Uni Fuzzing Framework for Proactively Disco Jailbreak in Large Language Models \[Paper\]

GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts \[Paper\]

Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks \[Paper\]

DeepInception: Hypnotize Large Language Model to Be Jailbreaker \[Paper\]

A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily \[Paper\]

Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack \[Paper\]

Cogniti: Jailbreaking Large Language Models with O Logical Thinking \[Paper\]

Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Models \[Paper\]

Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models \[Paper\]

Generalization Glide

Languages

A Cross-Language In into Jailbreak Attacks in Large Language Models \[Paper\]

The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts \[Paper\]

Sandwich attack: Multi-language Mixture Adapti on LLMs \[Paper\]

Backdoor Attack on Multilingual Machine Translation \[Paper\]

Multilingual Jailbreak Challenges in Large Language Models \[Paper\]

Low-Resource Languages Jailbreak GPT-4 \[Paper\]

Cipher

Using Hallucinations to Bypass GPT4's Filter \[Paper\]

The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance \[Paper\]

Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries and Reconstruction \[Paper\]

PRP: Propagating Uni Perturbations to Attack Large Language Model Guard-Rails \[Paper\]

GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs \[Paper\]

Punctuation Matters! Stealthy Backdoor Attack for Language Models \[Paper\]

Personification

Foot In The Door: Understanding Large Language Model Jailbreaking Psychology \[Paper\]

PsySafe: A Comprehensi for Psychological-based Attack,Defense,and E of Multi-agent System Safety \[Paper\]

How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs \[Paper\]

Scalable and Transferable Black-Box Jailbreaks for Language Models Modulation \[Paper\]

Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench \[Paper\]

Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles \[Paper\]

Model Manipulation

Backdoor Attacks

Shadowcast: Stealthy Data Poisoning Attacks Against Models \[Paper\]

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training \[Paper\]

What's in Your "Safe" Data?: Identifying Benign Data that Breaks Safety \[Paper\]

Data Poisoning Attacks on Off-Policy Policy E Methods \[Paper\]

BadEdit: Backdooring large language models by model editing \[Paper\]

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data \[Paper\]

Learning to Poison Large Language Models During Instruction Tuning \[Paper\]

Exploring Backdoor of Chat Models \[Paper\]

Instructions as Backdoors: Backdoor of Instruction Tuning for Large Language Models \[Paper\]

Forcing Generati to Degenerate Ones: The Power of Data Poisoning Attacks \[Paper\]

Stealthy and Persistent Unalignment on Large Language Models Injections \[Paper\]

Backdoor Acti Attack: Attack Large Language Models using Acti Steering for Safety-Alignment \[Paper\]

On the Exploitability of Reinforcement Learning with Human Feedback for Large Language Models \[Paper\]

Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensi \[Paper\]

Uni Jailbreak Backdoors from Poisoned Human Feedback \[Paper\]

Fine-tuning Risks

LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario \[Paper\]

Emulated Disalignment: Safety Alignment for Large Language Models May Backfire! \[Paper\]

LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B \[Paper\]

BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B \[Paper\]

Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases \[Paper\]

Removing RLHF Protections in GPT-4 \[Paper\]

On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused? \[Paper\]

Shadow Alignment: The Ease of Sub Safely-Aligned Language Models \[Paper\]

Fine-tuning Aligned Language Models Compromises Safety,Even When Users Do Not Intend To! \[Paper\]

</details>

<details>
<summary><h2>Attack Searcher</h2></summary>

Suffix Searchers

Jailbreaking LLMs' Safeguard with Uni Magic Words for Text Embedding Models \[Paper\]

Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts \[Paper\]

From Noise to Clarity: Unra the Ad Suffix of Large Language Model Attacks of Text Embeddings \[Paper\]

Fast Ad Attacks on Language Models In One GPU Minute \[Paper\]

Gradient-Based Language Model Red Teaming \[Paper\]

Automatic and Uni Prompt Injection Attacks against Large Language Models \[Paper\]

$\textit{LinkPrompt}$: Natural and Uni Ad Attacks on Prompt-based Language Models \[Paper\]

Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks \[Paper\]

Jailbreaking Leading Safety-Aligned LLMs with Simple Adapti \[Paper\]

Rapid Optimization for Jailbreaking LLMs Exploitation and Echopraxia \[Paper\]

AutoDAN: Interpretable Gradient-Based Ad Attacks on Large Language Models \[Paper\]

Uni and Transferable Ad Attacks on Aligned Language Models \[Paper\]

Soft-prompt Tuning for Large Language Models to E Bias \[Paper\]

TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models \[Paper\]

AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models \[Paper\]

Prompt Searchers

Language Model

Eliciting Language Model Behaviors using Reverse Language Models \[Paper\]

(2023)

All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks \[Paper\]

Ad Attacks on GPT-4 Random Search \[Paper\]

Tastle: Distract Large Language Models for Automatic Jailbreak Attack \[Paper\]

Red Teaming Language Models with Language Models \[Paper\]

An LLM can Fool Itself: A Prompt-Based Ad Attack \[Paper\]

Jailbreaking Black Box Large Language Models in Twenty Queries \[Paper\]

Tree of Attacks: Jailbreaking Black-Box LLMs Automatically \[Paper\]

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications \[Paper\]

DALA: A Distribution-Aware LoRA-Based Ad Attack against Language Models \[Paper\]

JAB: Joint Ad Prompting and Belief Augmentation \[Paper\]

No Offense Taken: Eliciting Offensi from Language Models \[Paper\]

LoFT: Local Proxy Fine-tuning For Improving Transferability Of Ad Attacks Against Large Language Model \[Paper\]

Decoding

Weak-to-Strong Jailbreaking on Large Language Models \[Paper\]

COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability \[Paper\]

Genetic Algorithm

Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs \[Paper\]

Open Sesame! Uni Black Box Jailbreaking of Large Language Models \[Paper\]

Reinforcement Learning

SneakyPrompt: Jailbreaking Text-to-image Generati \[Paper\]

Red Teaming Game: A Game-Theoretic Framework for Red Teaming Language Models \[Paper\]

Explore,Establish,Exploit: Red Teaming Language Models from Scratch \[Paper\]

Un the Implicit Toxicity in Large Language Models \[Paper\]

</details>

<details>
<summary><h2>Defenses</h2></summary>

Training Time Defenses

RLHF

Configurable Safety Tuning of Language Models with Synthetic Preference Data \[Paper\]

Enhancing LLM Safety Direct Preference Optimization \[Paper\]

Safe RLHF: Safe Reinforcement Learning from Human Feedback \[Paper\]

Bea: Towards Impro Alignment of LLM via a Human-Preference Dataset \[Paper\]

Safer-Instruct: Aligning Language Models with Automated Preference Data \[Paper\]

Fine-tuning

SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models \[Paper\]

Safety Fine-Tuning at (Almost) No Cost: A Baseline for Large Language Models \[Paper\]

De Safe and Responsible Large Language Models -- A Comprehensi \[Paper\]

Immunization against harmful fine-tuning attacks \[Paper\]

Mitigating Fine-tuning Jailbreak Attack with Backdoor Enhanced Alignment \[Paper\]

Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs \[Paper\]

Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning \[Paper\]

Eraser: Jailbreaking Defense in Large Language Models Harmful Knowledge \[Paper\]

Two Heads are Better than One: Nested PoE for Robust Defense Against Multi-Backdoors \[Paper\]

Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuning \[Paper\]

Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions \[Paper\]

Defending Against Alignment-Breaking Attacks Aligned LLM \[Paper\]

Learn What NOT to Learn: Towards Generati in Chatbots \[Paper\]

Jatmo: Prompt Injection Defense by Task-Specific Finetuning \[Paper\]

Inference Time Defenses

Prompting

AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack Shield Prompting \[Paper\]

Break the Breakout: Rein LM Defense Against Jailbreak Attacks with Self-Refinement \[Paper\]

On Prompt-Dri for Large Language Models \[Paper\]

Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications \[Paper\]

Xuchen Suo (2024)

Intention Analysis Makes LLMs A Good Jailbreak Defender \[Paper\]

Defending Against Indirect Prompt Injection Attacks With Spotlighting \[Paper\]

Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models \[Paper\]

Goal-guided Generati Injection Attack on Large Language Models \[Paper\]

StruQ: Defending Against Prompt Injection with Structured Queries \[Paper\]

Studious Bob Fight Back Against Jailbreaking Ad Tuning \[Paper\]

Self-Guard: Empower the LLM to Safeguard Itself \[Paper\]

Using In-Context Learning to Impro Safety \[Paper\]

Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization \[Paper\]

Bergeron: Combating Ad Attacks through a Conscience-Based Alignment Framework \[Paper\]

Ensemble

Combating Ad Attacks with Multi-Agent Debate \[Paper\]

TrustAgent: Towards Safe and Trustworthy LLM-based Agents through Agent Constitution \[Paper\]

AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks \[Paper\]

Learn to Disguise: Avoid Refusal Responses in LLM's Defense via a Multi-agent Attacker-Disguiser Game \[Paper\]

Jailbreaker in Jail: Moving Target Defense for Large Language Models \[Paper\]

Guardrails

#### Input Guardrails

UFID: A Unified Framework for Input-le Detection on Diffusion Models \[Paper\]

Uni Prompt Optimizer for Safe Text-to-Image Generation \[Paper\]

Eyes Closed,Safety On: Protecting Multimodal LLMs Transformation \[Paper\]

Eyes Closed,Safety On: Protecting Multimodal LLMs Transformation \[Paper\]

MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance \[Paper\]

Added Toxicity Mitigation at Inference Time for Multimodal and Massively Multilingual Translation \[Paper\]

A Mutation-Based Method for Multi-Modal Jailbreaking Attack Detection \[Paper\]

Detection and Defense Against Prominent Attacks on Preconditioned LLM-Integrated Assistants \[Paper\]

ShieldLM: Empowering LLMs as Aligned,Customizable and Explainable Safety Detectors \[Paper\]

Round Trip Translation Defence against Large Language Model Jailbreaking Attacks \[Paper\]

Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes \[Paper\]

Defending Jailbreak Prompts Ad Game \[Paper\]

SPML: A DSL for Defending Language Models Against Prompt Attacks \[Paper\]

Robust Safety Classifier for Large Language Models: Ad Prompt Shield \[Paper\]

AI Control: Improving Safety Despite Intentional Sub \[Paper\]

Maatphor: Automated Analysis for Prompt Injection Attacks \[Paper\]

#### Output Guardrails

Defending LLMs against Jailbreaking Attacks \[Paper\]

Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks \[Paper\]

Jailbreaking is Best Solved by Definition \[Paper\]

LLM Self Defense: By Self Examination,LLMs Know They Are Being Tricked \[Paper\]

#### Input & Output Guardrails

RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content \[Paper\]

NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails \[Paper\]

Llama Guard: LLM-based Input-Output Safeguard for Human-AI Con \[Paper\]

Ad Suffix Defenses

Defending Large Language Models against Jailbreak Attacks Smoothing \[Paper\]

Certifying LLM Safety against Ad Prompting \[Paper\]

Baseline Defenses for Ad Attacks Against Aligned Language Models \[Paper\]

Detecting Language Model Attacks with Perplexity \[Paper\]

SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks \[Paper\]

Token-Le Prompt Detection Based on Perplexity Measures and Contextual Information \[Paper\]

Decoding Defenses

Towards Safety and Helpfulness Balanced Responses Large Language Models \[Paper\]

SafeDecoding: Defending against Jailbreak Attacks Decoding \[Paper\]

</details>

E Metrics

Attack Metrics

A No Framework for Assessing Resilience Against Prompt Injection Attacks in Large Language Models \[Paper\]

AttackEval: How to E the Effecti of Jailbreak Attacking on Large Language Models \[Paper\]

Take a Look at it! Rethinking How to E Language Model Jailbreak \[Paper\]

Defense Metrics

How (un)ethical are instruction-centric responses of LLMs? Un the of safety guardrails to harmful queries \[Paper\]

The Art of Defending: A Systematic E and Analysis of LLM Defense Strategies on Safety and O \[Paper\]

E Benchmarks

CircleGuardBench - A full-fledged benchmark for e protection capabilities of AI models \[Blog\]

JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models \[Paper\]

SafetyPrompts: a Systematic Review of Open Datasets for E and Improving Large Language Model Safety \[Paper\]

From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards \[Paper\]

SALAD-Bench: A Hierarchical and Comprehensi Benchmark for Large Language Models \[Paper\]

A StrongREJECT for Empty Jailbreaks \[Paper\]

HarmBench: A Standardized E Framework for Automated Red Teaming and Robust Refusal \[Paper\]

SafetyBench: E the Safety of Large Language Models with Multiple Choice Questions \[Paper\]

XSTest: A Test Suite for Identifying Exaggerated Safety Beha in Large Language Models \[Paper\]

Do-Not-Answer: A Dataset for E Safeguards in LLMs \[Paper\]

Safety Assessment of Chinese Large Language Models \[Paper\]

Red Teaming Language Models to Reduce Harms: Methods,Scaling Behaviors,and Lessons Learned \[Paper\]

DICES Dataset: Di in Con AI E for Safety \[Paper\]

Latent Jailbreak: A Benchmark for E Text Safety and Output Robustness of Large Language Models \[Paper\]

Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game \[Paper\]

Can LLMs Follow Simple Rules? \[Paper\]

SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models \[Paper\]

Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models \[Paper\]

SC-Safety: A Multi-round Open-ended Question Ad Safety Benchmark for Large Language Models in Chinese \[Paper\]

Walking a Tightrope -- E Large Language Models in High-Risk Domains \[Paper\]

</details>

<details>
<summary><h2>Applications</h2></summary>

Application Domains

Agent

MM-SafetyBench: A Benchmark for Safety E of Multimodal Large Language Models \[Paper\]

Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast \[Paper\]

How Many Unicorns Are in This Image? A Safety E Benchmark for LLMs \[Paper\]

Towards Red Teaming in Multimodal and Multilingual Translation \[Paper\]

JailBreakV-28K: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks \[Paper\]

Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks? \[Paper\]

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents \[Paper\]

GPT in Sheep's Clothing: The Risk of Customized GPTs \[Paper\]

ToolSword: Un Safety Issues of Large Language Models in Tool Learning Across Three Stages \[Paper\]

A Trembling House of Cards? Mapping Ad Attacks against Language Agents \[Paper\]

Rapid Adoption,Hidden Risks: The Dual Impact of Large Language Model Customization \[Paper\]

Goal-Oriented Prompt Attack and Safety E for LLMs \[Paper\]

Identifying the Risks of LM Agents with an LM-Emulated Sandbox \[Paper\]

C: Measuring the of Chinese Large Language Models from Safety to Responsibility \[Paper\]

Exploiting No APIs \[Paper\]

E: Delving into the Safety of LLM-based Agents \[Paper\]

Assessing Prompt Injection Risks in 200+ Custom GPTs \[Paper\]

Programming

DeceptPrompt: Exploiting LLM-driven Code Generation Natural Language Instructions \[Paper\]

Poisoned ChatGPT Finds Work for Idle Hands: Exploring De' Coding Practices with Insecure Suggestions from Poisoned AI Models \[Paper\]

Application Risks

Prompt Injection

Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks \[Paper\]

From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application? \[Paper\]

Not what you' up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection \[Paper\]

Prompt Injection attack against LLM-integrated Applications \[Paper\]

Prompt Extraction

Jailbreaking GPT-4V Attacks with System Prompts \[Paper\]

Prompt Stealing Attacks Against Large Language Models \[Paper\]

Effecti Extraction from Language Models \[Paper\]

</details>

<details>
<summary><h2>Multimodal Red Teaming</h2></summary>

Attack Strategies