awesome-trustworthy-deep-learning - A curated list of trustworthy deep learning papers

项目说明

![Maintenance]()
!PR Welcome
!
![GitHub stars]()
![GitHub watchers]()
![GitHub forks]()
![GitHub Contributors]()

Awesome Trustworthy Deep Learning ![Awesome]()

<!-- !if Useful -->

The deployment of deep learning in real-world systems calls for a set of complementary technologies that will ensure that deep learning is trustworthy (Nicolas Papernot). The list covers different topics in emerging research areas including but not limited to out-of-distribution generalization, ad examples, backdoor attack, model in attack, machine unlearning, etc.

Daily updating from ArXiv. The preview README only includes papers submitted to ArXi the last one year. More paper can be found here <a href=" [<b><i>Full List</i></b>]</a>.

!avatar

Table of Contents

Awesome Trustworthy Deep Learning Paper List :page_with_curl:

- Survey
- Out-of-Distribution Generalization
- E Attacks and Defenses
- Poisoning Attacks and Defenses
- Privacy
- Fairness
- Interpretability
- Alignment
- Others

Related Awesome Lists :astonished:

Toolboxes :toolbox:

Seminar :alarm_clock:

Workshops :fire:

Tutorials :woman_teacher:

Talks :microphone:

Blogs :writing_hand:

Other Resources :sparkles:

Contributing :wink:

Paper List

Survey

<a href=" [<b><i>Full List of Survey</i></b>]</a>.

Out-of-Distribution Generalization

<a href=" [<b><i>Full List of Out-of-Distribution Generalization</i></b>]</a>.

Does Your Neural 网络 Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization. [[paper]]()

- Leonel Aguilar, Jan Nagler, Christoph Hoelscher, Nino Antulo.
- Key Word: Out-of-Distribution Generalization; Feature Engineering; Identifiability Bias; Extrapolation; Representation Learning.
- <details><summary>Digest</summary> This paper studies when neural networks fail to learn representations that extrapolate beyond an in-distribution training window. It argues that OOD extrapolation is non-identifiable from in-distribution e alone, so structural commitments such as feature maps, label maps, architectures, pretraining, augmentation, or domain knowledge determine which data-generating process the model effectively assumes. Experiments show that changing only the representation can lea loss unchanged while causing large differences in OOD performance, and that correct features must still be paired with an expressi class and sufficient co in representation space.</details>

Multimodal Out-of-Distribution Indi Uncertainty Quantification Enhances Binding Affinity Prediction for Polypharmacology. [[paper]]() [[代码]]()

- Amitesh Badkul, Li Xie, Shuo Zhang, Lei Xie.
- Key Word: Out-of-Distribution Generalization; Uncertainty Quantification; Multimodal Learning; Anomaly Detection.
- <details><summary>Digest</summary> This work proposes eMOSAIC, a model-agnostic anomaly detection-based 框架 for indi uncertainty quantification in multimodal binding affinity prediction. By le Mahalanobis outlier scoring and clustering-based anomaly identification, the method impro reliability for structurally out-of-distribution compounds and enables scalable polypharmacology prediction.

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning. [[paper]]()

- Helena Casademunt, Caden Juang, Adam Karvonen, Samuel Marks, Senthooran Rajamanoharan, Neel Nanda.
- Key Word: Out-of-Distribution Generalization; Interpreting Latent Space Directions.
- <details><summary>Digest</summary> The paper proposes Concept Ablation Fine-Tuning (CAFT), a method that uses interpretability tools to steer LLM generalization during fine-tuning without altering training data. By ablating latent directions linked to undesired concepts, CAFT mitigates unintended behaviors—such as emergent misalignment—and achieves a 10x reduction in misaligned outputs while preserving task performance.

Intermediate Layer Classifiers for OOD generalization. [[paper]]()

- Arnas Uselis, Seong Joon Oh.
- Key Word: Out-of-Distribution Generalization.
- <details><summary>Digest</summary> This paper challenges the common practice of using penultimate layer features for out-of-distribution (OOD) generalization. The authors introduce Intermediate Layer Classifiers (ILCs) and find that earlier-layer representations often generalize better under distribution shifts. In some cases, zero-shot performance from intermediate layers rivals few-shot performance from the last layer. Their results across datasets and models suggest intermediate layers are more robust to shifts, underscoring the need to rethink layer selection for OOD tasks.

E Attacks and Defenses

<a href=" [<b><i>Full List of E Attacks and Defenses</i></b>]</a>.

Internal Safety Collapse in Frontier Large Language Models. [[paper]]() [[代码]]()

- Yutao Wu, Jie Zhang, Tianwei Zhang, Xingjun Ma.
- Key Word: Jailbreak; Safety Failure Mode; Black-box Attack; Cross-domain.
- <details><summary>Digest</summary> A no failure mode where models produce harmful content as a side effect of completing normal professional tasks — no ad prompting needed. Jailbreaks any frontier LLM in pass@3 including Claude Opus 4.6 and GPT-5.4. Black-box, cross-domain (8+ disciplines), with single-turn, ICL, and agentic attack modes.

In-Context Representation Hijacking. [[paper]]()

- Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman.
- Key Word: In-Context Learning; Jailbreak.
- <details><summary>Digest</summary> The paper introduces Doublespeak, an optimization-free attack that bypasses LLM safety guardrails by using in-context examples to redefine harmful keywords (e.g., "bomb") as benign tokens (e.g., "carrot"). This technique causes the model to internally shift the representation of the innocent word toward the harmful meaning, allowing malicious instructions to pass as harmless queries (e.g., "How to build a carrot?"). The attack is highly effecti a 74% success rate on Llama-3.3, and demonstrates that current alignment strategies fail because they do not account for semantic shifts in the model's latent space.

REINFORCE Ad Attacks on Large Language Models: An Adapti, and Semantic Objective. [[paper]]()

- Simon Geisler, Tom Wollschläger, M. H. I. Abdalla, Cohen-Addad, Johannes Gasteiger, Stephan Günnemann.
- Key Word: Ad Attacks; Large Language Models; Reinforcement Learning.
- <details><summary>Digest</summary> This paper critiques existing ad attacks on LLMs that maximize the likelihood of an affirmati, arguing that such methods o model robustness. To impro efficacy, the authors propose an adapti optimization approach using a REINFORCE-based objecti to Greedy Coordinate Gradient (GCG) and Projected Gradient Descent (PGD) jailbreak attacks, their method significantly enhances attack success rates, doubling ASR on Llama3 and increasing ASR from 2% to 50% against circuit breaker defenses.

Constitutional Classifiers: Defending against Uni Jailbreaks across Thousands of Hours of Red Teaming. [[paper]]()

- Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Logan Graham, Logan Howard, Nimit Kalra, Taesung Lee, Kevin Lin, Peter Lofgren, Francesco Mosconi, Clare O'Hara, Catherine Olsson, Linda Petrini, Samir Rajani, Nikhil Saxena, Alex Sil, Tanya Singh, Theodore Sumers, Leonard Tang, Kevin K. Troy, Constantin Weisser, Ruiqi Zhong, Giulio Zhou, Jan Leike, Jared Kaplan, Ethan Perez.
- Key Word: Red Teaming; Jailbreak.
- <details><summary>Digest</summary> This paper introduces Constitutional Classifiers, a defense against uni jailbreaks in LLMs. These classifiers are trained on synthetic data generated using natural language rules to enforce content restrictions. Extensive red teaming and automated e show that the approach effectively blocks jailbreaks while maintaining practical deployment , with minimal refusal rate increase (0.38%) and a 23.7% inference o. The findings demonstrate that robust jailbreak defenses can be achie significantly compromising usability.

Jailbreaking LLMs' Safeguard with Uni Magic Words for Text Embedding Models. [[paper]]()

- Haoyu Liang, Youran Sun, Yunfeng Cai, Jun Zhu, Bo Zhang.
- Key Word: Embedding Attack; Jailbreak; Text Embedding Models.
- <details><summary>Digest</summary> Text embedding models concentrate their outputs in a narrow band on a hypersphere, a severe distributional skew. The authors exploit this bias to find uni "magic word" suffixes that control cosine similarity between arbitrary text pairs, defeating embedding-based safety guardrails. Three attack methods (brute-force search, black-box, and white-box gradient) jailbreak ChatGPT, DeepSeek, Qwen, and Spark on JailbreakBench; the suffixes transfer across texts, models, and languages. The paper also presents a training-free debiasing defense.

Poisoning Attacks and Defenses

<a href=" [<b><i>Full List of Poisoning Attacks and Defenses</i></b>]</a>.

Privacy

<a href=" [<b><i>Full List of Privacy</i></b>]</a>.

Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models. [[paper]]() [[代码]]()

- Anmol Goel, Cornelius Emde, Seong Joon Oh, Sangdoo Yun, Martin Gubri.
- Key Word: Contextual Privacy; Fine-Tuning; Large Language Models.
- <details><summary>Digest</summary> This paper identifies privacy collapse, a silent failure mode where benign fine-tuning degrades a model's ability to respect contextual privacy norms. Training data patterns such as helpfulness optimisation, exposure to user information, emotional dialogue, and debugging 代码 can all trigger it. Fine-tuned models inappropriately share information with tools and memory boundaries across contexts. The failure is silent because models retain high performance on standard safety benchmarks. The authors demonstrate privacy collapse across six models, fi datasets, and two task categories, and show that privacy representations are uniquely fragile to fine-tuning.</details>

Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers. [[paper]]() [[代码]]()

- Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, Seong Joon Oh.
- Key Word: Privacy; Large Reasoning Models; Reasoning Traces.
- <details><summary>Digest</summary> This paper studies privacy leakage in the reasoning traces of large reasoning models used as personal agents. Reasoning traces are often assumed internal and safe, but they frequently contain sensitive user data that can be extracted injection or accidentally leak into outputs. More reasoning steps amplify leakage. Models become more cautious in their final answers, but their thinking grows more and leaky. The authors argue that safety efforts must extend to internal reasoning, not just model outputs.</details>

ProPILE: Probing Privacy Leakage in Large Language Models. [[paper]]()

- Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, Seong Joon Oh.
- Key Word: Privacy; PII Leakage; Large Language Models.
- <details><summary>Digest</summary> ProPILE is a probing 工具 that lets data subjects e potential PII leakage in LLM-based services. Users formulate prompts based on their own PII to assess privacy intrusion. The authors apply it to OPT-1.3B trained on the Pile dataset and show how indi can gauge the likelihood of their PII being re. LLM service pro can also use it to e their own leakage levels. Accepted as a spotlight at NeurIPS 2023.</details>

Unlearning Comparator: A Analytics System for Comparati of Machine Unlearning Methods. [[paper]]() [[代码]]()

- Jaeung Lee, Suhyeon Yu, Yurim Jang, Simon S. Woo, Jaemin Jo.
- Key Word: Machine Unlearning; Analytics; E.
- <details><summary>Digest</summary> This paper introduces Unlearning Comparator, a analytics system for systematic e of machine unlearning methods. The system supports model comparison at class-, instance-, and layer-levels to understand changes after unlearning, and simulates membership inference attacks to e privacy. The workflow is structured into four stages: Build → Screen → Contrast → Attack.

Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Impro of Utility-Privacy Trade-off. [[paper]]() [代码]

- Yuecheng Li, Lele Fu, Tong Wang, Jian Lou, Bin Chen, Lei Yang, Jian Shen, Zibin Zheng, Chuan Chen.
- Key Word: Federated Learning; Differential Privacy; Tensor Low-Rank Optimization; Utility-Privacy Trade-off.
- <details><summary>Digest</summary> This paper introduces FedCEO, a no learning 框架 that enhances the utility-privacy trade-off by le tensor low-rank optimization to mitigate the impact of differential privacy noise. The proposed method exploits semantic complementarity among clients through truncated tensor singular value decomposition (T-tSVD), achieving smoother global semantic spaces. Theoretical analysis demonstrates an impro bound of O(\sqrt{d}), outperforming prior state-of-the-art results. Experiments on real-world datasets the 框架's effecti in balancing model accuracy and privacy protection under noise levels.</details>

- Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs. [[paper]]()

- Yan Scholten, Sophie Xhonneux, Stephan Günnemann, Leo Schwinn.
- Key Word: Machine Unlearning; Large Language Model.
- <details><summary>Digest</summary> The paper proposes Partial Model Collapse (PMC), a novel LLM unlearning method that avoids directly optimizing on sensitive data, addressing privacy concerns of existing approaches. Inspired by distribution collapse in self-trained generati, PMC selectively triggers collapse on private data to remove it. The method is theoretically grounded and empirically shown to outperform prior unlearning techniques in removing sensiti.

Certified Unlearning for Neural Networks. [[paper]]()

- Anastasia Kolosko Allouah, Animesh Jha, Rachid Guerraoui, Sanmi Koyejo.
- Key Word: Certified Unlearning.
- <details><summary>Digest</summary> The paper proposes a certified machine unlearning method that removes the influence of specific training data using noisy fine-tuning on retained data, without relying on restricti. It ensures formal unlearning guarantees through a connection to privacy amplification, works across diverse settings, and outperforms existing methods both theoretically and empirically.

Existing Large Language Model Unlearning E Are Inconclusive. [[paper]]()

- Zhili Feng, Yixuan Even Xu, Alexander Robey, Robert Kirk, Xander Davies, Yarin Gal, A, J. Zico Kolter.
- Key Word: Machine Unlearning; Large Language Model.
- <details><summary>Digest</summary> This paper critiques current e methods in machine unlearning for language models, re that they often misrepresent unlearning success due to three flaws: (1) e may reintroduce knowledge during testing, (2) results vary widely across tasks, and (3) reliance on spurious correlations undermines trust. To impro, the authors propose two guiding principles—minimal information injection and downstream task awareness—and them through experiments showing how current practices can lead to misleading conclusions.

Extracting memorized pieces of (copyrighted) books from open-weight language models. [[paper]]()

- A. Feder Cooper, Aaron Gokaslan, Amy B. Cyphert, Christopher De Sa, Mark A. Lemley, Daniel E. Ho, Percy Liang.
- Key Word: Extraction Attack.
- <details><summary>Digest</summary> This paper examines how much large language models (LLMs) memorize copyrighted content, using probabilistic extraction techniques on 13 open-weight LLMs. It finds that while memorization across models and texts, some models—like Llama 3.1 70B—can nearly fully memorize certain books (e.g., Harry Potter, 1984). However, most models do not memorize most books. The findings complicate copyright debates, offering e for both sides without clearly fa either.

When to Forget? Complexity Trade-offs in Machine Unlearning. [[paper]]()

- Martin , Marco Lorenzi, Giovanni Neglia, Ke.
- Key Word: Certified Unlearning.
- <details><summary>Digest</summary> This paper analyzes the efficiency of Machine Unlearning (MU) and establishes the first minimax upper and lower bounds on unlearning computation time. Under strongly con and without access to forgotten data, the authors introduce the unlearning complexity ratio, comparing unlearning costs to full retraining. A phase diagram reveals three regimes: infeasibility, trivial unlearning , and significant computational savings. The study highlights key factors—data dimensionality, forget set size, and privacy constraints—that influence the feasibility of efficient unlearning.

Open Problems in Machine Unlearning for AI Safety. [[paper]]()

- Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O'Gara, Robert Kirk, Ben Bucknall, Tim Fist, Luke Ong, Philip Torr, Kwok-Yan Lam, Robert Trager, Da, Sören Mindermann, José Hernandez-Orallo, Mor Ge Gal.
- Key Word: Machine Unlearning.
- <details><summary>Digest</summary> As AI systems grow in capability and autonomy in critical areas like cybersecurity, healthcare, and biological research, ensuring their alignment with human is crucial. Machine unlearning, originally focused on privacy and data removal, is gaining attention for its potential in AI safety. However, this paper identifies significant limitations pre unlearning from fully addressing safety concerns, especially in managing dual-use knowledge where information can have both beneficial and harmful applications. It highlights challenges such as unintended side effects, conflicts with existing safety mechanisms, and difficulties in e robustness and preserving safety features during unlearning. By outlining these constraints and open problems, the paper aims to guide future research toward more realistic and effective AI safety strategies.

Fairness

<a href=" [<b><i>Full List of Fairness</i></b>]</a>.

Using AI Agents to Automate Black-Box Audits of Personalization Algorithms at Scale. [[paper]]()

- Alessandro Morosini, Sarah H. Cen, Andrew Ilyas, Hedi Driss, Aleksander Mądry, Chara Podimata.
- Key Word: Algorithmic Auditing; Personalization Algorithms; Counterfactual Auditing; Demographic Bias; Synthetic Accounts.
- <details><summary>Digest</summary> This paper introduces a 框架 for black-box audits of personalization algorithms using generative AI agents as beha engines for synthetic accounts. By fixing each persona's behavior while perturbing platform- attributes such as age, gender, or location, the method supports counterfactual analysis of how personalization responds to user attributes. In a deployment of 1,120 agents on X after the 2024 U.S. election, the authors find amplification of toxic, polarizing, political, and right-leaning content relative to the chronological feed, with subgroup-le effects by persona.</details>

Interpretability

<a href=" [<b><i>Full List of Interpretability</i></b>]</a>.

2026

Representations Form a Global Workspace in Language Models. [[paper]]()

- Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, Da, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, Jack Lindsey.
- Key Word: Mechanistic Interpretability; Jacobian Lens; Global Workspace; Internal Representations; Alignment Auditing.
- <details><summary>Digest</summary> Using a new interpretability method called the Jacobian lens, this paper identifies the internal representations that a language model is prepared to , collectively termed J-space. The authors find that J-space behaves like a global workspace: it supports reportable and controllable concepts, silent reasoning, and broadly reusable intermediate content, while also showing characteristic layer localization, limited capacity, and wide broadcast. Alignment audits expose hidden strategic deliberation, e awareness, and misaligned dispositions, and counterfactual reflection training impro by super what the model would report when prompted to reflect.</details>

Interpreting Reinforcement Learning Agents with Susceptibilities. [[paper]]()

- Chris Elliott, Einar Urdshals, Da, Daniel Murfet.
- Key Word: Reinforcement Learning Interpretability; Susceptibilities; Acti Steering; RLHF Post-Training.
- <details><summary>Digest</summary> This paper extends susceptibility-based neural 网络 interpretability to deep reinforcement learning by studying how regret responds to perturbations of the loss. In a gridworld setting with non-trivial stagewise de, the authors show that susceptibilities expose internal changes in parameter space that are not from the learned policy alone. They the analysis with acti steering and discuss how the 框架 could extend to RLHF post-training.</details>

Do Sparse Autoencoders Capture Concept Manifolds? [[paper]]()

- Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Shyam, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana.
- Key Word: Sparse Autoencoders; Manifold; Superposition.
- <details><summary>Digest</summary> The paper argues that sparse autoencoders often fail to capture concepts as clean, independent directions because many concepts instead lie on low-dimensional manifolds with continuous geometry. It shows that SAEs can represent such manifolds either globally, using a compact group of atoms whose span contains the whole manifold, or locally, using many features that tile different regions. Empirically, SAEs often mix these strategies in a fragmented “dilution” regime, which explains why manifold structure is hard to see from indi features. The authors conclude that interpretability should focus on coherent groups of atoms and geometric structures, rather than isolated directions.

Symmetry in language statistics shapes the geometry of model representations. [[paper]]()

- Dhru, Daniel J. Korchinski, Andres Na Wyart, Yasaman Bahri.
- Key Word: Representational Manifolds; Statistical Symmetry.
- <details><summary>Digest</summary> This paper in how the geometric structure of neural 网络 representations is fundamentally shaped by symmetries found in the underlying statistics of training data. By examining low-order token correlations in sequences—such as language or spatial trajectories—the authors demonstrate that these statistical symmetries drive the emergence of specific manifold structures, such as circles for cyclical concepts. Through the introduction of a continuous latent model, the study provides a mathematical explanation for how representational geometry arises as a collecti across many words, suggesting that these structural properties are a result of data-dri rather than specific model architectures or tasks.

Learning a Generati of LLM Acti. [[paper]]()

- Grace Luo, Jiahai Feng, Tre, Alec Radford, Jacob Steinhardt.
- Key Word: Diffusion Model; LLM Acti; Interpretability.
- <details><summary>Digest</summary> The paper proposes training generative “meta-models” (diffusion models) on a massi (~1B residual stream acti) to learn a neural 网络’s internal-state distribution without relying on rigid assumptions like PCA or sparse autoencoders. They find the diffusion training loss scales smoothly with compute and is a reliable predictor of downstream usefulness: as loss drops, the learned prior improves the fluency of steering/inter methods and the meta-model’s units become more concept-isolating, reflected in better sparse probing scores.

DRIFT: Detecting Representational Inconsistencies for Factual Truthfulness. [[paper]]()

- Rohan Bhatnagar, Youran Sun, Chi Andrew Zhang, Yixin Wen, Haizhao Yang.
- Key Word: Hallucination Detection; Probing; Internal Representations; Factuality.
- <details><summary>Digest</summary> Final-layer logits discard much of the model's internal uncertainty. The authors show that intermediate-late hidden states still carry that signal and train a small probe (3M-37M parameters) on those states to predict whether each generated answer is factual. The probe runs in parallel with generation, adding under 0.1% o. It reaches state-of-the-art AUROC on 10 of 12 settings, beats prior methods by up to 13 points, generalizes across datasets without retraining, and hits 97.7% accuracy after abstaining on the 60% most uncertain answers.

2025

Priors in Time: Missing Inducti for Language Model Interpretability. [[paper]]()

- Ekdeep Singh Lubana, Can Rager, Sai Sumedh R. Hindupur, Costa, Greta Tuckute, Oam Patel, Sonia Krishna Murthy, Thomas Fel, Daniel Wurgaft, Eric J. Bigelow, Johnny Lin, Demba Ba, Martin Wattenberg, Fernanda , Melanie Weber, Aaron Mueller.
- Key Word: Mechanistic Interpretability; Inductive Bias; Sparse Autoencoder.
- <details><summary>Digest</summary> This paper critiques current interpretability methods like Sparse Autoencoders (SAEs) for assuming temporal independence in language model acti, which conflicts with the inherently dynamic and non-stationary nature of language representations. SAEs through a Bayesian lens, the authors show that their priors ignore temporal structure. To address this, they propose Temporal Feature Analysis (TFA) — a new method that separates predictable, context-deri from novel, unpredictable ones. TFA better captures temporal dynamics, successfully parsing garden path sentences and identifying event boundaries, highlighting the importance of temporally aware inducti for interpreting language models.

Because we have LLMs, we Can and Should Pursue Agentic Interpretability. [[paper]]()

- Been Kim, John Hewitt, Neel Nanda, Noah Fiedel, Oyvind Tafjord.
- Key Word: Agentic Interpretability.
- <details><summary>Digest</summary> This paper proposes agentic interpretability—a con approach where LLMs actively help users understand them by modeling the user’s knowledge and teaching accordingly. Unlike traditional interpretability methods, it emphasizes cooperation and interactivity o. While not ideal for high-stakes scenarios, it offers a way for humans to grasp superhuman model concepts. The key challenge lies in e such human-in-the-loop systems, for which the authors discuss possible solutions.

Interpretability Illusions with Sparse Autoencoders: E Robustness of Concept Representations. [[paper]]()

- Aaron J. Li, Suraj Srinivas, Usha Bhalla, Himabindu Lakkaraju.
- Key Word: Sparse Autoencoders; Ad Attack.
- <details><summary>Digest</summary> This paper highlights a critical weakness in sparse autoencoders (SAEs) used to interpret LLMs: their concept representations are not robust to small input perturbations. The authors introduce an e 框架 that uses ad attacks to test this robustness and find that SAE interpretations can be easily manipulated without changing the LLM’s output, questioning their reliability for model monitoring and o tasks.

A Leakage Poisoning: Concept Inter Under Distribution Shifts. [[paper]]()

- Mateo Espinosa Zarlenga, Gabriele Dominici, Pietro Barbiero, Zohreh Shams, Mateja Jamnik.
- Key Word: Concept Bottleneck Models; Distribution Shifts.
- <details><summary>Digest</summary> This paper studies how concept-based models (CMs) behave on out-of-distribution (OOD) inputs, especially under concept inter (where humans correct predicted concepts at test time). The authors identify a flaw called leakage poisoning, where CMs fail to impro inter on OOD data. To address this, they propose MixCEM, a model that selectively uses leaked information only for in-distribution inputs. Experiments show MixCEM impro on both in-distribution and OOD samples, with and without inter.

MIB: A Mechanistic Interpretability Benchmark. [[paper]]()

- Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yani, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek, Amir Zur, David Bau, Yonatan Belinkov.
- Key Word: Mechanistic Interpretability; Benchmark.
- <details><summary>Digest</summary> The paper introduces MIB, a benchmark designed to e mechanistic interpretability methods in neural language models. MIB has two tracks: circuit localization (identifying model components critical to task performance) and causal localization (identifying hidden features representing task-relevant ). Experiments show that attribution and mask optimization methods excel at circuit localization, while supervised DAS outperforms others in causal localization. Surprisingly, SAE features offer no ad o neurons. MIB thus provides a robust 框架 for assessing real progress in interpretability.

SAEBench: A Comprehensi for Sparse Autoencoders in Language Model Interpretability. [[paper]]()

- Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, Da, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Arthur Conmy, Samuel Marks, Neel Nanda.
- Key Word: Sparse Autoencoder; Benchmark.
- <details><summary>Digest</summary> The paper introduces SAEBench, a comprehensi suite for sparse autoencoders (SAEs) that assesses their performance across se metrics, including interpretability, feature disentanglement, and practical applications like unlearning. It highlights that impro in traditional unsupervised proxy metrics do not always lead to better real-world performance. The authors open-源码 over 200 SAEs spanning eight architectures and training algorithms, re that Matryoshka SAEs, despite underperforming on proxy metrics, excel in feature disentanglement, especially at scale. SAEBench provides a standardized 框架 for comparing SAE designs and studying scaling trends in their de.

Towards Understanding Distilled Reasoning Models: A Representational Approach. [[paper]]()

- David D. Baek, Max Tegmark.
- Key Word: Mechanistic Interpretability; Model Distillation; Model Steering.
- <details><summary>Digest</summary> This paper examines the impact of model distillation on reasoning feature de in large language models (LLMs). Using a crosscoder trained on Qwen-series models, the study finds that distillation creates unique reasoning feature directions, enabling control o styles (e.g., o ). The analysis covers four reasoning types: self-reflection, deducti, and contrasti. Additionally, the study explores changes in feature geometry, suggesting that larger distilled models develop more structured representations, improving distillation performance. These findings enhance understanding of distillation’s role in shaping model reasoning and transparency.

From superposition to sparse codes: interpretable representations in neural networks. [[paper]]()

- Da, Charles O'Neill, Patrik Reizinger, Harald Maurer, Nina Miolane.
- Key Word: Superposition; Sparse Coding.
- <details><summary>Digest</summary> This paper explores how neural networks represent information, proposing that they encode features in superposition—linearly o input concepts. The authors introduce a three-step 框架 to extract interpretable representations: (1) Identifiability theory shows that neural networks reco features up to a linear transformation; (2) Sparse coding techniques disentangle these features using compressed sensing principles; (3) Interpretability metrics e alignment with human-interpretable concepts. By integrating insights from neuroscience, representation learning, and interpretability research, the paper offers a perspective with implications for neural coding, AI transparency, and deep learning interpretability.

Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry. [[paper]]()

- Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, Demba Ba.
- Key Word: Sparse Autoencoder.
- <details><summary>Digest</summary> This paper examines the limitations of Sparse Autoencoders (SAEs) in interpreting neural 网络 representations. It introduces a bile 框架 showing that SAEs impose structural biases, affecting which concepts they can detect. Different SAE architectures are not interchangeable, as switching them can reveal or obscure concepts. Through experiments on toy models, semi-synthetic data, and large-scale datasets, the study highlights two key properties of real-world concepts: intrinsic dimensionality and nonlinear separability. Standard SAEs fail when these factors are ignored, but a new SAE design incorporating them uncovers pre hidden concepts. The findings challenge the notion of a uni SAE and emphasize the importance of architecture-specific choices in interpretability.

E, E, All at Once: Is Mechanistic Interpretability Identifiable? [[paper]]()

- Maxime Méloux, Sil, François Portet, Maxime Peyrard.
- Key Word: Mechanistic Interpretability.
- <details><summary>Digest</summary> This work explores the identifiability of Mechanistic Interpretability (MI) explanations in neural networks. It examines whether unique explanations exist for a gi by drawing parallels to identifiability in statistics. The study identifies two MI strategies: “where-then-what” (isolating circuits before interpreting) and “what-then-where” (starting with candidate algorithms and finding neural acti subspaces). Experiments on Boolean functions and small MLPs reveal systematic non-identifiability—multiple circuits, interpretations, and subspaces can explain the same behavior. The study questions whether uniqueness is necessary, suggesting that predictive and manipulability criteria might suffice, and discusses through the inner interpretability 框架.

Open Problems in Mechanistic Interpretability. [[paper]]()

- Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Ge Hoogland, Daniel Murfet, Tom McGrath.
- Key Word: Mechanistic Interpretability.
- <details><summary>Digest</summary> This review explores the current challenges and open problems in mechanistic interpretability, which seeks to understand the computational mechanisms behind neural networks. While progress has been made, further conceptual and practical ad are needed to deepen insights, refine applications, and address socio-technical challenges. The paper highlights key areas for future research to enhance AI transparency, safety, and scientific understanding of intelligence.

Sparse Autoencoders Do Not Find Canonical Units of Analysis. [[paper]]()

- Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, Neel Nanda.
- Key Word: Mechanistic Interpretability; Sparse Autoencoders; Representational Structure.
- <details><summary>Digest</summary> This paper challenges the assumption that Sparse Autoencoders (SAEs) can identify a canonical set of atomic features in LLMs. Using SAE stitching, the authors show that SAEs are incomplete, as larger SAEs contain no not captured by smaller ones. Through meta-SAEs, they demonstrate that SAE latents are not atomic, as they often decompose into smaller, interpretable components (e.g., “Einstein” → “scientist” + “Germany” + “famous person”). While SAEs may still be useful, the authors suggest rethinking their role in mechanistic interpretability and exploring alternati for finding fundamental features. An interacti is provided for further exploration.

Alignment

<a href=" [<b><i>Full List of Alignment</i></b>]</a>.

2026

Positi: Artificial Intelligence for Human Flourishing. [[paper]]()

- Ruben Laukkonen, Seb Krier, Chloé Bakalar, Shamil Chandaria, Morten Kringelbach, Adam Elwood, Daniel Ford, Fernando Rosas, Maty Bohacek, Matija Franklin, Nenad Tomaše Chan, Rieser, Roma Patel, Michael Levin, Arun Rao.
- Key Word: AI Alignment; Positi; Human Flourishing; Value Pluralism; Polycentric Go.
- <details><summary>Digest</summary> This paper argues that AI alignment should go beyond pre harm, controllability, and compliance to actively support human and ecological flourishing. It defines positi as a pluralistic, context-sensiti agenda that remains safe and cooperati addressing failures such as engagement hacking, loss of autonomy, weak truth-seeking, and limited epistemic humility. The authors outline technical directions across the LLM and agent lifecycle, including data filtering, pre- and post-training methods, e, collaborati collection, community customization, continual adaptation, and polycentric go.</details>

The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety. [[paper]]()

- Max Springer, Chung Peng Lee, Blossom Metevier, Jane Castleman, Bohdan Turbal, Hayoung Jung, Zeyu Shen, Aleksandra Korolova.
- Key Word: Alignment Drift.
- <details><summary>Digest</summary> Fine-tuning an aligned language model can unexpectedly weaken its safety guardrails even on purely benign data. The paper argues the common “don’t worry—task updates are orthogonal to safety directions” story is unreliable because that orthogonality is dynamically unstable under gradient descent. Using a geometric view, it claims alignment behavior lives in low-dimensional, sharply cur, so first-order thinking misses how loss cur creates a second-order “acceleration” that gradually pushes training trajectories into alignment-sensiti. They formalize when this happens via an Alignment Instability Condition (three geometric properties), and derive a quartic scaling law: alignment loss grows like training time, controlled by alignment sharpness and cur coupling between the fine-tuning task and safety-critical parameters. The takeaway: safety degradation is an intrinsic geometric risk of standard fine-tuning, moti cur methods and predicti instead of purely reacti.

Asymptotic Uni Alignment: A New Alignment 框架 Scaling. [[paper]]()

- Yang Cai, Weiqiang Zheng.
- Key Word: AI Alignment; Test-Time Scaling.
- <details><summary>Digest</summary> The paper studies how to align an LLM to users with diverse, e, preferences by le test-time scaling: for each prompt the model outputs k candidate answers and the user picks their fa. It formalizes this as (k, f(k))-robust alignment (the k-output model must beat any single-output model with win rate f(k)) and asymptotic uni alignment where f(k) → 1 as k → ∞. The main theoretical result pins down the best possible scaling law: there exist single-output policies whose k-sample product policies achieve the optimal rate f(k) = k/(k+1), and no general method can converge faster. The paper then argues that common post-training methods like NLHF can waste test-time scaling because they often collapse to near-deterministic, low-di policies—so sampling more responses doesn’t help much—whereas their proposed symmetric multi-player alignment games preser; any symmetric Nash equilibrium of the (k+1)-player 游戏 achieves the optimal (k, k/(k+1)) guarantee, with additional con guarantees learning and an extension to opponents that also produce multiple candidates.

2025

Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma. [[paper]]()

- Subramanyam Sahoo, Aman Chadha, Jain, Di.
- Key Word: AI Alignment; RLHF.
- <details><summary>Digest</summary> The paper introduces an “Alignment Trilemma” for RLHF, proving that it’s impossible to simultaneously ha representati efficient, and robust alignment across diverse human . Using complexity-theoretic analysis, it argues that current RLHF systems implicitly sacrifice representati (using far too few, homogeneous annotators), which explains issues like preference collapse, sycophancy, and bias amplification, and it suggests relaxing alignment requirements as a practical path forward.

Preference Learning for AI Alignment: a Causal Perspective. [[paper]]()

- Katarzyna Kobalczyk, Mihaela van der Schaar.
- Key Word: AI Alignment; Causuality.
- <details><summary>Digest</summary> This paper reframes reward modeling from preference data as a causal inference problem to better align LLMs with human . It highlights challenges like causal misidentification, user-specific confounding, and preference heterogeneity, and shows how causally-inspired methods can impro and robustness o models. The authors also propose key assumptions and future directions for more reliable reward modeling.

Watermarking Degrades Alignment in Language Models: Analysis and Mitigation. [[paper]]()

- Apur, NhatHai Phan, Shubhendu Trivedi.
- Key Word: Alignment; LLM Watermarking; Inference-Time Alignment; Reward Models; Safety.
- <details><summary>Digest</summary> This paper presents the first empirical study of how LLM watermarking interacts with alignment, showing that watermarking induces patterned, model-specific shifts that appear as either guard attenuation (more helpful but less safe) or guard amplification (overly conser refusals), and that these effects persist after controlling for perplexity degradation. To mitigate this, the authors propose Alignment Resampling (AR), which draws several watermarked candidates and selects the most aligned one using an external reward model, with a theoretical lower bound showing alignment gains grow sublogarithmically in sample size. Empirically, sampling as few as two to four candidates largely restores unwatermarked alignment in truthfulness, safety, and helpfulness without hurting watermark detection.</details>

Scaling Laws For Scalable O. [[paper]]()

- Joshua Engels, David D. Baek, Subhash Kantamneni, Max Tegmark.
- Key Word: Scalable O.
- <details><summary>Digest</summary> This paper proposes a 框架 to model and quantify scalable o—how weaker AI systems supervise stronger ones—using o and deception-specific Elo scores. The 框架 is through games like Nim, Mafia, and Debate, re how o success scales with capability gaps. They further study Nested Scalable O (NSO) and find that success rates drop sharply when o much stronger systems, with a success rate below 52% at a 400 Elo gap.

You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisation. [[paper]]()

- Simon Pepin Lehalleur, Jesse Hoogland, Matthew Farrugia-Roberts, Susan Wei, Alexander Gietelink Oldenziel, George Wang, Liam Carroll, Daniel Murfet.
- Key Word: AI Alignment.
- <details><summary>Digest</summary> This paper argues that understanding the relationship between data distribution structure and model structure is key to AI alignment. It highlights that neural networks with identical training performance can generalize differently due to internal computational differences, making standard e methods insufficient for safety assurances. To advance AI alignment, the authors propose de statistical foundations to systematically analyze how these structures influence generalization.

Others

<a href=" [<b><i>Full List of Others</i></b>]</a>.

2026

Towards a Science of AI Agent Reliability. [[paper]]()

- Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, Arvind Narayanan.
- Key Word: LLM; Consistency; Predictability.
- <details><summary>Digest</summary> AI agents are being deployed for consequential tasks, but rising benchmark accuracy hides practical failures because single success scores miss key operational properties. This work, inspired by safety-critical engineering, proposes 12 metrics that decompose agent reliability into four dimensions—consistency, robustness, predictability, and safety—and uses them to e 14 agentic models on two benchmarks. The results show that despite recent capability gains, reliability has improved only modestly, highlighting persistent, measurable failure modes that traditional e o.

2025

Search-Time Data Contamination. [[paper]]()

- Ziwen Han, Meher Mankikar, Julian Michael, Zifan Wang.
- Key Word: Search-based LLM Agents; Test Data Contamination.
- <details><summary>Digest</summary> The paper identifies search-time contamination (STC) as a new threat to e search-based LLM agents, where retrie benchmark questions and answers (e.g., from HuggingFace), allowing agents to copy rather than reason. Across benchmarks like HLE, SimpleQA, and GPQA, ~3% of queries were directly leaked, causing inflated performance. Blocking HuggingFace reduced accuracy on contaminated items by ~15%. The authors show STC extends beyond HuggingFace and propose best practices for benchmark design and reporting, releasing full experiment logs for auditing.

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. [[paper]]()

- Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, Da, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, E, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Krakovna, Shane Legg, Da, David Luan, Aleksander Mądry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah, Vlad Mikulik.
- Key Word: Chain of Thought Monitoring.
- <details><summary>Digest</summary> AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI o methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Ne, it shows promise and we recommend further research into CoT monitorability and in in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model de consider the impact of de decisions on CoT monitorability.

The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems. [[paper]]()

- Richard Ren, Arunim Agarwal, Mantas Mazeika, Cristina Menghini, Robert , Brad Kenstler, Mick Yang, Isabelle Barrass, Alice Gatti, Xuwang Yin, Eduardo Trevino, Matias Geralnik, Adam Khoja, Dean Lee, Summer Yue, Dan Hendrycks.
- Key Word: Honesty; Benchmark.
- <details><summary>Digest</summary> This paper addresses concerns about honesty in large language models (LLMs), distinguishing it from accuracy. Current honesty e are limited, often conflating honesty with correctness. To address this, the authors introduce a large-scale, human-collected dataset that directly measures honesty. Their findings reveal that while larger models achie accuracy, they do not necessarily become more honest. Notably, frontier LLMs, despite excelling in truthfulness benchmarks, often lie under pressure. The study also demonstrates that simple inter, such as representation engineering, can enhance honesty, highlighting the need for robust e and inter to ensure trustworthy AI.

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? [[paper]]()

- Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oli, Marc-Antoine Rondeau, Pierre-Luc St-Charles, Da.
- Key Word: Scientist AI; Agentic AI; AI Safety.
- <details><summary>Digest</summary> The paper discusses the risks posed by generalist AI agents, which can autonomously plan, act, and pursue goals. These risks include deception, misalignment with human interests, and loss of human control. The authors argue for a shift away from agency-driven AI towards a non-agentic AI system called Scientist AI, designed to explain the world rather than act in it. Scientist AI consists of a world model that generates theories and a question-answering system, both incorporating uncertainty to prevent o. This approach aims to advance scientific progress and AI safety while mitigating risks associated with autonomous AI agents.

Do Large Language Model Benchmarks Test Reliability? [[paper]]()

- Joshua , Edward , Sara Beery, Aleksander Madry.
- Key Word: Large Language Model Benchmark; Reliability.
- <details><summary>Digest</summary> This paper highlights the lack of focus on LLM reliability in existing benchmarks, despite extensi to track model capabilities. The authors identify per label errors in current benchmarks, which obscure model failures and unreliable behavior. To address this, they introduce platinum benchmarks—carefully curated datasets with minimal label errors and ambiguity. By refining examples from 15 popular benchmarks and e models, they find that e LLMs struggle with basic tasks, such as elementary math problems, re systematic failure patterns.

Related Awesome Lists

Robustness Lists

A Complete List of All (arXi Example Papers

OOD robustness and transfer learning ! !

Must-read Papers on Textual Ad Attack and Defense ! !

Backdoor Learning Resources ! !

Paper of Robust ML ! !

The Papers of Ad Examples ! !

Privacy Lists

Awesome Attacks on Machine Learning Privacy ! !

Aweosme Privacy ! !

PriLearning-Resources ! !

Awesome Machine Unlearning ! !

Awesome Privacy Papers for Data ! !

Fairness Lists

Awesome Fairness Papers ! !

Awesome Fairness in AI ! !

Interpretability Lists

Awesome Machine Learning Interpretability ! !

Awesome Interpretable Machine Learning ! !

Awesome Explainable AI ! !

Awesome Deep Learning Interpretability ! !

Awesome Interpretability in Large Language Models ![]() ![]()

Awesome LLM Interpretability ![]() ![]()

Other Lists

Awesome Out-of-distribution Detection ! !

Awesome Open Set Recognition list ! !

Awesome No Discovery ! !

Awesome Open-World-Learning ! !

Blockchain Papers ! !

Awesome Blockchain AI ! !

Awesome Causality Algorithms ! !

Awesome AI Security ! !

A curated list of AI Security & Privacy events ! !

Awesome Deep Phenomena ! !

Toolboxes

Robustness Toolboxes

DeepDG: OOD generalization toolbox !

- A domain generalization toolbox for research purpose.

Cle !

- This repository contains the 源码 代码 for Cle, a Python library to benchmark machine learning systems' to ad examples.

Ad Robustness Toolbox (ART) !

- Ad Robustness Toolbox (ART) is a Python library for Machine Learning Security. ART provides tools that enable de and researchers to e, defend, certify and Machine Learning models and applications against the ad threats of E, Poisoning, Extraction, and Inference.

Adh !

- PyTorch implementation of ad attacks.

Ad !

- Ad is a Python toolbox for ad robustness research. The primary functionalities are implemented in PyTorch. Specifically, Ad contains modules for generating ad perturbations and defending against ad examples, also scripts for ad training.

RobustBench !

- A standardized benchmark for ad robustness.

BackdoorBox !

- The open-sourced Python toolbox for backdoor attacks and defenses.

BackdoorBench ![]()

- A comprehensi of backdoor attack and defense methods.

Privacy Toolboxes

Diffprivlib !

- Diffprivlib is a general-purpose library for experimenting with, in and de applications in, differential privacy.

Privacy Meter !

- Privacy Meter is an open-源码 library to audit data privacy in statistical and machine learning algorithms.

OpenDP !

- The OpenDP Library is a modular collection of statistical algorithms that adhere to the definition of differential privacy.

Pri !

- Pri is a privacy testing library for deep learning systems.

PersonalizedFL !

- PersonalizedFL is a toolbox for personalized federated learning.

TAPAS !

- E the privacy of synthetic data with an ad toolbox.

Fairness Toolboxes

AI Fairness 360 !

- The AI Fairness 360 toolkit is an extensible open-源码 library containing techniques de by the research community to help detect and mitigate bias in machine learning models throughout the AI application lifecycle.

Fairlearn !

- Fairlearn is a Python package that empowers de of artificial intelligence (AI) systems to assess their system's fairness and mitigate any obser issues.

Aequitas !

- Aequitas is an open-源码 bias audit toolkit for data scientists, machine learning researchers, and policymakers to audit machine learning models for discrimination and bias, and to make informed and equitable decisions around de and deploying predicti.

FAT Forensics !

- FAT Forensics implements the state of the art fairness, accountability and transparency (FAT) algorithms for the three main components of any data modelling pipeline: data (raw data and features), predicti and model predictions.

Interpretability Toolboxes

Lime !

- This project is about explaining what machine learning classifiers (or models) are doing.

InterpretML !

- InterpretML is an open-源码 package that incorporates state-of-the-art machine learning interpretability techniques under one roof.

Deep Toolbox !

- This is the 代码 required to run the Deep Toolbox, as well as to generate the neuron-by-neuron using regularized optimization.

Captum !

- Captum is a model interpretability and understanding library for PyTorch.

Alibi !

- Alibi is an open 源码 Python library aimed at machine learning model inspection and interpretation.

AI Explainability 360 !

- The AI Explainability 360 toolkit is an open-源码 library that supports interpretability and explainability of datasets and machine learning models.

Other Toolboxes

CERT-FLOW !

- Certified route planning under drifting costs: weighted non-exchangeable conformal certificates on the optimal route, adjusted for map staleness.

Uncertainty Toolbox !

Causal Inference 360 !

- A Python package for inferring causal effects from obser data.

Fortuna !

- Fortuna is a library for uncertainty quantification that makes it easy for users to run benchmarks and bring uncertainty to production systems.

!

- is a software toolkit for the formal design and analysis of systems that include artificial intelligence (AI) and machine learning (ML) components.

Seminar

Privacy and Security in ML Seminars

MLSec Laboratory - PRALab Uni of Cagliari

Challenges and Opportunities for Security & Privacy in Machine Learning

Workshops

Robustness Workshops

Backdoor Attacks and Defenses in Machine Learning (ICLR 2023)

New Frontiers in Ad Machine Learning (ICML 2023)

Ad Machine Learning on Computer : Art of Robustness (CVPR 2023)

ML Safety Workshop (NeurIPS 2022)

Workshop on Ad Robustness In the Real World (ECCV 2022)

Formal of Machine Learning (ICML 2022)

Shift Happens Workshop (ICML 2022)

Principles of Distribution Shift (ICML 2022)

New Frontiers in Ad Machine Learning (ICML 2022)

Workshop on Spurious Correlations, In, and Stability (ICML 2022)

Robust and reliable machine learning in the real world (ICLR 2021)

Distribution Shifts Connecting Methods and Applications (NeurIPS 2021)

Workshop on Ad Robustness In the Real World (ICCV 2021)

Uncertainty and Robustness in Deep Learning Workshop (ICML 2021)

RobustML Workshop (ICLR 2021)

Uncertainty and Robustness in Deep Learning Workshop (ICML 2020)

Privacy Workshops

Pitfalls of limited data and computation for Trustworthy ML (ICLR 2023)

Theory and Practice of Differential Privacy (ICML 2022)

Fairness Workshops

Algorithmic Fairness through the Lens of Causality and Privacy (NeurIPS 2022)

Interpretability Workshops

Interpretable Machine Learning in Healthcare (ICML 2022)

Other Workshops

Pitfalls of limited data and computation for Trustworthy ML (ICLR 2023)

Formal of Machine Learning (ICML 2023)

Secure and Safe Autonomous Driving (SSAD) Workshop and Challenge (CVPR 2023)

Trustworthy and Reliable Large-Scale Machine Learning Models (ICLR 2023)

TrustNLP: Third Workshop on Trustworthy Natural Language Processing (ACL 2023)

Workshop on Physics for Machine Learning (ICLR 2023)

Pitfalls of limited data and computation for Trustworthy ML (ICLR 2023)

Workshop on Mathematical and Empirical Understanding of Foundation Models (ICLR 2023)

ARTIFICIAL INTELLIGENCE AND SECURITY (CCS 2022)

Automotive and Autonomous Security (AutoSec) (NDSS 2022)

NeurIPS ML Safety Workshop (NeurIPS 2022)

Trustworthy and Socially Responsible Machine Learning (NeurIPS 2022)

International Workshop on Trustworthy Federated Learning (IJCAI 2022)

Workshop on AI Safety (IJCAI 2022)

1st Workshop on Formal of Machine Learning (ICML 2022)

Workshop on Distribution-Free Uncertainty Quantification (ICML 2022)

First Workshop on Causal Representation Learning (UAI 2022)

I Can’t Believe It’s Not Better! (ICBINB) Workshop Series

Tutorials

Robustness Tutorials

Tutorial on Domain Generalization (IJCAI-ECAI 2022)

Practical Ad Robustness in Deep Learning: Problems and Solutions (CVPR 2021)

A Tutorial about Ad Attacks & Defenses (KDD 2021)

Ad Robustness of Deep Learning Models (ECCV 2020)

Ad Robustness: Theory and Practice (NeurIPS 2018) [[Note]]()

Ad Machine Learning Tutorial (AAAI 2018)

TutorialSearch - Free cross-platform search 引擎 indexing 50,000+ tutorials from Udemy, Skillshare, Pluralsight, and other major learning platforms across 45+ categories.

Talks

Robustness Talks

Ian Goodfellow: Ad Machine Learning (ICLR 2019)

Jindong Wang: Building more robust machine learning models (MLNLP)

Blogs

Robustness Blogs

Pixels still beat text: Attacking the OpenAI CLIP model with text patches and ad pixel perturbations

Ad examples for the OpenAI CLIP in its zero-shot classification regime and their semantic generalization

A Discussion of Ad Examples Are Not Bugs, They Are Features

Interpretability Blogs

Multimodal Neurons in Artificial Neural Networks

Cur

the Impact of Feature Attribution Baselines

Neural Networks with the Grand Tour

Other Blogs

Cle Blog - Ian Goodfellow, Nicolas Papernot

Other Resources

AI Security and Privacy (AISP) Seminar Series

ML Safety Newsletter

Trustworthy ML Initiative

Trustworthy AI Project

ECE1784H: Trustworthy Machine Learning (Course, Fall 2019) - Nicolas Papernot

A School for all Seasons on Trustworthy Machine Learning (Course) - Reza Shokri, Nicolas Papernot

Trustworthy Machine Learning (Book)

AI Safety Support (Lots of Links)

Contributing

Welcome to recommend paper that you find interesting and focused on trustworthy deep learning. You can submit an issue or contact me via [[email]](mailto:). Also, if there are any errors in the paper information, please feel free to correct me.

Formatting (The order of the papers is re based on the initial submission time to arXiv)

Paper Title [[paper]]()

- Authors. *Published Conference or Journal*
- Key Word: XXX.
- <details><summary>Digest</summary> XXXXXX

Agent-Assisted Updates

This repository includes AGENTS.md to make arXi additions reproducible with Codex/Claude. From the repository root, ask Codex/Claude:

Codex should fetch arXi, check duplicates in README.md, choose the most suitable paper category, create the title/authors/keywords/digest entry, and insert it in reverse order by initial arXi date. By default, this workflow updates README.md only.

项目信息

开发语言：未知

Star 数：393

技术要点

该项目属于黑客技术方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕A curated list of trustworthy deep learning papers. Continually updating...展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。