Defensive Prompt Patch: A Robust and Generalizable Defense of Large Language Models against Jailbreak Attacks
Hong-Kong Science University, Princeton University, IBM (2025)
https://arxiv.org/pdf/2405.20099

Метод, аналогичный RPO

#MLSecOps
"The Art of Hide and Seek: Making Pickle-Based Model Supply Chain Poisoning Stealthy Again", 2025.

// the first systematic disclosure of the picklebased model poisoning surface from both the model loading and risky function perspectives

#MLSecOps
#Red_Team_Tactics
"Web Fraud Attacks Against LLM-Driven Multi-Agent Systems", 2025.
]-> Examples of WFA (Repo)

// In this paper, we propose Web Fraud Attacks, a novel type of attack aiming at inducing MAS to visit malicious websites. We design 11 representative attack variants that encompass domain name tampering, link structure camouflage (sub-directory nesting, sub-domain grafting, parameter obfuscation, etc.), and other deceptive techniques tailored to exploit MAS's vulnerabilities in link validation

An Automated Multi-Agent Framework for Reproducing CVEs
https://arxiv.org/pdf/2509.01835

#AIOps
#Offensive_security
"Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees", COLM 2025.
]-> https://github.com/KatsuNK/stt-reasoning

// a guided reasoning pipeline for pentesting LLM agents that incorporates a deterministic task tree built from the MITRE ATT&CK Matrix, a proven penetration testing kill chain, to constrain the LLM’s reasoning process to explicitly defined tactics, techniques, and procedures

#tools
#MLSecOps
"Stealth by Conformity: Evading Robust Aggregation through Adaptive Poisoning", 2025.
]-> AIJack: Security and Privacy Risk Simulator for Machine Learning

// In this paper, we challenge this underlying assumption by showing that a model can be poisoned while keeping malicious updates within the main distribution

CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning - https://ai.meta.com/research/publications/cybersoceval-benchmarking-llms-capabilities-for-malware-analysis-and-threat-intelligence-reasoning/

Today’s cyber defenders are overwhelmed by a deluge of security alerts, threat intelligence signals, and shifting business context, creating an urgent need for AI systems that can enhance operational security work. Despite the potential of Large Language Models (LLMs) to automate and scale Security Operations Center (SOC) operations, existing evaluations are incomplete in assessing the scenarios that matter most to real-world cyber defenders. This lack of informed evaluation has significant implications for both AI developers and those seeking to apply LLMs to SOC automation.

Without a clear understanding of how LLMs perform in real-world security scenarios, AI system developers lack a north star to guide their development efforts, and users are left without a reliable way to select the most effective models. Furthermore, malicious actors have begun using AI to scale cyber attacks, emphasizing the need for open source benchmarks to drive adoption and community-driven improvement among defenders and AI model developers.

To address this gap, we introduce CyberSOCEval, a new suite of open source benchmarks that are part of CyberSecEval 4. CyberSOCEval consists of benchmarks tailored to evaluate LLMs in two tasks: Malware Analysis and Threat Intelligence Reasoning, core defensive domains that have inadequate coverage in current security benchmarks. Our evaluations reveal that larger, more modern LLMs tend to perform better, confirming the training scaling laws paradigm. We also find that reasoning models leveraging test time scaling do not achieve the boost they do in areas like coding and math, suggesting that these models have not been trained to reason about cybersecurity analysis, and pointing to a key opportunity for improvement.

Finally, we find that current LLMs are far from saturating our evaluations, demonstrating that CyberSOCEval presents a significant hill to climb for AI developers to improve AI cyber defense capabilities.