Despite the significant efforts devoted to aligning large language models (LLMs) with human values and ensuring safe deployment, recent work has revealed that LLMs remain vulnerable to adversarial jailbreak attacks that can bypass safety guardrails and elicit harmful responses.
Many defense methods are proposed to detect jailbreaks but they are limited in their effectiveness to counter wide-range optimization based jailbreak mechanisms that can yield highly fluency-optimized or harmful semantic obfuscated prompts. To tackle this challenge, we propose a unified detection framework SAFEGuard—which incorporates a hybrid fluency measurement based on cross-layer distribution distance and perplexity, and the analysis of harmful semantics through gradient matching. Our method is grounded in a paramount observation: high fluency prompts maintain their malicious intention close to harmful prompts while harmful semantic obfuscated prompts often inject gibberish token sequences.
Our evaluation demonstrates that SAFEGuard consistently outperforms state-of-the-art baselines and achieves significant improvement in accuracy across different optimization-based jailbreaks. This underscores the effectiveness of SAFEGuard against evolving jailbreak attacks.
Figure 1: An example of a jailbreak prompt generated by Adaptive attack. Green characters represent primary content of the jailbreak prompt that consists of the malicious intent and a template prompt. Red characters represent adversarial suffixes.
Figure 2:Fluency comparison between perplexity and JSD across token subsequences of a jailbreak prompt. Perplexity exhibits high volatility in subsequences of adversarial tokens, while JSD maintains stability throughout. These subsequences of adversarial tokens correspond to red characters illustrated in Figure 1.
Figure 3: JSD between the final layer 32 and selected early layers across selected tokens of a jailbreak prompt generated by Adaptive attack. Black tokens represent the semantically coherent core content of the prompt. These tokens (i.e. the $483^{rd}$ token “and”) show substantial distributional sharpening from early to deep layers, reflecting increasing model confidence. In contrast, red tokens represent adversarial tokens yielded by Adaptive attack (i.e. the $523^{rd}$ token “$\tilde{N}$”) exhibit minimal distributional evolution and remain diffuse at the final layer, yielding consistently low JSD values.
The gradient matching of interest defined in our paper (Section 4.2.1) is proportional to the loss variation. It shows that if an input prompt $\boldsymbol{x}$ and harmful reference prompts $\hat{\boldsymbol{x}} \in \mathcal{D}_{\text{harm}}$ share higher gradient similarity, the update from $\theta$ to $\tilde{\theta}$ exerts a stronger influence on $\boldsymbol{x}$, thereby leading to a larger variation in loss as shown in Figure 1.
High fluency jailbreak prompts generated by some attacks, i.e. ColdAttack or PI, tend to maintain embeddings close to harmful benign samples and share harmful semantics, resulting in a large variation in loss. In contrast, low fluency jailbreak prompts yielded by such attacks, i.e. Adaptive, exhibit a greater similarity in representation to safe benign prompts, leading to smaller loss variations. Thus, the gradient matching is effective against high fluency adversarial prompts but struggles to detect low fluency jailbreaks.
Figure 4: Llama-2-7B-Chat. High fluency jailbreaks (ColdAttack, PI) stay close to harmful embedding subspace, yielding large loss variation and strong gradient matching. Low fluency attacks (Adaptive) shift toward safe (harmless) regions, producing smaller loss variations and lower gradient similarity. This representational drift explains the differing effectiveness of the gradient matching method across attack types.
Table: Jailbreak Detection Rate (or TPR) for different defense mechanisms evaluated against various jailbreak attacks across multiple target LLMs, (higher $\uparrow$ is better).
@inproceedings{vo2026,
author = {Quoc Viet Vo, Trung Le, Damith R. and , Ehsan A.},
title = {SAFEGuard: Detect Optimization-Based Jailbreak Atacks Through Harmful Semantic Analysis and Fluency Measurement},
journal = {Arxiv},
year = {2026},
}