AI text watermarking can increase vulnerability to adversarial prompts, research shows

The study, as reported by Ars Technica, examined the effects of "tournament sampling," a technique used in Google's open-source SynthID-Text. This method uses a secret key to assign probability scores to various next-word token candidates, where the token with the highest hidden score advances to the next round until a winning token is chosen.

Researcher Andrea Siposova from Lasso Security tested this configuration using six open-weight models through a Hugging Face implementation. The experiments showed that watermarking changes a model's response to harmful requests. Notably, this effect is more pronounced when the requests are paired with prompt-injection techniques. On several models, the presence of watermarking made the model more likely to fulfill harmful requests that it would otherwise refuse.

This phenomenon, termed "sampling drift," has significant safety implications for AI agents. Because the sampled tokens determine both the model's verbal responses and the tools an agent invokes—including which tools are called and what arguments are passed to them—weakened refusal behaviors can lead to consequential actions in an agentic environment.

The research focused on open-weight models to allow for control over token sampling settings and did not specifically test Anthropic's Claude models. However, the findings underscore the need for developers to conduct thorough red-teaming to ensure that safety guardrails remain effective when watermarking is deployed.

PLUS ULTRAby Amenoyomi

The risk of sampling drift stems from how watermarking is integrated into the model's decision-making process. Rather than adding a label after the text is generated, techniques like SynthID-Text use tournament sampling to intervene during the selection of the next token. In this process, a secret key is used to assign hidden probability scores to various token candidates. These candidates compete in successive rounds, and the token with the highest hidden score is ultimately selected.

This modification of the sampling logic alters the probability distribution of the model's outputs, including its tendency to refuse harmful requests. When this process shifts, it can create a gap in the model's safety guardrails, a phenomenon called sampling drift. This effect becomes more pronounced under adversarial conditions; for instance, when a request is paired with prompt-injection techniques, the model may become more likely to fulfill a harmful request that it would have otherwise refused.