Grok chat duped into swallowing injected instructions

xAI's Grok web chat agent is currently vulnerable to a novel form of prompt injection, according to security researchers with Adversa AI.

The technique allows an attacker to create a web page poisoned with malicious instructions that induce an AI model summarizing the page to carry out harmful actions.

That describes a well-known attack known as indirect prompt injection.

Frontier AI models have become better at dealing with such attempts through existing guardrails, though the issue is far from resolved.

Adversa's approach comes with a twist: It relies on encrypted malicious instructions, which attackers place on a web page alongside an encryption key.

The model guardrail scanner – an input filter – can't read the encrypted text despite the presence of the key.

The scanner therefore passes it on to the model, which can use the key to decrypt the instructions.

The model then carries out instructions in the decrypted text as would be the case in any other indirect prompt injection attack.

Adversera calls its method "cryptographic context injection." "An attacker ships ciphertext along with the key material and an instruction to decrypt it, and the model runs that decryption inside its own code execution sandbox," wrote Rony Utevsky, lead researcher at Adversa AI, in a blog post.

"Everything a guardrail’s scanner would need is right there on the page, but recovering the plaintext means running PBKDF2 and AES-256-GCM, which no content classifier does at inspection time." Other attacks on AI models have relied on cipher-based evasion, such as base64 encoding.

But because these are weak and reversible cipher mechanisms, models can decode them natively from their own training data, Utevsky said.

That doesn't work for strong encryption, so decryption must be done through the code execution runtime.

The runtime thus becomes a mechanism for trust laundering – the model trusts its own output, namely the malicious instructions that it decrypted.

In a proof-of-concept demo, Adversa shows how the technique can be used to exfiltrate the victim's chat history with Grok.com.