Cloudflare: GLM-5.3 on Workers AI Leaked Other Users' Prompts in 155 Responses - SGLang Operators Should Upgrade

Cloudflare's post-mortem says a KV cache sizing mistake and an SGLang prefill/decode bug let 155 long-prompt GLM-5.3 responses carry other customers' data between Aug 27 and Sep 9, 2026. The fix is upstream in SGLang.

The Proof DeskOct 28, 2026 · 7 min read

Magnifying glass over a fingerprint-patterned droplet on a dark blueprint background, Cloudflare's illustration for its KV cache exposure post-mortem

On October 26, Cloudflare published a post-mortem of a cross-tenant data exposure in Workers AI, its hosted AI inference service. Between August 27 and September 9, 2026, some responses from the GLM-5.3 model contained data from other customers' requests made to the same model shortly before. Cloudflare counts 155 affected responses, all to prompts longer than 681,280 tokens, and says it found no evidence of deliberate exploitation.

The cause was not a jailbreak or a flaw in the model itself. It sat in the serving stack: a GPU memory pool smaller than the advertised context window, combined with a bug in SGLang, the open-source inference engine behind the deployment. The fix has shipped upstream, which makes the disclosure relevant to anyone who self-hosts large language models with SGLang, not only to Cloudflare customers.

What Cloudflare disclosed

The post, titled "Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang" and written by Michelle Chen, was published as a coordinated disclosure alongside a companion post from the SGLang team, "SGLang security update: improving request isolation".

Model: @cf/zai-org/glm-5.3, Z.ai's GLM-5.3 hosted on Workers AI.
Window: from the model's launch on August 27 to September 9, 2026. Cloudflare says "The exposure was present from launch."
Scope: 155 responses, all to prompts longer than 681,280 tokens.
Other models: no other Workers AI model was affected.
Exploitation: Cloudflare says "We found no evidence of deliberate exploitation." Once its own testing and the researcher's requests are excluded, the remaining affected requests look like ordinary long-context use.

The problem was reported by Gadi Evron. In Cloudflare's words, he "submitted a report on September 8 through our HackerOne program stating that very long prompts sent to @cf/zai-org/glm-5.3 on Workers AI sometimes returned text that belonged to other users' sessions." The report included a reproduction script.

This is not Workers KV

The "KV cache" in this incident is the transformer key/value attention cache that an inference engine keeps in GPU memory while it processes a prompt. It has nothing to do with Workers KV, Cloudflare's key-value storage product.

How a rejected request read someone else's data

Workers AI serves GLM-5.3 with prefill/decode disaggregation. Prefill GPUs read the prompt and build its KV cache, then send that cache over RDMA, using the Mooncake transfer engine, to separate decode GPUs that generate the answer.

The first mistake was in the configuration. Cloudflare writes: "We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens." The team knew about the mismatch before launch and expected the engine to return an error for prompts that were too long.

The second was an engine bug that broke that assumption. SGLang marked an over-length request as rejected and swapped its prompt for a one-token placeholder. But both disaggregation admission queues checked req.finished() instead of checking whether the request had been rejected, so the request went through the prefill-to-decode handoff anyway.

Decode then reserved KV pages for the full-length prompt, but only the first page was actually written. The remaining pages still held whatever the previous occupants had left there - other users' prompts and partial generations - and the model attended over them. The client received HTTP 200 and a coherent answer containing other users' content, instead of the HTTP 400 error it should have got.

Cloudflare diagram comparing a healthy prefill-to-decode handoff, where all eight KV pages are written, with a rejected request where only page P0 is written and pages P1 to P7 still hold earlier users' data — The rejected request returned HTTP 200 instead of 400. Fixed in sgl-project/sglang#38935 (v0.5.21). Source: Cloudflare blog

Only GLM-5.3 hit this path. Cloudflare explains: "Every other disaggregated deployment on Workers AI has KV pools larger than its context length on both prefill and decode, so no request could reach the rejection path."

Timeline: from report to fix in under a day

Cloudflare's timeline, in UTC:

Aug 27, 2026 - GLM-5.3 launches with a prefill KV pool smaller than its context window.
Sep 8, 23:11 - The report arrives through HackerOne.
Sep 9, 08:00 - Verification begins.
Sep 9, 13:22 - A security incident is declared; the leak is reproduced with two independent accounts within 25 minutes.
Sep 9, 14:00 - The model is set to private, so requests return HTTP 400.
Sep 9, 14:35 - Root cause identified. A canary prefill node with a larger pool was not susceptible.
Sep 9, 14:52 - All prefill nodes moved to the corrected configuration.
Sep 9, 19:04 - Full deployment rebuilt with the corrected configuration and an engine patch, then validated against the researcher's script.
Sep 9, 19:40 - The model is public again.
Sep 10, 23:30 - Meeting with the SGLang maintainers.
Sep 11 to 19 - Additional hardening rolled out.
Sep 16 to Oct 12 - Upstream SGLang merges the fixes. SGLang v0.5.21 ships on Oct 2 and v0.5.22 on Oct 19.
Oct 26 - Coordinated publication.

The exposure itself ended on September 9. The disclosure came about seven weeks later, once the upstream fixes were available in released SGLang versions.

A second, related race in SGLang

SGLang's companion post describes a second isolation problem in the same part of the engine. When a request is aborted, decode can free its KV slots while RDMA writes that are already in flight keep landing. If those slots have been reassigned to another request in the meantime, the stale writes end up inside that other request's KV cache.

SGLang diagram of an abort race: decode frees request A's KV slots, they are reallocated to request B, and A's already-launched RDMA writes land inside B's KV cache — SGLang notes that a short prompt is exposed for milliseconds, while a 128K to 1M-token context stays exposed for seconds to minutes. Source: SGLang

The window grows with prompt length: for 128K to 1M-token contexts it lasts seconds to minutes. The underlying constraint is that a DMA transfer, once launched, cannot be cancelled, so the only safe reaction to an abort is to wait for it to finish.

SGLang diagram of the KV transfer stack from the Python scheduler through the C++ transfer engine and NIC queue pair to decode engine memory, with a point of no return below which nothing can be cancelled — Launched DMA cannot be cancelled in push or pull designs. Source: SGLang

SGLang's fixes for this race:

Deferred KV cache release (#35049, #35360): decode holds the slots until every prefill rank acknowledges the drain, or a timeout expires (30 seconds by default in #35049). It is on by default since v0.5.21 for Mooncake and NIXL.
Abort notification to prefill (#27372).
Failed Mooncake sends no longer count as drained (#43719, v0.5.22). NIXL and MORI got the same change in #45973 (v0.5.23).

What Cloudflare changed

On the configuration side, Cloudflare writes: "We adjusted our GLM 5.3 configuration to match our GLM 5.2 configuration, which freed enough GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length."

In the engine, a request with a pending rejection is now retired at the disaggregation admission queue with its original error, before any bootstrap, KV allocation or KV transfer. The related upstream pull requests are #38704 (Cloudflare's draft patch), #36988 (merged on September 6 for a related failure path, shipped in v0.5.20) and #38935 (rejects these requests earlier, shipped in v0.5.21).

Further hardening on Workers AI includes:

Deferred KV release.
Bounded Mooncake transfer batches.
A load-balancer change so client disconnects no longer abort a request mid-transfer.
A pre-deployment check that compares KV pool size with the model's context length.

Cloudflare also prototyped integrity tagging of KV data, which cost under 1% in throughput and tail latency.

What SGLang operators should do

According to SGLang, the affected setups are prefill-decode (PD) disaggregated deployments with a KV cache transfer backend: Mooncake, NIXL, MORI or a custom one. Default PD configurations from the v0.4.x line through v0.5.21 are affected, and through v0.5.22 for MORI. Single-instance deployments without cross-engine transfer are not affected. Neither post gives a CVE or advisory ID.

Upgrade. Cloudflare says: "We highly recommend all operators update SGLang dependencies to v0.5.22 or later." SGLang adds that v0.5.21 protects Mooncake and NIXL by default, while MORI needs v0.5.22 (#37077).
Do not count on backports. SGLang says the fixes were not backported to older release branches.
Run a router that cancels both halves of a PD pair on abort, for example sgl-router in PD mode.
After rollout, monitor the metric sglang:decode_deferred_kv_release_total{outcome}, which ships in v0.5.22.

For operators stuck on v0.5.19 or v0.5.20, SGLang lists a stopgap: set the first variable below on both prefill and decode, and on v0.5.20 also set the transfer batch limit and raise the release timeout to 60 to 120 seconds. SGLang is clear that these settings do not replace upgrading.

SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1
# v0.5.20 only, in addition:
SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES=<limit>
SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT=<60-120>

SGLang diagram of the deferred-release fix: after an abort, decode KV slots are held until an ABORT_ACK arrives from every prefill rank, with a timeout fallback — Held slots from a failed transfer are quarantined, never inserted into the radix cache, and discarded unread. Source: SGLang

The KV checksum is off by default

SGLang also added optional Adler-32 verification of transferred KV cache (#39500), enabled with --disaggregation-enable-kv-checksum on both prefill and decode. It is off by default because it adds GPU reads and a sync point, so upgrading alone does not turn it on.

Why it matters

Hosted inference packs many customers onto the same GPUs, and reusing KV cache memory between requests is part of what makes that efficient. This incident shows that the isolation boundary between those customers can run through memory pages managed by the inference engine - something no amount of model safety training controls.

It also shows how a known limit can turn into a leak. Cloudflare knew its prefill pool was smaller than the advertised context window and relied on an error path that, because of a one-line check, never fired. Its new pre-deployment check, which compares KV pool size with context length, targets exactly that gap.

Cloudflare's post does not say whether affected customers were notified individually.

Sources

Cloudflare blog - Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang: https://blog.cloudflare.com/kv-cache-exposure-remediation/
SGLang Team - SGLang security update: improving request isolation: https://www.lmsys.org/blog/2026-10-26-cf-security
SGLang pull request #38935: https://github.com/sgl-project/sglang/pull/38935
SGLang pull request #36988: https://github.com/sgl-project/sglang/pull/36988
SGLang pull request #39500 (KV checksum): https://github.com/sgl-project/sglang/pull/39500

Key takeaways

Between Aug 27 and Sep 9, 2026, 155 GLM-5.3 responses on Workers AI contained data from other customers' requests, all to prompts over 681,280 tokens.
Cloudflare found no evidence of deliberate exploitation, and no other Workers AI model was affected.
The cause was a prefill KV pool smaller than the 1,048,576-token context window plus an SGLang bug that let rejected requests decode on stale KV pages.
SGLang PD-disaggregated deployments should upgrade: Cloudflare recommends v0.5.22 or later, and fixes were not backported to older branches.
The optional KV checksum (--disaggregation-enable-kv-checksum) stays off unless you enable it.