The ChatGPT Redis Bug: How a Tiny Async Race Condition Exposed User Data

On 20 March 2023, some ChatGPT users saw the titles of other users' conversations in their sidebar. For a smaller group, billing details were also visible. OpenAI took ChatGPT offline to fix it. The cause was a bug in an open-source library, not an attack.

How ChatGPT uses Redis

ChatGPT caches user information in Redis so it does not have to hit the database on every request. The Python code talks to Redis through the redis-py library, using asyncio and a pool of shared connections: a request borrows one, sends a command, reads the reply and gives the connection back.

A Redis connection is just a socket. Replies arrive in order, with no request id attached. The connection only works if every request is paired with its own reply.

The bug

OpenAI traced it to redis-py. If a request was cancelled after the command was sent but before the reply was read, the connection was still returned to the pool. The reply was still on its way. It arrived and sat unread.

The next request to borrow that connection sent its own command, then read the first reply in the queue. That reply belonged to the previous user.

You can reproduce the whole thing in a few lines. This is a model of the bug, not redis-py itself:

import asyncio


class Connection:
    """One socket: replies come back in the order requests were sent. No request ids."""

    def __init__(self):
        self.replies = asyncio.Queue()

    def send(self, user):                       # the request goes out; the server replies a moment later
        asyncio.get_running_loop().call_later(0.05, self.replies.put_nowait, f"data for {user}")

    async def read(self):
        return await self.replies.get()


pool = [Connection()]                           # a pool with one connection


async def get(user):
    conn = pool.pop()
    try:
        conn.send(user)
        return await conn.read()
    finally:
        pool.append(conn)                       # BUG: returned even if we were cancelled mid-flight


async def main():
    alice = asyncio.create_task(get("alice"))
    await asyncio.sleep(0.01)
    alice.cancel()                              # Alice closes the tab; her reply is still on the wire
    await asyncio.sleep(0.1)                    # ...it arrives and sits unread on the connection
    print("bob sees:", await get("bob"))


asyncio.run(main())

Run it and Bob prints data for alice.

The fix is to return the connection to the pool only if the exchange finished. If it was interrupted, close it and let the pool open a fresh one:

async def get(user):
    conn = pool.pop()
    clean = False
    try:
        conn.send(user)
        reply = await conn.read()
        clean = True
        return reply
    finally:
        if clean:
            pool.append(conn)
        else:
            pool.append(Connection())           # discard the tainted connection

Why it only showed up that day

The bug had been in the library for a while. OpenAI made a server-side change that morning that caused a spike in Redis request cancellations, and cancelled requests are exactly what trigger it. A latent bug plus a change in traffic pattern gave a visible incident.

What to take away

1. Cancellation is a code path. Every await can be interrupted. Ask what state your connection, lock or buffer is in when that happens.
2. Never reuse a connection of unknown state. If you are not sure the exchange finished, throw it away.
3. Responses should carry the request's identity. A protocol with request ids would have caught the mix-up. Where it does not, add a check at the edge.
4. Cache keys and user ids belong in the response path. Verify that what you are about to return belongs to the user you are returning it to.
5. Your dependencies are your code. Read their issue trackers, pin versions and test against them under load and with cancellations.