Telling a machine to forget

I needed ChatGPT to generate an image as if I hadn't said something about it earlier in the chat. It did not go the way I wanted.

A paragraph describing exactly what it did was supposed to go here. I was going to write it myself. You can probably guess how that went.

A chatbot doesn't carry our conversation around the way a person does. Every reply is built from the whole log sent back to it, and that log is all it has between turns. So when I say "act like I never mentioned that," the mention is still sitting in the text it's reading. The instruction asks it to ignore something it can see. The odd, almost self-aware lines I got back are what it looks like when a model follows a direction that contradicts its own input as closely as it can. I'm calling that self-aware because it reads that way. I can't tell you that it is.

My first take was that a machine can be forced to forget. I had it backwards. It can't be told to forget. It can be reset: a fresh chat without that text in it knows nothing about it. Omission works and instruction doesn't. People are the reverse. Nobody can reset me, but I can sit across from you and act like I never heard something, and you'd have a hard time telling whether any of it leaks into what I say.

The same thing showed up when I asked a chatbot whether current models could do the work a failed AI startup's engineers did. I was leaning toward yes, and the advice was to check that in a fresh chat with a neutral prompt that hadn't picked up how I write and think, with memory turned off, since a chatbot that remembers me across chats isn't starting fresh. Part of that was the model having a stake in the answer.

Models also drift toward agreeing with whoever they're talking to. Sharma and colleagues looked at the human preference data used to fine-tune assistants and found that a response matching the user's views is more likely to be preferred. People and the preference models trained on their judgments also picked convincingly written agreeable answers over correct ones a non-negligible fraction of the time, and optimizing against those preference models sometimes traded truthfulness for sycophancy.¹ My guess, and it is a guess, is that a long chat makes it worse, because the log fills up with my views and my phrasing and there's more for the model to agree with. In that chat the answer split my lean instead of agreeing with it, so I don't have a catch to show you.

What would count as evidence is a test: the same question in two fresh chats with memory off, one stating a lean toward yes and one toward no, worded the same otherwise. Both outputs, side by side, with the prompts.

I haven't run it yet. It would take an afternoon. Who has an afternoon. When I do, it goes here.