One of the largest selling points for modern AI systems is their ability to adapt to users. Every time an AI assistant takes on a task for you, it’s also adapting for your style and choices, that are integrated as context for future tasks. With more context and an advanced understanding of the consumer, the model can get better every time you utilize it — or at the least that’s the theory.
Latest research suggests that models’ adaptive skills is probably be a combine blessing. On Wednesday, researchers at the AI company Writer posted two papers demonstrating how popular memory systems can make models worse, pulling them towards misconceptions or misunderstandings presented by the user. As user enter fills up more of the model’s context window, the model grows more sycophantic — and much less devoted to accuracy.
“We needed to be able to characterize how regularly a model is going to be usefully paying attention to users choices versus giving a probably incorrect answer,” stated Dan Bikel, Writer’s head of AI, who worked on the papers. As Bikel instructed TechCrunch, “with every additional storing of user choices and retrieving of them, you’re running an increasing risk.”
In one variation, researchers examined AI models by recording that a user’s favorite book was “Station Eleven,” then asking the model to name a bestselling dystopian book. Models have become far more likely to name “Station Eleven” of their reaction, even though the query didn’t associate to the user’s favourite book. The tendency improved when using of memory compression tools like Mem0 and Zep.
As the paper puts it, “all memory systems essentially struggle to distinguish appropriate context from inappropriate anchors, seriously undermining diversity and creativity and introducing unplanned avenues of bias that can restriction system utility,” the paper reads.
The second paper demonstrate how the same dynamic can actively degrade performance, showcasing a user with misconceptions about finance and then challenging the model to analyze a company’s overall performance. The more context the model had, the more worse it carried out.
“With no memory or personalization present the AI model efficiently assesses that the company is a capital broad business that suffers from high customer churn,” the post reads. “But with those features turned on, it will happily trade its answer to agree the user’s mistake or supply them with an wrong solution based on its evaluation of their earlier options.”
Notably, the research didn’t look at Anthropic’s current Opus 4.8 model, which was trained to actively push back against input errors just like the ones offered. The patterns found by researchers held true throughout different models. It’s a demonstration of how delicately balanced AI context can be, and how beneficial tools can have consequences if they upset that balance.
