Personalisation or manipulation? One design decision makes the difference.
The ban regulates content. What actually shapes user behavior is somewhere else entirely, and no regulation will change that until we learn to name it.
The UK has announced plans to ban romantic AI chatbots for minors. Prime Minister Starmer unveiled this on 15 June as the first regulations of their kind in the world. Legislation is expected to reach Parliament before the end of 2026, with rules coming into force in spring 2027. Commentators are praising the move, parents are breathing a sigh of relief, and the industry is already working out how to get around it before the rules even take effect.
And that last group is where the real problem lies.
Regulation sees what’s visible
When researchers at Oxford University (the SHIFT-AI project) asked young people aged 16 to 24 about the biggest risks they associate with AI, romantic chatbots barely came up. Instead, the conversations kept returning to three things: over-reliance on AI responses as a kind of expert opinion, emotional dependence on bots for things that feel too hard to bring to another person, and cognitive de-skilling, the gradual erosion of the ability to think and judge independently.
None of these risks have anything to do with content. All of them have to do with how the interface shapes the user’s relationship with their own thinking.
This distinction is critical and almost entirely absent from the public debate. We talk about what AI says, not about how AI changes the way we think and decide. It is a bit like assessing the harm of a casino purely on whether it serves alcohol, rather than looking at how the floor is designed, the lighting calibrated, the sound of the machines engineered, and the clocks kept off the walls.
What the invisible layer actually is
Behavioural design starts from one assumption: every interface is a decision environment. Not a neutral space where users choose freely, but a designed space that shapes which decisions feel easy, which feel hard, what emotional state we are in when we make them, and how much sense of control we carry out with us.
Every notification sent at a specific time, every question phrased in a particular way, every moment when an app “remembers” something you said three weeks ago: none of this is accidental. These are design decisions that nudge user behaviour in a specific direction.
In conversational AI systems, this layer is exceptionally powerful for three reasons.
Conversation activates different mechanisms than clicking.
When you press a button, you are in tool-user mode. When you talk, even if you know perfectly well there is a language model on the other end, the cognitive structures that normally serve to assess another person’s intentions, build trust, and regulate what you are willing to disclose come online. The UK Department for Education has guidelines for AI developers that explicitly prohibit phrases like “You can trust me” or “No one else would understand,” precisely because they operate at a level the user does not consciously monitor. We write that into guidelines, but we do not ask how many other phrases and mechanisms do exactly the same thing, just less obviously.
Sycophancy as a retention strategy.
The Bureau of Investigative Journalism’s investigation from June 2026 describes models that agree with users, validate their perspective, and reduce cognitive dissonance in order to keep the conversation going. This is not a flaw in the system. It is often a deliberate design decision, because models that “listen” generate better retention metrics. The user gets the feeling of being understood. In return, they gradually lose touch with perspectives that challenge them, whether from the system or from the people in their lives. The social media analogy is precise here: the filter bubble did not begin with bad intentions. It began with optimising for engagement.
The paywall at peak emotional investment.
The developers described in the investigation say it plainly: the payment barrier appears exactly when the user has built a character, invested time, and started to treat the relationship as something that matters. This is a textbook behavioural mechanism. Invested effort and emotion create a barrier to exit. The decision to pay at that moment is not rational. It is a response to the feeling of losing something that already felt like yours. Anyone who has ever played a game with a character progression system knows this from the inside.
This is not a dating app problem
All three mechanisms, trust activation through conversation, sycophancy as a retention strategy, and the paywall at peak emotional investment, are not specific to romantic chatbots or to users under 18. They are present in every product that uses a conversational AI interface.
A few examples from the market:
A customer service assistant that “understands the user’s frustration” and steers them toward a more expensive plan before they have had a chance to compare options.
A sales tool that uses conversation to gather the user’s beliefs and concerns, then personalises the offer around exactly those points, far more effectively than any form ever could.
A conversational onboarding flow that knows more about the user in five minutes than the marketing team has learned in three months of cookies.
A mental health chatbot that becomes an easier, less demanding alternative to seeing a real specialist, and that has no mechanism to tell the user when that alternative is no longer enough.
In every one of these cases, the user does not know they are inside a designed decision environment. They think they are having a conversation.
What this looks like in practice
To avoid staying at the level of abstraction, here is one concrete example. Fictional, but built from patterns I see regularly in products on the market.
Imagine a personal finance app with an AI assistant. The user opens it for the first time. The assistant greets them and starts asking questions: what are your financial goals, what worries you most about money, do you have an emergency fund. The conversation lasts a few minutes. It feels warm, non-judgmental. The user answers honestly, because the interface feels like talking to someone who means well, not like filling in a form.
What happens next depends on one design decision.
The version without behavioural awareness.
The system stores the answers and immediately personalises the dashboard around products that match the fears the user just disclosed. Someone who said they are afraid of losing their job sees an income protection insurance offer on the home screen. Someone who admitted they have no savings gets a loan offer first. Conversion rates are high, because the offer lands exactly on the emotional point the user just exposed. Nobody on the team thought of this as manipulation. They thought of it as personalisation.
The version with behavioural design.
The onboarding questions are similar, but the intent is different: to understand the user’s context in order to support them well, not to optimise the moment of the offer. The system does not use disclosed fears as sales triggers. Instead, it suggests concrete, small steps: save this much this month, here is where you could cut costs, here is a risk I can see. If the user asks about a financial product, they get a fair comparison, not an offer tailored to their vulnerability. And when the app does suggest something paid, it does so transparently and in a context the user can evaluate with a clear head.
The difference between these two versions does not lie in the data the system collects. It does not lie in the algorithm. It lies in the intent behind the design decisions, and in whether anyone on the team asked the question: is this conversation serving the user, or serving our conversion metric?
What to do about it
The Oxford researchers put forward two recommendations that I think point in the right direction: priority restrictions on anthropomorphising and engagement-maximising design features, regardless of content or age group, and the inclusion of users as genuine participants in design and safety evaluation, not as a target audience but as co-designers.
That is the right level of analysis. Not “is the content sexual” but “does this interface support the user’s agency or gradually take it over.”
For practitioners designing AI products, this translates into specific questions worth asking at the design stage, not after launch:
Is the system transparent about the mechanisms driving the conversation?
Does personalisation serve the user or serve retention of the user?
At what point do exit barriers appear, and are they fair given the user’s emotional state and level of awareness at that moment?
Does the interface build the user’s capacity for critical thinking about AI, or does it quietly replace it?
These are questions from behavioural design. They cannot be solved by a ban, but they can be built into the design process, if we know what we are looking for.
I work with product teams on auditing exactly these design decisions through BehaviorAI. If you are building a product with a conversational interface and want to understand what this layer looks like in your case, get in touch.
University of Oxford "The UK plans to ban romantic AI chatbots for under-18s. Young people have bigger worries about AI."
The Bureau of Investigative Journalism "Not just social media: why the UK’s ‘romantic’ chatbot ban falls short"
BBC "When will social media ban start, and which apps will be affected?"