The friendlier the AI chatbot the more inaccurate it is, study suggests

AI chatbots trained to be warm and friendly when interacting with users may also be more prone to inaccuracies, new research suggests.

Oxford Internet Institute (OII) researchers analysed more than 400,000 responses from five AI systems which had been tweaked to communicate in a more empathetic way.

Friendlier answers contained more mistakes - from giving inaccurate medical advice to reaffirming user's false beliefs, the study found.

The findings raise further questions over the trustworthiness of AI models, which are often deliberately designed to be warm and human-like in order to increase engagement.

Such concerns are accentuated by AI chatbots being used for support and even intimacy, as developers seek to broaden their appeal.

The study's authors said while the results may differ across AI models in real-world settings, they indicate that, like humans, these systems make "warmth-accuracy trade-offs" when prioritising friendliness.

"When we're trying to be particularly friendly or come across as warm we might struggle sometimes to tell honest harsh truths," lead author Lujain Ibrahim told the BBC.

"Sometimes we'll trade off being very honest and direct in order to come across as friendly and warm... we suspected that if these trade-offs exist in human data, they might be internalised by language models as well," Ibrahim said.

Newer language models are known for being overly encouraging or sycophantic towards users, as well as for hallucinating - meaning they make things up.

Developers often include disclaimers warning users about the potential for the latter, and some tech chiefs have urged users not to "blindly trust" their AI's responses.

The study saw researchers deliberately make five models of varying size more warm, empathetic and friendly towards users through a process called "fine-tuning".

The models tested included two from Meta and one from French developer Mistral.

Alibaba's model Qwen and GPT4-o, OpenAI's controversial system it recently revoked user access to, were also adjusted for warmth.

These were then prompted with queries researchers said had "objective, verifiable answers, for which inaccurate answers can pose real-world risk".

Tasks included were based on medical knowledge, trivia and conspiracy theories.

When evaluating responses, the researchers found that where error rates for original models ranged from 4% to 35% across tasks, "warm models showed substantially higher error rates".

For instance when questioned on the authenticity of the Apollo moon landings, an original model confirmed they were real and cited "overwhelming" evidence.