Are We Building AI We Can’t Trust?

This week I came across three New Scientist articles which hint at a deeper issue we’re not talking about enough.

In the past few weeks, New Scientist has published three separate articles - one on hallucinating chatbots, another on distorted AI benchmarking, and a third on a national-scale medical AI trained on NHS data. On the surface, they cover different aspects of artificial intelligence: technical flaws, transparency gaps, and privacy concerns. But taken together, they tell a broader, more unsettling story.

That story? We’re racing to build smarter, bigger AI systems, but we’re skipping the hard conversations about trust, transparency, and ethics. Whether it’s a chatbot confidently making things up, benchmark results engineered for PR wins, or a health AI trained on 57 million people’s data without clear consent, the same question keeps coming up: can we really trust the AI we’re building?

Hallucinations Aren’t a Bug

We often hear that AI is getting smarter, more capable, and more "reasoned". But according to recent analysis, those upgrades might come with a hidden cost: accuracy. The first New Scientist article highlights that newer AI models, those with enhanced reasoning features, are hallucinating more, not less. OpenAI’s latest "o3" model, for instance, hallucinated 33% of the time when summarising public information about people. Its smaller sibling, o4-mini? A whopping 48%. That’s triple the rate of the earlier "o1" model.

So what exactly is an AI hallucination? It’s when a language model confidently delivers false information or provides a factually accurate answer that’s irrelevant or incorrectly framed. It’s like asking a student for a summary of a novel, and getting back an eloquent essay that sounds brilliant but describes a completely different book.

What’s especially concerning is that these hallucinations aren’t random glitches. They’re built into how these systems work. Large language models don’t "know" anything in the traditional sense, they predict what the next word should be, based on patterns in massive datasets. That means they can sound authoritative without being accurate.

And that’s not just a minor flaw. It’s a core limitation. You can’t reliably use a chatbot as a research assistant, a paralegal, or a customer service agent if it keeps fabricating facts. Despite claims that these issues would improve with each update, the trend line, at least for now, appears to be heading in the wrong direction.

The Illusion of Progress

If hallucinations cast doubt on what AI says, the second article makes us question how we measure AI progress in the first place. At the center of this concern is Chatbot Arena, a popular benchmarking platform where users compare responses from different AI models in head-to-head battles.

In theory, it's a great idea. Real users voting on real answers sounds democratic and data-driven. But the research highlighted in New Scientist reveals something murkier. Big tech companies like Meta and Google have been testing dozens of model variants privately before releasing only the best-scoring versions publicly. These "invisible" models don’t appear in the rankings and leave no trace, except in back-end data.

That means the leader board isn't showing the best models, it’s showing the best-edited versions of them. Imagine running a race, trying twenty different runners in secret, and only recording the time of the one who crosses the finish line fastest. It’s not cheating in the strictest sense, but it certainly isn’t science.

The imbalance goes deeper. The most powerful proprietary models from companies like OpenAI and Google are shown more often to users than open-source alternatives. That gives them more feedback, more exposure, and ultimately a better chance to perform well. The result is a kind of feedback loop, where the most visible models get stronger while others are buried in the noise.

As one researcher put it, this isn't just about flawed rankings. It's about commercial incentives overshadowing scientific rigor. And when those incentives are driving how we assess the most influential AI systems on the planet, it raises uncomfortable questions about transparency, fairness, and accountability.

AI and the 57 Million Patient Problem

While technical flaws and gaming benchmarks are troubling, the third article shifts the spotlight to something even more personal, our health data. The Foresight model, developed by researchers at University College London, is being trained on the de-identified medical records of 57 million people in England. That’s nearly the entire NHS patient population.

The aim sounds promising. Foresight could one day predict disease complications, forecast hospital demand, and help doctors intervene earlier. But it also raises one of the thorniest issues in modern AI: consent and privacy in the age of massive datasets.

Even though the records are technically de-identified, experts warn that truly anonymising rich health data is extremely difficult. With enough cross-referenced information, it may still be possible to re-identify individuals, especially in a model trained on 10 billion medical events.

To make matters more complex, patients can’t opt out. Because the data has been de-identified, it isn’t legally classified as personal data under GDPR. That legal distinction means individuals have no right to request removal or object to its use, even if they’re uncomfortable with it.

There’s also a deeper concern. The Foresight team hasn’t yet tested whether the model memorises specific data points, an important safeguard in evaluating whether private information could leak out of the system. Without that test, we’re left hoping the model behaves as intended.

In principle, using AI to improve healthcare is a powerful idea. But when ethical oversight lags behind technical capability, we risk eroding public trust, exactly when we need it most.

Building Smarter Systems Without Losing Sight of Trust

Put side by side, these three stories sketch out a clear pattern. AI systems are becoming more powerful, but not necessarily more reliable. The way we evaluate them is increasingly shaped by corporate strategy, not scientific transparency. And the data that fuels them, is often handled in ways that sideline public consent.

This isn’t a warning against AI. It's a reminder that how we build and evaluate these systems matters just as much as what they can do. Hallucinations may never fully go away, but we can be honest about their limits. Benchmarks can be more transparent and equitable. And health data, no matter how promising the use case, must be treated with respect and accountability.

There’s no shortage of technical brilliance in the AI field right now. But if we want that brilliance to lead to systems people actually trust, then ethics, transparency, and public engagement need to come first, not last.

It’s not just about asking what AI can do. It’s about asking what we’re willing to accept, and who gets to decide. Links to the three articles: https://www.newscientist.com/article/2479545-ai-hallucinations-are-getting-worse-and-theyre-here-to-stay/ https://www.newscientist.com/article/2479302-concerns-raised-over-ai-trained-on-57-million-nhs-medical-records/ https://www.newscientist.com/article/2478521-meta-amazon-and-google-accused-of-distorting-key-ai-rankings/