Interesting post from Giorgio Di Tunno and Neil Starr on how they are trying to tackle AI misinformation at the Department for Business and Trade (DBT).
I'll be honest, the first half describes the challenge AI summaries pose and it sounds like a horror show. Here's the set up:
Imagine you want to set up a small charity in your local area. You need to know the costs involved, so you Google ‘how much does it cost to set up a charity UK’. An AI overview appears. It states in plain English that it costs £40 if you apply by post or £13 if you apply online.
Let's be clear about what is happening here. The user is looking for the answer to a fairly simple question. There is a carefully written page on GOV.UK that provides an answer. But instead of linking directly to that page, Google now inserts its AI overview into prime position in its search results. The aim is to provide an answer that likely means the user will never reach that GOV.UK page.
I think you know what comes next. The AI-generated answer is – you guessed it – completely wrong. And not just by a bit. It's miles off.
Since 1 February 2026, the cost to incorporate a company in the UK is £124 if you apply by post or £100 if you apply online - as set out in the Companies House fees guidance.
This kind of misinformation is a huge problem generally, but especially in government where accuracy and trust is so important.
What's the argument for this? It would be that the user is able to get the information they need more quickly, I suppose. One search. An immediate answer. User need met. But that's only if the answer is accurate.
Let's be clear. This kind of misinformation causes harm. It actively affects people's lives in material ways. And for what? Not very much, as far as I can tell.
I recommend reading the whole post by to understand why Google is finding and then sharing the wrong information. It's entirely possible this is happening with key details on your own website too.
What struck me most, apart from the folly of this situation, is the fairly routine approach that the DBT content team are taking. And by routine, I mean it seems to me the response is to content design harder. What else can they (or we) do?
In this case, content design harder means:
- auditing and removing outdated content
- establishing a plan for better maintaining content
- testing a new pattern that tells the user a page is reviewed and up to date, as well as when the next review is due
This is all good stuff! It is great that government departments with big teams and even bigger budgets are taking steps to manage the misinformation posed by AI summaries.
But this is an absolute nightmare for small companies, charities with one content designer if they're lucky, and those organisations with no support at all. It's incredibly difficult for them to find the time and skills needed to spend so much time on existing content.
So what's the plan? I guess we have to hope that eventually AI gets better and governments, companies and charities are able to better manage their content. While in the meantime, we get used to the idea that the information we're shown is incorrect. We learn to trust nothing.
