BUILD
Where an AI assistant helps on a website, and where it doesn’t
5 min read
An AI assistant on a website works best where the answer already exists in documented content and the next step is routing, and works worst where it is placed between a frustrated person and a human.
Website assistants have a reputation problem, earned by a decade of chat widgets that could not answer anything and would not let you leave.
The technology is considerably better now and the failure mode has not changed, because it was never mainly about capability. It was about being deployed in the wrong place.
What it is genuinely good at
Three jobs, where the economics are clear and the failure modes are mild.
- Answering questions whose answers exist in your documented content, phrased the way a person actually asked
- Routing an enquiry to the right team, which people are bad at and which wastes a surprising amount of time
- Collecting structured detail before a human picks it up, so the first human reply can be useful rather than a request for more information
Why retrieval matters more than the model
An assistant answering from your own content is doing retrieval with language on top. It finds the relevant passage and phrases it conversationally. When the content is good, the answers are good.
This has a consequence worth stating plainly: most of the work in a useful assistant is the content and the retrieval, not the model. An assistant deployed over thin or contradictory documentation will produce thin and contradictory answers, confidently.
Where it should not be
Not between a frustrated person and a human being. If someone is trying to report a fault or chase an order, the assistant should recognise that quickly and hand over, and the route to a person should be visible at all times rather than unlocked after three failed attempts.
Not answering questions with contractual or regulatory weight — pricing commitments, eligibility, anything someone could reasonably rely on. Those need a person or a page, not a generated sentence.
And not making changes to systems without a confirmation step. Reading is a much more forgiving operation than writing, and an assistant that can alter a record should do so through an explicit, reviewable action.
The controls that make it deployable
Scope it to approved sources rather than general knowledge, so it answers from your content or says it does not know. Make saying "I don't know, here is a person" an acceptable and frequent outcome.
Log conversations and read them, particularly in the first weeks. The transcripts are the best list of gaps in your documentation you will ever get, and acting on them improves the website for everyone, including the people who never open the assistant.
Keep a human review path for anything the assistant flags as an exception, and be clear to the visitor that they are talking to an assistant.
A reasonable way to start
One narrow job, on content you know is accurate, with an obvious route to a person. Measure whether enquiries arrive better qualified and whether the questions people ask it are ones your site should have answered.
Expand only where the transcripts show it is working. An assistant that does one thing reliably is worth more than one that attempts everything and is trusted for nothing.
Questions we get asked
How do we stop an AI assistant making things up?
Constrain it to retrieve from approved content and answer only from what it retrieves, rather than relying on the model's general knowledge. Make not knowing an acceptable answer with a handover attached, and review transcripts regularly, since the failures you have not read are the ones still happening.
Can an AI assistant replace our support team?
It can reduce the volume of repetitive, documented questions and make the remaining contacts better prepared. Treating it as a replacement is where the well-known failures come from, because the queries that reach a human are the complex ones that most need judgement.
What happens to the data people type into an assistant?
That depends on the architecture and it needs deciding before launch: what is retained, for how long, whether it leaves your environment, and what the privacy notice says. Visitors do paste sensitive information into chat boxes, so the handling has to be deliberate rather than inherited from a default.