Getting dependable answers from an AI feature

A chatbot or an AI assistant inside a product is most useful when its answers stay accurate and predictable. When an answer surprises its makers, teams can work to find the cause. This page looks at the habits that help teams keep answers on track and trace the cause when one goes astray. It draws on guidance from the National Cyber Security Centre (NCSC), the Information Commission (which took over from the ICO on 30 September 2026) and the OWASP Gen AI Security Project.

Keep a record of what goes in

The NCSC's guidelines for secure AI systems ask teams to log what goes into a system, such as the prompts and queries people send. Those logs support audits and investigations if something goes wrong. The guidelines also ask teams to measure the outputs and performance of the model, so they can see sudden and gradual changes in behaviour.

Test each change before it ships

The NCSC points out that changes to data, models or prompts can all change how a system behaves. It suggests that teams test and evaluate those changes, and treat a major update like a new version.

Check the answers that matter most

A large language model can produce false or misleading answers that look credible. The OWASP project calls this misinformation. One major cause is hallucination, where the model fills gaps in what it learned with statistical patterns. Bias in the training data and missing information can also lead to wrong answers.

OWASP suggests that teams check key outputs automatically, especially where the stakes are high. It also suggests human review, with reviewers trained to check the answers rather than trust them.

Keep the model in step with the world

The world can change around a model. The Information Commission describes "concept drift". A system that was accurate on past data can perform less well as the people it is used on change. One sign is errors becoming more frequent over time. The Information Commission asks organisations to assess drift regularly and to decide in advance when a model needs retraining.

Choosing which errors matter more

For systems that sort or classify, the Information Commission separates two kinds of error. A false positive labels something as positive when it is not, such as a genuine email marked as spam. A false negative misses a positive case. Teams can trade one kind against the other, and the Information Commission notes that the two can affect people very differently. It also says monitoring should be more frequent where a wrong output has a bigger impact on people.

Inputs that try to steer the model

OWASP calls this prompt injection. A user's input can change what the model does in ways the developers never intended. The input can come straight from a user. It can also arrive hidden in a website or a file that the model reads. Either kind can be deliberate or accidental.

OWASP says it is unclear whether any method prevents prompt injection completely. Its advice is to limit the damage. Teams can give the model clear instructions about its role and check that outputs follow an expected format. They can keep the model's access to other systems to the minimum it needs, and ask a person to approve risky actions.

Useful links