What are AI hallucinations?

What Is an AI Hallucination?

An AI hallucination refers to an incorrect or even fabricated output that an AI system generates. The term is used, for example, in connection with generative AI and large language models that generate text, images, etc.

These types of errors result in responses which contain completely fabricated information, mix fact and fiction, or present single facts for which there is no supporting evidence for the assertion. Many of these errors sound very plausible and are only discovered by subsequent verification.

This can mean that an AI model cites a non-existent research paper, incorrectly attributes a statement to someone, invents a new event or even comes up with the most creative but entirely incorrect explanation for a technical term or concept.

It is crucial to distinguish between an AI intentionally producing false information and an AI merely generating information based on learned patterns within the data provided to it. While a model can verify specific information against a trusted source before releasing an answer, the majority of AI models are merely designed to provide the most relevant and highest quality information based on what was asked within the provided context.

Why do AI Hallucinations Occur?

Why Do AI Hallucinations Occur? In AI generative models, the aim is to produce the most likely output based on the patterns within the AI’s training data and the context within the prompt. In many cases, generating a very fluent response does not necessarily mean that every statement in the response is 100% factually correct.

The AI system’s output may also include incomplete information. It is possible for a model’s training data to be incomplete in several ways including missing information, inconsistent information, outdated information and underrepresented groups and classes.

The prompt itself is another point of failure. A poorly worded question, for example, leaves key details to the user’s imagination, which can lead to surprising results. Assumptions embedded in a prompt are also a problem, as the AI system processes them as given. Finally, the lack of context in a prompt can cause the AI to produce responses that contain unsupported information that seems correct but is actually not.

Another important point is that generative models do not function as a database and therefore do not check every generated statement against stored information. Rather, a generative model is a statistical model that predicts the most likely next tokens given a set of prior tokens.

This can also happen when a user asks a very specific question that can’t be answered by the model because it requires knowledge of very specialized information or of recent events that have not yet been recorded.

Types of AI Hallucinations

The different types of hallucinations of an AI system.

A factual hallucination refers to when an AI generates information that is completely incorrect. For example, this may be an incorrect date, completely fabricated statistics, a false quote, non-existent sources or an incorrect description of events that occurred.

A contextual hallucination is when the generated answer is not correct for the given context, even though each generated statement is correct on its own. It means that the answer generated by the model does not match the given prompt or the rest of the conversation.

Consistency hallucination – The system generates inconsistent information (contradictory) within a single response or across many responses (e.g., different descriptions of a person, event, process, etc.).

Of course, these categories are not necessarily exclusive. In practice, there are many scenarios in which the different categories are combined.

Real-World Examples of AI Hallucinations

AI hallucinations can appear in many types of applications.

For example, a language model could be asked to summarize a research paper. The summary would contain findings that were not in the paper. In addition, the model could create a list of references for the paper, with some of the references not actually existing.

In customer support a hallucination could lead to the AI-powered chatbot incorrectly explaining a return policy, or even invent a feature of a product or service that does not in fact exist. The likelihood of such a hallucination can be greatly reduced by ensuring that the chatbot is connected to up-to-date and reliable information about the business or organization.

A coding assistant might also provide the developer with hallucinated information, for example suggesting a software library, a function or API parameter that looks correct but does not in fact exist. The developer can test this generated code against the documentation for that API or library and against his own development environment.

Other examples of areas in which hallucinations can lead to significant problems are the legal, financial, health-care and research domains. As has already been stated, inaccuracies in these domains can have serious consequences that go beyond simple incorrect answers.

Why Are AI Hallucinations a Problem?

The biggest problem with AI-generated hallucinations is that they can seem very credible and look like reasonable answers.

People judge the accuracy of answers in part based on clarity and confidence. For this reason, highly polished AI generated content is particularly problematic until the person generating it has verified its accuracy.

Additionally, businesses have to worry about the potential damage that an AI system could cause. If an AI assistant provides customers with incorrect information, it could create even more problems for customer service. Employees also rely on AI to complete tasks, so there is a risk that they could be misled by inaccurate information retrieved by an AI system or even generated by it.

The consequences of inaccuracies may be especially problematic for professionals in various fields for whom accuracy is paramount, such as health care providers, financial managers, attorneys, researchers, and more. They must scrutinize AI-generated content with the same skepticism with which they would treat any external information source.

As AI is integrated into workflows, the effects of hallucinations can become amplified. An assistant’s mistake may only cause mild irritation for an individual but could have the potential to cause significant problems when integrated into an automated process.

In addition to testing a language model’s ability to generate answers, it is also important to test what happens when that model generates an incorrect answer.

How Can AI Hallucinations Be Reduced?

While there is currently no way to fully eliminate hallucinations from AI-generated content, there are a number of steps and measures which can reduce their frequency and impact.

Retrieval-Augmented Generation (RAG)

Retrieval-augmented generation (RAG) is a form of interaction with AI that can reduce hallucinations by providing a generative model with relevant information retrieved from external sources before it generates output.

This can include documents, databases, as well as knowledge bases that are specific to a company, or general knowledge bases. The model then uses this context in addition to its general knowledge, to generate the best possible answer.

For instance, an enterprise assistant answers questions of employees within a company by first retrieving the most up-to-date internal policy on the matter and then uses this to generate an answer to their question.

Even when Retrieval-Augmented Generation is used to reduce hallucinations, there is no guarantee of accuracy. The AI hallucination may still occur if the Retrieval system itself retrieved irrelevant, incomplete or even outdated information to use as a basis for generation.

Prompt Engineering and Guardrails

Clear prompts that define the scope of the model are an important aspect to ensure that the model behaves as intended.

Instructions can also be used to specify that the AI system should only use the information supplied, identify situations of uncertainty, refrain from making any claims for which there is no support, and seek human intervention when necessary.

Guardrails: These restrictions surrounding the AI model are used to control what type of content it can generate, to validate its output, to restrict the access to sensitive information and to prevent it from doing anything else that could be undesirable.

Prompt design and guardrails are only as good as the overall design of the application into which they are integrated.

Human Review and Fact-Checking

Human review remains important for high-risk applications.

After generation, the automatically produced information is checked against trusted sources. Depending on the potential consequences of incorrect information, the amount of required checks by a human reviewer can vary greatly.

While automated validation may be sufficient for low-risk tasks, a qualified person should review the output of an AI system for higher risk applications before any action is taken.

Monitoring and Evaluation

AI systems need to be tested after they have been deployed, not just during development.

For example, evaluation can look at frequency of unsupported claims, adherence to prompt, accuracy of information retrieval and performance on a variety of different question types.

Real-world interactions can expose issues that in controlled testing weren’t encountered. Subsequently, the AI system’s prompts, the retrieval sources, the evaluation data set, and the system’s controls can be fine-tuned as needed.

AI Hallucination vs. AI Mistake

Hallucination and AI mistake – two terms related to each other but with different focus points.

The term ‘AI hallucination’ and ‘AI mistake’ are often used interchangeably and not all systems would draw a clear distinction between the two. It is, however, generally important to verify important AI system outputs.