Chapter 8: Avoiding Hallucinations

• Lesson
• Exercises
• Example Playground

Setup

Run the following setup cell to load your API key and establish the get_completion helper function.

Lesson

Some bad news: Claude sometimes "hallucinates" and makes claims that are untrue or unjustified. The good news: there are techniques you can use to minimize hallucinations.

Below, we'll go over a few of these techniques, namely:

• Giving Claude the option to say it doesn't know the answer to a question
• Asking Claude to find evidence before answering

However, there are many methods to avoid hallucinations, including many of the techniques you've already learned in this course. If Claude hallucinates, experiment with multiple techniques to get Claude to increase its accuracy.

Examples

Here is a question about general factual knowledge in answer to which Claude hallucinates several large hippos because it's trying to be as helpful as possible.

A solution we can try here is to "give Claude an out" — tell Claude that it's OK for it to decline to answer, or to only answer if it actually knows the answer with certainty.

In the prompt below, we give Claude a long document containing some "distractor information" that is almost but not quite relevant to the user's question. Without prompting help, Claude falls for the distractor information and gives an incorrect "hallucinated" answer as to the size of Matterport's subscriber base as of May 31, 2020.

Note: As you'll learn later in the next chapter, it's best practice to have the question at the bottom after any text or document, but we put it at the top here to make the prompt easier to read. Feel free to double click on the prompt cell to get the full prompt text (it's very long!).

How do we fix this? Well, a great way to reduce hallucinations on long documents is to make Claude gather evidence first.

In this case, we tell Claude to first extract relevant quotes, then base its answer on those quotes. Telling Claude to do so here makes it correctly notice that the quote does not answer the question.

Bonus lesson

Sometimes, Claude's hallucinations can be solved by lowering the temperature of Claude's responses. Temperature is a measurement of answer creativity between 0 and 1, with 1 being more unpredictable and less standardized, and 0 being the most consistent.

Asking Claude something at temperature 0 will generally yield an almost-deterministic answer set across repeated trials (although complete determinism is not guaranteed). Asking Claude something at temperature 1 (or gradations in between) will yield more variable answers. Learn more about temperature and other parameters here.

If you would like to experiment with the lesson prompts without changing any content above, scroll all the way to the bottom of the lesson notebook to visit the Example Playground.

Exercises

• Exercise 8.1 - Beyoncé Hallucination
• Exercise 8.2 - Prospectus Hallucination

Exercise 8.1 - Beyoncé Hallucination

Modify the PROMPT to fix Claude's hallucination issue by giving Claude an out. (Renaissance is Beyoncé's seventh studio album, not her eigthth.)

We suggest you run the cell first to see what Claude hallucinates before trying to fix it.

❓ If you want a hint, run the cell below!

Exercise 8.1 - Prospectus Hallucination

Modify the PROMPT to fix Claude's hallucination issue by asking for citations. The correct answer is that subscribers went up 49x.

❓ If you want a hint, run the cell below!

Congrats!

If you've solved all exercises up until this point, you're ready to move to the next chapter. Happy prompting!

Example Playground

This is an area for you to experiment freely with the prompt examples shown in this lesson and tweak prompts to see how it may affect Claude's responses.