Hello Llama Cloud

This demo app shows:

• How to run Llama 3.1 in the cloud hosted on Replicate
• How to use LangChain to ask Llama general questions and follow up questions
• How to use LangChain to load a recent web page - Hugging Face's blog post on Llama 3.1 - and chat about it. This is the well known RAG (Retrieval Augmented Generation) method to let LLM such as Llama 3 be able to answer questions about the data not publicly available when Llama 3 was trained, or about your own data. RAG is one way to prevent LLM's hallucination

Note We will be using Replicate to run the examples here. You will need to first sign in with Replicate with your github account, then create a free API token here that you can use for a while. You can also use other Llama 3.1 cloud providers such as Groq, Together, or Anyscale - see Section 2 of the Getting to Know Llama notebook for more information.

Let's start by installing the necessary packages:

• sentence-transformers for text embeddings
• FAISS gives us database capabilities
• LangChain provides necessary RAG tools for this demo

Next we call the Llama 3.1 405b chat model from Replicate. You can also use Llama 3 8B or 70B model by replacing the model name with the respective model URL(s).

With the model set up, you are now ready to ask some questions. Here is an example of the simplest way to ask the model some general questions.

We will then try to follow up the response with a question asking for more information on the book.

Since the chat history is not passed on Llama doesn't have the context and doesn't know this is more about the book thus it treats this as new query.

To get around this we will need to provide the model with history of the chat.

To do this, we will use ConversationBufferMemory to pass the chat history to the model and give it the capability to handle follow up questions.

Once this is set up, let us repeat the steps from before and ask the model a simple question.

Then we pass the question and answer back into the model for context along with the follow up question.

Next, let's explore using Llama 3.1 to answer questions using documents for context. This gives us the ability to update Llama 3.1's knowledge thus giving it better context without needing to finetune.

We need to store our document in a vector store. There are more than 30 vector stores (DBs) supported by LangChain. For this example we will use FAISS, a popular open source vector store by Facebook. For other vector stores especially if you need to store a large amount of data - see here.

We will also import the HuggingFaceEmbeddings and RecursiveCharacterTextSplitter to assist in storing the documents.

To store the documents, we will need to split them into chunks using RecursiveCharacterTextSplitter and create vector representations of these chunks using HuggingFaceEmbeddings on them before storing them into our vector database.

In general, you should use larger chuck sizes for highly structured text such as code and smaller size for less structured text. You may need to experiment with different chunk sizes and overlap values to find out the best numbers.

We then use RetrievalQA to retrieve the documents from the vector database and give the model more context on Llama 3.1, thereby increasing its knowledge. 3.1 also really shines with the new 128k context!

For each question, LangChain performs a semantic similarity search of it in the vector db, then passes the search results as the context to Llama to answer the question.

Now, lets bring it all together by incorporating follow up questions.

First we ask a follow up questions without giving the model context of the previous conversation. Without this context, the answer we get does not relate to our original question.

As we did before, let us use the ConversationalRetrievalChain package to give the model context of our previous question so we can add follow up questions.

Note: If results can get cut off, you can set "max_new_tokens" in the Replicate call above to a larger number (like shown below) to avoid the cut off.

model_kwargs={"temperature": 0.01, "top_p": 1, "max_new_tokens": 1000}