Notebooks
M
Meta Llama Part 3 RAG Setup And Validation

Part 3 RAG Setup And Validation

llamaMulti-Modal-RAGAIvllmmachine-learningend-to-end-use-casesllama2LLMllama-cookbooknotebooksPythonfinetuningpytorchlangchain
alph-notebooks/llama-cookbook / Part_3_RAG_Setup_and_Validation.ipynb

Part 3: Setting up RAG Example and Validating our Retrieval Pipeline

We are ready for the finale, but let's recap:

  • We started with an example dataset of 5000 images
  • In the first notebook, we cleaned this up for labelling and used Llama-3.2-11B model for labelling
  • In the second notebook, we cleaned up some hallucinations of the model and pre-processed the descriptions that were synthetically generated

Step 3 is to setup a RAG pipeline and profit.

We will use lance-db since it's open source and Llama and open source go well together 🤝

We also love free API credits and Together is hosting 11B model for free. For our final demo, we will use their API and validate the same in this example.

[12]

Loading the Dataset and Creating Embeddings

[21]
[22]
[24]
'This white tank top is a stylish and trendy piece of clothing that features a captivating celestial design. The design showcases a mesmerizing array of stars, planets, and moons in black and white, creating a visually appealing and eye-catching pattern. The tank top has a relaxed fit with a round neckline and sleeveless design, making it a comfortable and versatile piece that can be worn on its own or layered under a cardigan or jacket. Perfect for a casual day out or a night on the town, this tank top is a must-have for any fashion-conscious woman.'

Creating Embeddings:

We will define a Schema and use BAAI/bge-small-en-v1.5 embeddings to create our vector embeddings. This is a first step, we can iterate with more embeddings in our final app later.

For retrieval, we will embed descriptions of all clothes and use these.

[37]
Number of empty descriptions: 0
[38]
[39]

Approach 1/3: Using Text Search

The approach we will take first is:

  • Upload an image and ask 3.2-11B-Instruct from Together to describe the image
  • We will then try to find similar images in the dataset

Note: This is to validate that our pipeline is working, in the final demo-we prompt Llama to find complementary clothes.

[43]
Jeans are a classic pairing option. Dark wash skinny jeans create a sleek look, while light wash boyfriend jeans add a casual touch. Black trousers offer a dressier alternative. Khaki chinos provide a neutral background for the plaid pattern. Corduroy pants add a rustic feel. Wool trousers bring a sophisticated element. Leather pants create a bold statement.
[44]
[45]
[46]
Output
[47]
Output
[48]
Output

Approach 2/3: Using Full Text-Search

[128]
[139]
[140]
[145]
Output
[146]
Output
[148]
Output

Approach 3/3: Using Hybrid search

[153]
Loading ColBERTRanker model colbert-ir/colbertv2.0
No device set
Using device cuda
No dtype set
Using dtype torch.float16
Loading model colbert-ir/colbertv2.0, this might take a while...
Linear Dim set to: {linear_dim} for downcasting
[154]
[155]
Output
[156]
Output
[ ]