Llama Index Multimodal Image Reasoning
Multimodal LLM Ops - Tracing, Evaluation, and Analysis of Multimodal Models
In this notebook, we show how to use a Mult-Modal LLM, i.e., OpenAI's gpt-4o, to ask questions about images (image reasoning) using the chat API. In addition, we use Arize's Phoenix and OpenInference AutoInstrumentor to trace the operation.
- Framework: LlamaIndex
- LLM: OpenAI's GPT-4o
- LLM Observability: Arize Phoenix (GitHub)
- LLM Tracing: Arize's OpenInference Auto-Instrumentor
Steps:
- Install dependencies
- Setup Tracing
- Download Images from Tesla
- Setup the Multi-Modal LLM application
- Use the Multi-Modal LLM application
⚠️ This tutorial requires an OpenAI key to run
Install dependencies
Setup Tracing
First, we launch the phoenix app, which will act as an OTEL collector of the generated spans.
Next, we setup the tracing by declaring a tracer provider and a span processor with an OTLP span exporter:
Finally, we use OpenInference's Llama-Index auto-instrumentor.
That's it! With these 2 cells you have correctly set up the tracing of your Llama-Index application. As you use this application, spans will be exported to Phoenix for observability and analysis.
Download images from Tesla's website
Next, we simply plot the images so you know what we just downloaded
Setup the Multi-Modal LLM application
First things first, we need an OpenAI (our LLM provider) API key.
Last, we need to declare our OpenAI from the multi_nmodal_llms module and use a SimpleDirectoryReader to have access to the downloaded images
Use the Multi-Modal LLM application
We set up multimodal chat messages and call the LLM
We can also simulate a conversation by passing the response as a message from the "assistant", and ask further questions
Let's try make the last question more difficult. We can ask the last question without directly passing the image documents. Sometimes the LLM will rembemer the images passed in the first message and responde correctly. However, some other times it will be unaware of them and give an incomplete or wrong answer.
Observability
Now that we've run the application a couple of times, let's take a look at the traces in the UI:
The UI will give you an interactive troubleshooting experience. You can sort, filter, and search for traces. You can also view the questions asked and the images in the message. For instance you can see how in the second trace, the images were passed to every user message, but in the third trace only the first message had the images attached.