Notebooks
M
Milvus Mistral Ocr With Milvus

Mistral Ocr With Milvus

image-searchvector-databasesemantic-searchIntegrationmilvusembeddingsunstructured-dataquestion-answeringLLMmilvus-bootcampdeep-learningimage-recognitionimage-classificationaudio-searchPythonragNLP
alph-notebooks/milvus-bootcamp / mistral_ocr_with_milvus.ipynb

Document Understanding with Mistral OCR and Milvus

This tutorial demonstrates how to build a document understanding system using:

Mistral OCR

A powerful optical character recognition service that:

  • Processes PDFs, images, and other document formats
  • Preserves document structure and formatting
  • Handles multi-page documents
  • Recognizes tables, lists, and other complex elements

Mistral Embeddings

  • Transforms text into numerical representations:
  • Converts text into 1024-dimensional vectors
  • Captures semantic relationships between concepts
  • Enables similarity matching based on meaning
  • Provides foundation for semantic search

Milvus Vector Database

Specialized database for vector similarity search:

  • Open-Source
  • Performs efficient Vector Search
  • Scales to large document collections
  • Supports hybrid search (vector similarity + metadata filtering)
  • Optimized for AI applications

By the end of this tutorial, you'll have a system that can:

  1. Process documents (PDFs/images) via URLs
  2. Extract text using OCR
  3. Store the text and vector embeddings in Milvus
  4. Perform semantic search across your document collection

Setup and Dependencies

First, let's install the required packages:

[1]
Requirement already satisfied: mistralai in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (1.5.1)
Requirement already satisfied: pymilvus in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (2.5.3)
Requirement already satisfied: python-dotenv in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (1.0.1)
Requirement already satisfied: eval-type-backport>=0.2.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from mistralai) (0.2.2)
Requirement already satisfied: httpx>=0.27.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from mistralai) (0.28.1)
Requirement already satisfied: jsonpath-python>=1.0.6 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from mistralai) (1.0.6)
Requirement already satisfied: pydantic>=2.9.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from mistralai) (2.10.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from mistralai) (2.9.0.post0)
Requirement already satisfied: typing-inspect>=0.9.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from mistralai) (0.9.0)
Requirement already satisfied: setuptools>69 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pymilvus) (75.6.0)
Requirement already satisfied: grpcio<=1.67.1,>=1.49.1 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pymilvus) (1.67.1)
Requirement already satisfied: protobuf>=3.20.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pymilvus) (5.29.2)
Requirement already satisfied: ujson>=2.0.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pymilvus) (5.10.0)
Requirement already satisfied: pandas>=1.2.4 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pymilvus) (2.2.3)
Requirement already satisfied: milvus-lite>=2.4.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pymilvus) (2.4.11)
Requirement already satisfied: anyio in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from httpx>=0.27.0->mistralai) (4.7.0)
Requirement already satisfied: certifi in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from httpx>=0.27.0->mistralai) (2024.2.2)
Requirement already satisfied: httpcore==1.* in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from httpx>=0.27.0->mistralai) (1.0.7)
Requirement already satisfied: idna in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from httpx>=0.27.0->mistralai) (3.6)
Requirement already satisfied: h11<0.15,>=0.13 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from httpcore==1.*->httpx>=0.27.0->mistralai) (0.14.0)
Requirement already satisfied: tqdm in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from milvus-lite>=2.4.0->pymilvus) (4.67.1)
Requirement already satisfied: numpy>=1.26.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pandas>=1.2.4->pymilvus) (2.2.1)
Requirement already satisfied: pytz>=2020.1 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pandas>=1.2.4->pymilvus) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pandas>=1.2.4->pymilvus) (2024.2)
Requirement already satisfied: annotated-types>=0.6.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pydantic>=2.9.0->mistralai) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.2 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pydantic>=2.9.0->mistralai) (2.27.2)
Requirement already satisfied: typing-extensions>=4.12.2 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from pydantic>=2.9.0->mistralai) (4.12.2)
Requirement already satisfied: six>=1.5 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from python-dateutil>=2.8.2->mistralai) (1.17.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from typing-inspect>=0.9.0->mistralai) (1.0.0)
Requirement already satisfied: sniffio>=1.1 in /Users/stephen/.pyenv/versions/3.12.2/lib/python3.12/site-packages (from anyio->httpx>=0.27.0->mistralai) (1.3.1)

[notice] A new release of pip is available: 24.0 -> 25.0.1
[notice] To update, run: pip install --upgrade pip

Environment Setup

You'll need:

  1. A Mistral API key (get one at https://console.mistral.ai/)
  2. Milvus running locally through Docker or with Zilliz Cloud

Let's set up our environment:

[1]
Connected to Mistral API and Milvus at http://localhost:19530

Setting Up Milvus Collection

Now, let's create a Milvus collection to store our document data. The collection will have the following fields:

  • id: Primary key (auto-generated)
  • url: Source URL of the document
  • page_num: Page number within the document
  • content: Extracted text content
  • embedding: Vector representation of the content (1024 dimensions)
[2]
Collection 'document_ocr' already exists.

Core Functionality

Let's implement the core functions for our document understanding system:

[3]

Search Functionality

Now, let's implement the search functionality to retrieve relevant document content:

[4]

Demo: Processing Documents

Let's process some example documents. You can replace these URLs with your own documents.

[5]
Processing document: https://arxiv.org/pdf/2310.06825.pdf
Stored 9 pages from https://arxiv.org/pdf/2310.06825.pdf in Milvus.

OCR Result Preview:
====================
### Page 1
# Mistral 7B 

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed

![img-0.jpeg](img-0.jpeg)


#### Abstract

We introduce Mistral 7B, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms the best open 13B model (Llama 2) across all evaluated benchmarks, and the best released 34B model (Llama 1) in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B - Instruct, that surpasses Llama 2 13B - chat mod...

Let's process an image as well:

[6]
Processing document: https://s3.eu-central-1.amazonaws.com/readcoop.cis.public-assets.prod/hero/old-german-scripts.png
Stored 1 pages from https://s3.eu-central-1.amazonaws.com/readcoop.cis.public-assets.prod/hero/old-german-scripts.png in Milvus.

Image OCR Result:
=================
### Page 1
![img-0.jpeg](img-0.jpeg)
![img-1.jpeg](img-1.jpeg)
![img-2.jpeg](img-2.jpeg)
![img-3.jpeg](img-3.jpeg)
![img-4.jpeg](img-4.jpeg)
![img-5.jpeg](img-5.jpeg)
![img-6.jpeg](img-6.jpeg)
![img-7.jpeg](img-7.jpeg)
![img-8.jpeg](img-8.jpeg)
![img-9.jpeg](img-9.jpeg)
![img-10.jpeg](img-10.jpeg)
![img-11.jpeg](img-11.jpeg)
![img-12.jpeg](img-12.jpeg)
![img-13.jpeg](img-13.jpeg)
![img-14.jpeg](img-14.jpeg)
![img-15.jpeg](img-15.jpeg)
![img-16.jpeg](img-16.jpeg)
![img-17.jpeg](img-17.jpeg)
![img-18.jpeg](img-18.jpeg)
![img-19.jpeg](img-19.jpeg)
![img-20.jpeg](img-20.jpeg)
![img-21.jpeg](img-21.jpeg)
![img-22.jpeg](img-22.jpeg)
![img-23.jpeg](img-23.jpeg)
![img-24.jpeg](img-24.jpeg)
![img-25.jpeg](img-25.jpeg)
![img-26.jpeg](img-26.jpeg)
![img-27.jpeg](img-27.jpeg)
![img-28.jpeg](img-28.jpeg)
![img-29.jpeg](img-29.jpeg)
![img-30.jpeg](img-30.jpeg)

Demo: Searching Documents

Now that we've processed some documents, let's search through them:

[7]
Total pages stored: 58
Unique documents: 3

Processed documents:
1. https://arxiv.org/pdf/2310.06825.pdf
2. https://s3.eu-central-1.amazonaws.com/readcoop.cis.public-assets.prod/hero/old-german-scripts.png
3. https://arxiv.org/pdf/2410.07073
[8]
Searching for: What is Mistral 7B?
Search results for: 'What is Mistral 7B?'

Result 1 (Score: 0.83)
Source: https://arxiv.org/pdf/2310.06825.pdf (Page 2)
Content: Mistral 7B is released under the Apache 2.0 license. This release is accompanied by a reference implementation ${ }^{1}$ facilitating easy deployment either locally or on cloud platforms such as AWS, GCP, or Azure using the vLLM [17] inference server and SkyPilot ${ }^{2}$. Integration with Hugging Face ${ }^{3}$ is also streamlined for easier integration. Moreover, Mistral 7B is crafted for ease of fine-tuning across a myriad of tasks. As a demonstration of its adaptability and superior perform...

Result 2 (Score: 0.83)
Source: https://arxiv.org/pdf/2310.06825.pdf (Page 2)
Content: Mistral 7B is released under the Apache 2.0 license. This release is accompanied by a reference implementation ${ }^{1}$ facilitating easy deployment either locally or on cloud platforms such as AWS, GCP, or Azure using the vLLM [17] inference server and SkyPilot ${ }^{2}$. Integration with Hugging Face ${ }^{3}$ is also streamlined for easier integration. Moreover, Mistral 7B is crafted for ease of fine-tuning across a myriad of tasks. As a demonstration of its adaptability and superior perform...

Result 3 (Score: 0.82)
Source: https://arxiv.org/pdf/2310.06825.pdf (Page 1)
Content: # Mistral 7B 

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed

![img-0.jpeg](img-0.jpeg)


#### Abstract

We introduce Mistral 7B, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7...

Try another search query:

[9]
Searching for: What are the capabilities of Mistral's language models?
Search results for: 'What are the capabilities of Mistral's language models?'

Result 1 (Score: 0.85)
Source: https://arxiv.org/pdf/2310.06825.pdf (Page 2)
Content: Mistral 7B is released under the Apache 2.0 license. This release is accompanied by a reference implementation ${ }^{1}$ facilitating easy deployment either locally or on cloud platforms such as AWS, GCP, or Azure using the vLLM [17] inference server and SkyPilot ${ }^{2}$. Integration with Hugging Face ${ }^{3}$ is also streamlined for easier integration. Moreover, Mistral 7B is crafted for ease of fine-tuning across a myriad of tasks. As a demonstration of its adaptability and superior perform...

Result 2 (Score: 0.85)
Source: https://arxiv.org/pdf/2310.06825.pdf (Page 2)
Content: Mistral 7B is released under the Apache 2.0 license. This release is accompanied by a reference implementation ${ }^{1}$ facilitating easy deployment either locally or on cloud platforms such as AWS, GCP, or Azure using the vLLM [17] inference server and SkyPilot ${ }^{2}$. Integration with Hugging Face ${ }^{3}$ is also streamlined for easier integration. Moreover, Mistral 7B is crafted for ease of fine-tuning across a myriad of tasks. As a demonstration of its adaptability and superior perform...

Result 3 (Score: 0.84)
Source: https://arxiv.org/pdf/2310.06825.pdf (Page 6)
Content: | Model | Answer |
| :--: | :--: |
| Mistral 7B - Instruct with Mistral system prompt | To kill a Linux process, you can use the `kill' command followed by the process ID (PID) of the process you want to terminate. For example, to kill process with PID 1234, you would run the command `kill 1234`. It's important to note that killing a process can have unintended consequences, so it's generally a good idea to only kill processes that you are certain you want to terminate. Additionally, it's genera...

Conclusion

In this tutorial, we've built a complete document understanding system using Mistral OCR and Milvus. This system can:

  1. Process documents from URLs
  2. Extract text using Mistral's OCR capabilities
  3. Generate vector embeddings for the content
  4. Store both text and vectors in Milvus
  5. Perform semantic search across all processed documents

This approach enables powerful document understanding capabilities that go beyond simple keyword matching, allowing users to find information based on meaning rather than exact text matches.

⭐️ Github

We hope you liked this tutorial showcasing how to use Milvus with OpenAI Agents. If you liked it and our project, please give us a star on Github! ⭐

image.png

🤝 Add me on Linkedin!

If you have some questions related to Milvus, GenAI, etc, I am Stephen Batifol, you can add me on LinkedIn and I'll gladly help you.

image.png

💬 Join our Discord

If you're interested in learning more about Milvus or you wanna share some feedback, feel free to join our Discord channel.