Ingest Data with Retry

This recipe demonstrates how to ingest data into Weaviate with retry logic.

Weaviate Setup

The sample code is for a local Weaviate deployment with Docker and Ollama running on localhost. However, the retry logic is agnostic to the deployment method and the underlying vector embedding service and model.

Steps to deploy Weaviate locally with CLIP

We will use docker-compose.yaml and verify that ollama is running. Note that we set ASYNC_INDEXING: 'true' in the weaviate environment variables to speed up the ingestion process.

Run:

docker compose -f docker-compose.yaml up -d ollama serve ollama pull mxbai-embed-large:latest ollama pull llama3.2:latest curl http://localhost:11434

Dependencies

Configuration

Create Product collection

The collection has the following key characteristics:

1. Name: "Product"
2. Vectorizer: text2vec-ollama

Import Product Logic

Some of the data in the Amazon product dataset is stored inconsisntely, so we need to normalize the data before importing it into Weaviate.

Note that the import_products will first import the data, and then enter an infinite loop to retry the objects that failed to import.

Import Products

The below code assumes you have downloaded the Amazon product dataset and split the JSON data into multiple files named amazon_products_00.json, amazon_products_01.json, etc.

See the README for more details.

Verify the number of objects in the Product collection

There should be 10,000 objects in the Product collection after ingestion.

Run a vector search query