Notebooks
W
Weaviate Rag Databricks

Rag Databricks

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksweaviate-featuresfunction-callingweaviate-recipesmodel-providersPythongenerative-aidatabricks
alph-notebooks/weaviate-recipes / rag_databricks.ipynb

Open In Colab

Dependencies

Check that the weaviate-client and spark connector is installed:

[0]
Name: weaviate-client
Version: 4.8.1
Summary: A python native Weaviate client
Home-page: https://github.com/weaviate/weaviate-python-client
Author: Weaviate
Author-email: hello@weaviate.io,
License: BSD 3-clause
Location: /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages
Requires: authlib, grpcio, grpcio-health-checking, grpcio-tools, httpx, pydantic, requests, validators
Required-by:
[0]

Imports

[0]

Weaviate Cloud Setup

Grab some secrets:

[0]

Connect to weaviate and clear all collections:

[0]
<frozen importlib._bootstrap>:1049: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
[0]
Weaviate server version: 1.26.4

Dataset

We'll ingest a job posting dataset stored in our catalog into Weaviate:

[0]

Add a uuid to make sure each row is unique:

[0]

Ingest

Create the collection in Weaviate:

[0]

Write the dataframe to Weaviate:

[0]

Sanity Checks

Check that the collection has 10 objects:

[0]

Check that the object has a vector:

[0]

Randomly sample the dataframe and check each row is identifcal in weaviate:

[0]

Search

Use hybrid search to search for a job:

[0]

Generative Search

Ask the LLM to translate each job:

[0]
| Job Title | German Translation |
|---|---|
| Laundry Worker | Wäschereiarbeiter |
| Wire Drawer Technician | Drahtzieh-Techniker |
| Resident Care Companion or STNA - DSL Elyria | Pflegebegleiter oder STNA - DSL Elyria |

Upsert

Change a field in a row and upload it to weaviate:

[0]
<frozen importlib._bootstrap>:1049: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
/usr/lib/python3.11/socket.py:776: ResourceWarning: unclosed <socket.socket fd=103, family=2, type=1, proto=6, laddr=('127.0.0.1', 54442), raddr=('127.0.0.1', 36897)>
  self._sock = None
ResourceWarning: Enable tracemalloc to get the object allocation traceback

original row:
+--------------------+-------------------+--------------------+
|                uuid|              title|         description|
+--------------------+-------------------+--------------------+
|52fad351-0bef-555...|DELIVERY TECHNICIAN|Our Company\n\nAm...|
+--------------------+-------------------+--------------------+

updated row:
+--------------------+-----+--------------------+
|                uuid|title|         description|
+--------------------+-----+--------------------+
|52fad351-0bef-555...|  foo|Our Company\n\nAm...|
+--------------------+-----+--------------------+

updated object in weaviate:
uuid: 52fad351-0bef-5550-8af5-3d1f89a3fcf8, title: foo

Error Handling

Make sure the collection exists in weaviate before writing to it:

[0]
WeaviateClassNotFoundError: The specified class 'JobAds2' was not found.

<frozen importlib._bootstrap>:1049: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()

The dataframe must contain all the properties defined by the collection in weaviate, otherwise it will throw a CANNOT_FIND_DATA error:

[0]

If disk usage of a node in the Weaviate cluster is higher than the DISK_USE_READONLY_PERCENTAGE percentage, then all shards on the affected node will be marked as READONLY, meaning all future write requests will fail.

To recover from this, increase the node's disk capacity and mark the shards as ready again. Refer to the weaviate docs for detailed steps

Further Reading