Notebooks
W
Weaviate Hybrid Search Artic Embed S

Hybrid Search Artic Embed S

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksweaviate-featuresfunction-callingweaviate-recipesmodel-providersPythongenerative-aisnowflake
alph-notebooks/weaviate-recipes / hybrid_search_artic_embed_s.ipynb

text2vec-transformers is only available through Weaviate open-source. Here are options to select your desired model:

  1. Pre-built transformers model containers

  2. Any model from Hugging Face Model Hub

  3. Use any private or local PyTorch or Tensorflow transformer model

Dependencies

[ ]

Connect to Weaviate

[3]
/var/folders/gr/_p7k3mjs6fscm89t6353bfrr0000gn/T/ipykernel_56532/4278117560.py:4: ResourceWarning: unclosed <socket.socket fd=75, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('::1', 64838, 0, 0), raddr=('::1', 8080, 0, 0)>
  client = weaviate.connect_to_local(
ResourceWarning: Enable tracemalloc to get the object allocation traceback

True
[4]
Timeout(query=900, insert=1200, init=2)

Create a collection

Collection stores your data and vector embeddings.

[5]
Successfully created collection: JeopardyQuestion.

Import the Data

[6]
Insert complete.

Hybrid Search

The alpha parameter determines the weight given to the sparse and dense search methods. alpha = 0 is pure sparse (bm25) search, whereas alpha = 1 is pure dense (vector) search.

Alpha is an optional parameter. The default is set to 0.75.

Hybrid Search only

The below query is finding Jeopardy questions about animals and is limiting the output to only two results. Notice alpha is set to 0.80, which means it is weighing the vector search results more than bm25. If you were to set alpha = 0.25, you would get different results.

[7]
ID: 1e895b51-a59e-4de4-bdf9-f22ece9180d4
Data: {
  "answer": "species",
  "question": "2000 news: the Gunnison sage grouse isn't just another northern sage grouse, but a new one of this classification",
  "category": "SCIENCE"
} 

ID: 73fb3875-3783-4333-978f-7f2e3888aecf
Data: {
  "answer": "wire",
  "question": "A metal that is \"ductile\" can be pulled into this while cold & under pressure",
  "category": "SCIENCE"
} 

ID: 71df197f-a667-4b51-9e58-4e9c55531429
Data: {
  "answer": "the diamondback rattler",
  "question": "Heaviest of all poisonous snakes is this North American rattlesnake",
  "category": "ANIMALS"
}

Hybrid Search on a specific property

The properties parameter allows you to list the properties that you want bm25 to search on.

[8]
ID: 1e895b51-a59e-4de4-bdf9-f22ece9180d4
Data: {
  "answer": "species",
  "question": "2000 news: the Gunnison sage grouse isn't just another northern sage grouse, but a new one of this classification",
  "category": "SCIENCE"
} 

ID: 73fb3875-3783-4333-978f-7f2e3888aecf
Data: {
  "answer": "wire",
  "question": "A metal that is \"ductile\" can be pulled into this while cold & under pressure",
  "category": "SCIENCE"
} 

ID: 71df197f-a667-4b51-9e58-4e9c55531429
Data: {
  "answer": "the diamondback rattler",
  "question": "Heaviest of all poisonous snakes is this North American rattlesnake",
  "category": "ANIMALS"
}

Hybrid Search with a where filter

Find Jeopardy questions about elephants, where the category is set to Animals.

[9]
ID: 71df197f-a667-4b51-9e58-4e9c55531429
Data: {
  "answer": "the diamondback rattler",
  "question": "Heaviest of all poisonous snakes is this North American rattlesnake",
  "category": "ANIMALS"
} 

ID: bdc0d9de-f263-4faf-a9f7-eedd655123b4
Data: {
  "answer": "Elephant",
  "question": "It's the only living mammal in the order Proboseidea",
  "category": "ANIMALS"
} 

ID: d829d2e7-c4fd-4a1e-8607-ade70da7b816
Data: {
  "answer": "the nose or snout",
  "question": "The gavial looks very much like a crocodile except for this bodily feature",
  "category": "ANIMALS"
}

Hybrid Search with a custom vector

You can pass in your own vector as input into the hybrid query, by using the vector parameter.

[10]
ID: ee3b9fd9-92ef-4122-ad1a-375cbcaccf54
Data: {
  "answer": "Antelope",
  "question": "Weighing around a ton, the eland is the largest species of this animal in Africa",
  "category": "ANIMALS"
} 

ID: bdc0d9de-f263-4faf-a9f7-eedd655123b4
Data: {
  "answer": "Elephant",
  "question": "It's the only living mammal in the order Proboseidea",
  "category": "ANIMALS"
}