Notebooks
W
Weaviate Hybrid Search Embed Multilingual V3.0

Hybrid Search Embed Multilingual V3.0

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksweaviate-featuresfunction-callingcohereweaviate-recipesmodel-providersPythongenerative-ai
alph-notebooks/weaviate-recipes / hybrid_search_embed_multilingual_v3.0.ipynb

Open In Colab

Dependencies

[ ]
Collecting git+https://github.com/weaviate/weaviate-python-client.git@fd9dbdda75b64d59ec7742a0e95157a9a608a0de
  Cloning https://github.com/weaviate/weaviate-python-client.git (to revision fd9dbdda75b64d59ec7742a0e95157a9a608a0de) to /private/var/folders/20/sww521bs0s71v4pt86_05g340000gn/T/pip-req-build-9lgynotr
  Running command git clone --filter=blob:none --quiet https://github.com/weaviate/weaviate-python-client.git /private/var/folders/20/sww521bs0s71v4pt86_05g340000gn/T/pip-req-build-9lgynotr
  Running command git rev-parse -q --verify 'sha^fd9dbdda75b64d59ec7742a0e95157a9a608a0de'
  Running command git fetch -q https://github.com/weaviate/weaviate-python-client.git fd9dbdda75b64d59ec7742a0e95157a9a608a0de
  Running command git checkout -q fd9dbdda75b64d59ec7742a0e95157a9a608a0de
  Resolved https://github.com/weaviate/weaviate-python-client.git to commit fd9dbdda75b64d59ec7742a0e95157a9a608a0de
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests<3.0.0,>=2.30.0 in /opt/homebrew/lib/python3.11/site-packages (from weaviate-client==4.9.6.dev2+gfd9dbdda) (2.31.0)
Requirement already satisfied: httpx<=0.27.0,>=0.25.0 in /opt/homebrew/lib/python3.11/site-packages (from weaviate-client==4.9.6.dev2+gfd9dbdda) (0.27.0)
Requirement already satisfied: validators==0.34.0 in /opt/homebrew/lib/python3.11/site-packages (from weaviate-client==4.9.6.dev2+gfd9dbdda) (0.34.0)
Requirement already satisfied: authlib<1.3.2,>=1.2.1 in /opt/homebrew/lib/python3.11/site-packages (from weaviate-client==4.9.6.dev2+gfd9dbdda) (1.2.1)
Requirement already satisfied: pydantic<3.0.0,>=2.5.0 in /opt/homebrew/lib/python3.11/site-packages (from weaviate-client==4.9.6.dev2+gfd9dbdda) (2.6.2)
Requirement already satisfied: grpcio<2.0.0,>=1.57.0 in /opt/homebrew/lib/python3.11/site-packages (from weaviate-client==4.9.6.dev2+gfd9dbdda) (1.64.1)
Requirement already satisfied: grpcio-tools<2.0.0,>=1.57.0 in /opt/homebrew/lib/python3.11/site-packages (from weaviate-client==4.9.6.dev2+gfd9dbdda) (1.59.0)
Requirement already satisfied: grpcio-health-checking<2.0.0,>=1.57.0 in /opt/homebrew/lib/python3.11/site-packages (from weaviate-client==4.9.6.dev2+gfd9dbdda) (1.60.0)
Requirement already satisfied: cryptography>=3.2 in /opt/homebrew/lib/python3.11/site-packages (from authlib<1.3.2,>=1.2.1->weaviate-client==4.9.6.dev2+gfd9dbdda) (41.0.4)
Requirement already satisfied: protobuf>=4.21.6 in /opt/homebrew/lib/python3.11/site-packages (from grpcio-health-checking<2.0.0,>=1.57.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (4.25.1)
Requirement already satisfied: setuptools in /opt/homebrew/lib/python3.11/site-packages (from grpcio-tools<2.0.0,>=1.57.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (69.1.1)
Requirement already satisfied: anyio in /opt/homebrew/lib/python3.11/site-packages (from httpx<=0.27.0,>=0.25.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (4.3.0)
Requirement already satisfied: certifi in /opt/homebrew/lib/python3.11/site-packages (from httpx<=0.27.0,>=0.25.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (2024.2.2)
Requirement already satisfied: httpcore==1.* in /opt/homebrew/lib/python3.11/site-packages (from httpx<=0.27.0,>=0.25.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (1.0.4)
Requirement already satisfied: idna in /opt/homebrew/lib/python3.11/site-packages (from httpx<=0.27.0,>=0.25.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (3.6)
Requirement already satisfied: sniffio in /opt/homebrew/lib/python3.11/site-packages (from httpx<=0.27.0,>=0.25.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (1.3.1)
Requirement already satisfied: h11<0.15,>=0.13 in /opt/homebrew/lib/python3.11/site-packages (from httpcore==1.*->httpx<=0.27.0,>=0.25.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (0.14.0)
Requirement already satisfied: annotated-types>=0.4.0 in /opt/homebrew/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.5.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (0.6.0)
Requirement already satisfied: pydantic-core==2.16.3 in /opt/homebrew/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.5.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (2.16.3)
Requirement already satisfied: typing-extensions>=4.6.1 in /opt/homebrew/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.5.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (4.10.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/homebrew/lib/python3.11/site-packages (from requests<3.0.0,>=2.30.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/homebrew/lib/python3.11/site-packages (from requests<3.0.0,>=2.30.0->weaviate-client==4.9.6.dev2+gfd9dbdda) (2.2.1)
Requirement already satisfied: cffi>=1.12 in /opt/homebrew/lib/python3.11/site-packages (from cryptography>=3.2->authlib<1.3.2,>=1.2.1->weaviate-client==4.9.6.dev2+gfd9dbdda) (1.16.0)
Requirement already satisfied: pycparser in /opt/homebrew/lib/python3.11/site-packages (from cffi>=1.12->cryptography>=3.2->authlib<1.3.2,>=1.2.1->weaviate-client==4.9.6.dev2+gfd9dbdda) (2.21)

[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: python3.11 -m pip install --upgrade pip

You will need a weaviate-client version 4.9.6 or higher.

[2]
Name: weaviate-client
Version: 4.9.6.dev2+gfd9dbdda
Summary: A python native Weaviate client
Home-page: https://github.com/weaviate/weaviate-python-client
Author: Weaviate
Author-email: hello@weaviate.io,
License: BSD 3-clause
Location: /opt/homebrew/lib/python3.11/site-packages
Requires: authlib, grpcio, grpcio-health-checking, grpcio-tools, httpx, pydantic, requests, validators
Required-by: langchain-weaviate, llama-index-vector-stores-weaviate, weaviate-cli

Connect to Weaviate

You will need Weaviate v1.28 or higher. You can run

docker run -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:1.28.1
```
[3]
True

Create a collection

Collection stores your data and vector embeddings.

The collection below uses the "Kagome" tokenizer option to tokenize Japanese sentences. (by adding tokenization=wc.Tokenization.KAGOME_JA to the properties)

[ ]
Successfully created collection: JeopardyQuestion.

Import Data

[5]
[6]
19
[7]
春の花柄ドレス: 春の装いにぴったりの鮮やかな花柄ドレス。柔らかい素材で着心地も抜群です。
ライトウールスカーフ: 秋や冬にぴったりのウールスカーフ。軽量で肌触りが良く、暖かさをプラスします。
防水ジャケット: 雨の日の外出を快適にする防水ジャケット。軽量で動きやすいデザインが魅力です。

Keyword Search

[8]
長袖の紺色のドレス: 落ち着いた紺色がエレガントな長袖ドレス。フォーマルな場にも最適です。
ニットドレス: 柔らかく暖かい素材で作られたニットドレス。寒い季節のカジュアルスタイルに最適です。
ひまわりの模様のドレス: 明るいひまわり柄が夏らしい印象のドレス。軽やかで涼しい素材が魅力です。

Hybrid Search

The alpha parameter determines the weight given to the sparse and dense search methods. alpha = 0 is pure sparse (bm25) search, whereas alpha = 1 is pure dense (vector) search.

Alpha is an optional parameter. The default is set to 0.75.

The below query is finding Jeopardy questions about animals and is limiting the output to only two results. Notice alpha is set to 0.80, which means it is weighing the vector search results more than bm25. If you were to set alpha = 0.25, you would get different results.

[9]
春の花柄ドレス: 春の装いにぴったりの鮮やかな花柄ドレス。柔らかい素材で着心地も抜群です。
ひまわりの模様のドレス: 明るいひまわり柄が夏らしい印象のドレス。軽やかで涼しい素材が魅力です。
フローラルミニスカート: 春や夏にぴったりの花柄ミニスカート。華やかなデザインで女性らしさを引き立てます。

What next?

Weaviate's Japanese Kagome tokenizer is a significant step forward in helping Japanese developers build great AI applications.

Try out Weaviate, starting with the Quickstart.

And where you have Japanese data, set the property tokenizer to "kagome_jp" as shown in the code above.

Note

As of 1.28, the tokenizer must be separately enabled by setting ENABLE_TOKENIZER_KAGOME_JA environment variable to true. (For example, in the docker-compose.yml file.)