Weaviate How To Choose An Embedding Model

How To Choose An Embedding Model

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksweaviate-featuresfunction-callingevaluationweaviate-recipesPythongenerative-ai

alph-notebooks/weaviate-recipes / how_to_choose_an_embedding_model.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

How to Choose an Embedding Model

[1]

Requirement already satisfied: pytrec_eval in /opt/homebrew/lib/python3.11/site-packages (0.5)

[2]

[3]

Started /Users/leonie/.cache/weaviate-embedded: process ID 87673

{"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-05-08T12:29:09+02:00"}
{"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-05-08T12:29:09+02:00"}
{"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-05-08T12:29:09+02:00"}
{"level":"warning","msg":"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.","time":"2024-05-08T12:29:09+02:00"}
{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50050","time":"2024-05-08T12:29:09+02:00"}
{"action":"restapi_management","level":"info","msg":"Serving weaviate at http://127.0.0.1:8079","time":"2024-05-08T12:29:09+02:00"}

True

[4]

{"level":"info","msg":"Created shard pastries_6h48j8vP30f0 in 1.561167ms","time":"2024-05-08T12:29:09+02:00"}
{"action":"hnsw_vector_cache_prefill","count":1000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:09+02:00","took":41958}
{"level":"info","msg":"Completed loading shard dim_512_vIr0JrwWdLwP in 3.483791ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":219958}
{"level":"info","msg":"Completed loading shard dimensions__512_ALiufnMMSrhQ in 5.809542ms","time":"2024-05-08T12:29:10+02:00"}
{"level":"info","msg":"Completed loading shard dimension_1536_xLwNXnHZeLcm in 6.313459ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":45083}
{"level":"info","msg":"Completed loading shard dimension_512_JzsLHTawD3Y9 in 6.486ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":96708}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":45125}
{"level":"info","msg":"Completed loading shard dimensi_512_Y29FQu1Vc0xQ in 6.497125ms","time":"2024-05-08T12:29:10+02:00"}
{"level":"info","msg":"Completed loading shard dimensions512_Zvz765WZOg7H in 6.555208ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":45333}
{"level":"info","msg":"Completed loading shard docs_SN8loOvzlYv7 in 5.827542ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":43666}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":60959}
{"level":"info","msg":"Completed loading shard dimensions_512_j71GKbSxo54r in 6.35925ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":49583}
{"level":"info","msg":"Completed loading shard mycontent_oiMgIfNpvwWZ in 5.154083ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":30958}
{"level":"info","msg":"Completed loading shard llamaindex_dWivqPiChdO8 in 7.110875ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":160791}
{"level":"info","msg":"Completed loading shard question_QJgilDdFfzJ2 in 6.577666ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":29875}
{"level":"info","msg":"Completed loading shard llamaindex_filter_fKXpSjFDc0KK in 14.567042ms","time":"2024-05-08T12:29:10+02:00"}
{"level":"info","msg":"Completed loading shard question3_jYcpPpqcO5H9 in 3.889875ms","time":"2024-05-08T12:29:10+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":36709}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":404375}
{"level":"info","msg":"Completed loading shard myexampleindex_XGMjGqT60mbO in 9.050417ms","time":"2024-05-08T12:29:10+02:00"}
{"level":"info","msg":"Completed loading shard myexternalcontext_T58iI8xW5iA8 in 14.116958ms","time":"2024-05-08T12:29:10+02:00"}

BatchObjectReturn(all_responses=[UUID('74d4e157-886b-4075-9849-5b5c30733ce6'), UUID('8556cc0c-f8f3-4bd2-95d7-76aee821e1df'), UUID('bc49a7d3-c2c4-4fed-80fb-717aff02f8b3'), UUID('5660da7c-396d-4663-bfe1-6439e8b261e4'), UUID('47ca5c52-f841-45d7-99c7-68136d1abfa3'), UUID('32cb8665-c9b3-4482-840d-6003d6f22672'), UUID('342e40f9-f352-4155-803b-f3e644e6a6a6'), UUID('6a09544a-21f2-42b4-9636-a45361cfb887')], elapsed_seconds=0.7803750038146973, errors={}, uuids={0: UUID('74d4e157-886b-4075-9849-5b5c30733ce6'), 1: UUID('8556cc0c-f8f3-4bd2-95d7-76aee821e1df'), 2: UUID('bc49a7d3-c2c4-4fed-80fb-717aff02f8b3'), 3: UUID('5660da7c-396d-4663-bfe1-6439e8b261e4'), 4: UUID('47ca5c52-f841-45d7-99c7-68136d1abfa3'), 5: UUID('32cb8665-c9b3-4482-840d-6003d6f22672'), 6: UUID('342e40f9-f352-4155-803b-f3e644e6a6a6'), 7: UUID('6a09544a-21f2-42b4-9636-a45361cfb887')}, has_errors=False)

Setup Evaluation

[5]

{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":6396625}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-08T12:29:10+02:00","took":9203584}

[6]

{'Sweet pastry': {'Croissant': 0.4081610441207886,
,  'Donut': 0.4437391757965088,
,  'Scone': 0.4496232271194458,
,  'Sandwich': 0.48311829566955566},
, 'Suitable for lunch': {'Sandwich': 0.6699223518371582,
,  'Roll': 0.710200846195221,
,  'Bagel': 0.7621301412582397,
,  'Croissant': 0.7663571238517761},
, 'Goes well with jam': {'Scone': 0.6245813369750977,
,  'Bagel': 0.6838458776473999,
,  'Donut': 0.6856768131256104,
,  'Sandwich': 0.7056261301040649}}

[7]

{
 "Sweet pastry": {
  "P_4": 0.5,
  "recall_4": 0.6666666666666666
 },
 "Suitable for lunch": {
  "P_4": 0.75,
  "recall_4": 0.75
 },
 "Goes well with jam": {
  "P_4": 0.25,
  "recall_4": 0.3333333333333333
 }
}