1.Getting Started With RAG In DSPy
vector-searchdspyvector-databaseretrieval-augmented-generationllm-frameworksfunction-callingweaviate-recipesintegrationsPythongenerative-aillm-agent-frameworks
Export
Getting Started with RAG in DSPy
This notebook will show you how to use DSPy to compile a RAG program! DSPy compilation is a fairly new tool for LLM developers, so let's start with an overview of the concept. By compiling, we mean finding the prompts that elicit the behavior we want from LLMs when connected in some kind of pipeline.
For example, RAG is a very common LLM pipeline. In it's simplest form, RAG consists of 2 steps, (1) Retrieve and (2) Answer a Question. Part (2), Answering a Question, has an associated prompt, for example, people generally use:
--
Please answer the question based on the following context.
context {context}
question {question}
--
This prompt may be a good initial point for an LLM to understand the task. However, it is not the optimal prompt. DSPy optimizes the prompt for you by jointly (1) tweaking the instructions, such as rewriting an initial prompt like:
Please answer the question based on the following context.
to
Assess the context and answer the given questions that are predominantly about software usage, process optimization, and troubleshooting. Focus on providing accurate information related to tech or software-related queries.
Further, DSPy (2) finds examples of desired input-outputs in the prompt to further improve performance, also known as In-Context Learning. In this example, we will begin with the simple prompt: Please answer the question based on the following context. and end up with:
In order to leverage black-box optimization techniques like random search, bayesian optimization, or evolutionary algorithms, we need a metric. Coming up with metrics to describe desired system behavior has been a longstanding challenge in Machine Learning research. Excitingly, LLMs have made amazing progress. For example, we can evaluate a RAG answer by prompting an LLM with, Is the assessed text grounded in the context? Say no if it includes significant facts not in the context. We then optimize the RAG program to increase the metric LLM's assessment of answer quality.
This example contains 4 parts:
- 0: DSPy Settings and Installation
- 1: DSPy Datasets with
dspy.Example - 2: LLM Metrics in DSPy
- 3: LLM Programming with
dspy.Module - 4: Optimization with
BootstrapFewShot,BootstrapFewShotRandomSearch, andBayesianSignatureOptimizer.
We are using 2 datasets for this example. Firstly, we have an index of the Weaviate Blog Posts. We will use the Weaviate Blog Posts as the retrieved context to help with our second dataset, the Weaviate FAQs. The Weaviate FAQs consists of 44 question-answer pairs of frequently asked Weaviate questions such as: Do I need to know about Docker (Compose) to use Weaviate?
We isolate 10 examples to use as our test set and optimize our program with the remaining 34.
Our uncompiled RAG program achieves a score of 270 on the held-out test set.
Our RAG program compiled with the BayesianSignatureOptimizer achieves a score of 340! A ~30% improvement!
0: DSPy Settings and Installations
[ ]
[1]
[4]
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/weaviate/warnings.py:121: DeprecationWarning: Dep005: You are using weaviate-client version 3.26.2. The latest version is 4.4.4.
Please consider upgrading to the latest version. See https://weaviate.io/developers/weaviate/client-libraries/python for details.
warnings.warn(
[6]
['Neurons intertwine,\nSynapses ignite, design,\nMinds in code align.'] ['In the realm of silicon thought,\nNeural networks, with knowledge fraught.\nA dance of data, endlessly taught.']
1. DSPy Datasets with dspy.Example
Our retrieval engine is filled with chunks from the Weaviate Blogs.
Please see weaviate/recipes/integrations/dspy/Weaviate-Import.ipynb for a full tutorial.
Import FAQs from a markdown file
[7]
['Why would I use Weaviate as my vector database?', , 'What is the difference between Weaviate and for example Elasticsearch?', , 'Do you offer Weaviate as a managed service?', , 'How should I configure the size of my instance?', , 'Do I need to know about Docker (Compose) to use Weaviate?']
[8]
44
Wrap each FAQ into an Example object
The dspy Example object optionally lets you attach metadata, or additional labels, to input/output pairs.
For example, you may want to jointly supervise the answer as well as the context the retrieval system produced to feed into the answer generator.
[9]
[12]
Example({'question': 'Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions)'}) (input_keys={'question'}) 2. LLM Metrics
Define a Metric for Performance.
[17]
Inspect the metric
[18]
Test Question: What do cross encoders do? Predicted Answer: predicted_answer Faithful: 5 Detail: 1 Overall: 5
float
[19]
Test Question: What do cross encoders do? Predicted Answer: predicted_answer Faithful: 1 Detail: 1 Overall: 1
float
[20]
Assess the quality of an answer to a question.
---
Follow the following format.
Context: The context for answering the question.
Assessed Question: The evaluation criterion.
Assessed Answer: The answer to the question.
Reasoning: Let's think step by step in order to ${produce the assessment_answer}. We ...
Assessment Answer: A rating between 1 and 5. Only output the rating and nothing else.
---
Context: N/A
Assessed Question: Is the assessed answer detailed?
Assessed Answer: They index data.
Reasoning: Let's think step by step in order to produce the assessment answer. We need to consider if the answer provides enough detail to fully answer the question. In this case, the answer "They index data" is very vague and does not provide any specific details about who "they" are or what it means to "index data".
Assessment Answer: 1
Assess the quality of an answer to a question.
---
Follow the following format.
Context: The context for answering the question.
Assessed Question: The evaluation criterion.
Assessed Answer: The answer to the question.
Reasoning: Let's think step by step in order to ${produce the assessment_answer}. We ...
Assessment Answer: A rating between 1 and 5. Only output the rating and nothing else.
---
Context:
[1] «[Cross Encoders](#cross-encoders) (collapsing the use of Large Language Models for ranking into this category as well)
1. [Metadata Rankers](#metadata-rankers)
1. [Score Rankers](#score-rankers)
## Cross Encoders
Cross Encoders are one of the most well known ranking models for content-based re-ranking. There is quite a collection of pre-trained cross encoders available on [sentence transformers](https://www.sbert.net/docs/pretrained_cross-encoders.html). We are currently envisioning interfacing cross encoders with Weaviate using the following syntax.»
[2] «Thus, quality at the expense of speed, becomes more interesting. ### LLMs as Cross Encoders
So, let’s dive into the LLM hype a little more, how can we use LLMs for re-ranking? There are generally 2 ways to do this. The first strategy is identical to the cross encoder, we give the LLM the [query, document] input and prompt it to output a score of how relevant the document is to the query. The tricky thing with this is bounding the score.»
[3] «They offer the advantage of further reasoning about the relevance of results without needing specialized training. Cross Encoders can be interfaced with Weaviate to re-rank search results, trading off performance for slower search speed. * **Metadata Rankers** are context-based re-rankers that use symbolic features to rank relevance. They take into account user and document features, such as age, gender, location, preferences, release year, genre, and box office, to predict the relevance of candidate documents. By incorporating metadata features, these rankers offer a more personalized and context-aware search experience.»
[4] «These kinds of models are increasingly being used as guardrails for generative models. For example, a harmful or NSFW content detector can prevent these generations from making it through the search pipeline. An interesting idea I recently heard from Eddie Zhou on Jerry Liu’s Llama Index Fireside Chat is the idea of using Natural Language Inference models to prevent hallucination by predicting the entailment or contradiction taking as the [retrieved context, generated output] as input. Because large language models are stochastic models, we can sample several candidate generations and filter them through score rankers like these. ## A Recap of the Ranking Models
* **Cross Encoders** are content-based re-ranking models that utilize pre-trained models, such as those available on Sentence Transformers, to rank the relevance of documents.»
[5] «
The blog post included this great visual to help with the visualization of combining Bi-Encoders and Cross-Encoders. This fishing example explains the concept of coarse-grained retrieval (fishing net = vector search / bm25) and manual inspection of the fish (fishermen = ranking models). Depicted with manual inspection of fish, the main cost of ranking models is speed. In March, Bob van Luijt appeared on a Cohere panel to discuss [“AI and The Future of Search”](https://twitter.com/cohereai/status/1636396916157079554?s=46&t=Zzg6vgh4rwmYEkdV-3v5gg). Bob explained the effectiveness of combining zero-shot vector embedding models from providers such as Cohere, OpenAI, or HuggingFace with BM25 sparse search together in Hybrid Search.»
Assessed Question: Is the assessed text grounded in the context? Say no if it includes significant facts not in the context.
Assessed Answer: They index data.
Reasoning: Let's think step by step in order to produce the assessment answer. We can see that the assessed answer "They index data" is not directly related to the context provided. The context discusses various ranking models, including Cross Encoders, Metadata Rankers, and Score Rankers, and how they are used in search and retrieval systems. However, the context does not mention anything about indexing data. Therefore, the assessed answer is not grounded in the context.
Assessment Answer: 1
Assess the quality of an answer to a question.
---
Follow the following format.
Context: The context for answering the question.
Assessed Question: The evaluation criterion.
Assessed Answer: The answer to the question.
Reasoning: Let's think step by step in order to ${produce the assessment_answer}. We ...
Assessment Answer: A rating between 1 and 5. Only output the rating and nothing else.
---
Context:
[1] «[Cross Encoders](#cross-encoders) (collapsing the use of Large Language Models for ranking into this category as well)
1. [Metadata Rankers](#metadata-rankers)
1. [Score Rankers](#score-rankers)
## Cross Encoders
Cross Encoders are one of the most well known ranking models for content-based re-ranking. There is quite a collection of pre-trained cross encoders available on [sentence transformers](https://www.sbert.net/docs/pretrained_cross-encoders.html). We are currently envisioning interfacing cross encoders with Weaviate using the following syntax.»
[2] «Thus, quality at the expense of speed, becomes more interesting. ### LLMs as Cross Encoders
So, let’s dive into the LLM hype a little more, how can we use LLMs for re-ranking? There are generally 2 ways to do this. The first strategy is identical to the cross encoder, we give the LLM the [query, document] input and prompt it to output a score of how relevant the document is to the query. The tricky thing with this is bounding the score.»
[3] «They offer the advantage of further reasoning about the relevance of results without needing specialized training. Cross Encoders can be interfaced with Weaviate to re-rank search results, trading off performance for slower search speed. * **Metadata Rankers** are context-based re-rankers that use symbolic features to rank relevance. They take into account user and document features, such as age, gender, location, preferences, release year, genre, and box office, to predict the relevance of candidate documents. By incorporating metadata features, these rankers offer a more personalized and context-aware search experience.»
[4] «These kinds of models are increasingly being used as guardrails for generative models. For example, a harmful or NSFW content detector can prevent these generations from making it through the search pipeline. An interesting idea I recently heard from Eddie Zhou on Jerry Liu’s Llama Index Fireside Chat is the idea of using Natural Language Inference models to prevent hallucination by predicting the entailment or contradiction taking as the [retrieved context, generated output] as input. Because large language models are stochastic models, we can sample several candidate generations and filter them through score rankers like these. ## A Recap of the Ranking Models
* **Cross Encoders** are content-based re-ranking models that utilize pre-trained models, such as those available on Sentence Transformers, to rank the relevance of documents.»
[5] «
The blog post included this great visual to help with the visualization of combining Bi-Encoders and Cross-Encoders. This fishing example explains the concept of coarse-grained retrieval (fishing net = vector search / bm25) and manual inspection of the fish (fishermen = ranking models). Depicted with manual inspection of fish, the main cost of ranking models is speed. In March, Bob van Luijt appeared on a Cohere panel to discuss [“AI and The Future of Search”](https://twitter.com/cohereai/status/1636396916157079554?s=46&t=Zzg6vgh4rwmYEkdV-3v5gg). Bob explained the effectiveness of combining zero-shot vector embedding models from providers such as Cohere, OpenAI, or HuggingFace with BM25 sparse search together in Hybrid Search.»
Assessed Question: Please rate how well this answer answers the question, `What do cross encoders do?` based on the context. `They index data.`
Assessed Answer: They index data.
Reasoning: Let's think step by step in order to produce the assessment answer. We can see from the context that Cross Encoders are used for content-based re-ranking. They take in a query and a document and output a score of how relevant the document is to the query. They are also used to rank the relevance of documents using pre-trained models. However, the assessed answer states that they index data, which is not mentioned or implied in the context. Therefore, the answer does not accurately reflect the information provided in the context.
Assessment Answer: 1
3. The DSPy Programming Model
This block of first code will initilaize the GenerateAnswer signature.
Then we will compose a dspy.Module consisting of:
- Retrieve
- GenerateAnswer
The DSPy programming model is one of the most powerful aspects of DSPy, we get:
- An intuitive interface to compose prompts into programs.
- A clean way to organize prompts into Signatures.
- Structured output parsing with
dspy.OutputField - Built-in prompt extensions such as
ChainOfThought,ReAct, and more!
[23]
[24]
A little more info on built-in dspy modules
The DSPy programming model gives you a lot of cool features out of the box. Observe how different modules implement signatures with additional prompting techniques like ChainOfThought and ReAct. Predict is the base class to observe what a standrd prompt looks like without the module extensions.
dspy.Predict
[30]
Answer questions based on the context.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Answer: ${answer}
---
Question: What are Cross Encoders?
Answer: Context: Cross Encoders are a type of neural network model used in natural language processing tasks. They are designed to encode pairs of sentences and generate a similarity score between them.
Question: How do Cross Encoders work?
Answer: Cross Encoders work by taking two sentences as input and encoding them into fixed-length vectors. These vectors are then compared using a similarity metric to determine the similarity score between the sentences.
Question: What is the purpose of Cross Encoders?
Answer: The purpose of Cross Encoders is to measure the semantic similarity between pairs of sentences. They can be used in various applications such as question answering, text classification, and information retrieval.
Question: How are Cross Encoders different from other neural network models?
Answer: Cross Encoders
dspy.ChainOfThought
[31]
Answer questions based on the context.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ${answer}
---
Context:
produce the answer. We will first define what cross encoders are and then explain their purpose.
Answer: Cross Encoders are a type of neural network model that are used in natural language processing tasks. They are designed to encode pairs of sentences or documents and produce a similarity score between them. The purpose of cross encoders is to capture the semantic relationship between two pieces of text and determine how similar or related they are.
Question: What are Cross Encoders?
Reasoning: Let's think step by step in order to produce the answer. We will first define what cross encoders are and then explain their purpose.
Answer: Cross Encoders are a type of neural network model that are used in natural language processing tasks. They are designed to encode pairs of sentences or documents and produce a similarity score between them. The purpose of cross encoders is to capture the semantic relationship between two pieces of text
dspy.ReAct
[32]
You will be given `context`, `question` and you will respond with `answer`.
To do this, you will interleave Thought, Action, and Observation steps.
Thought can reason about the current situation, and Action can be the following types:
(1) Search[query], which takes a search query and returns one or more potentially relevant passages from a corpus
(2) Finish[answer], which returns the final `answer` and finishes the task
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Thought 1: next steps to take based on last observation
Action 1: always either Search[query] or, when done, Finish[answer]
Observation 1: observations based on action
Thought 2: next steps to take based on last observation
Action 2: always either Search[query] or, when done, Finish[answer]
---
Context:
Cross encoders are a type of neural network model used in natural language processing tasks. They are designed to encode pairs of sentences or documents and capture the semantic relationship between them. They are commonly used in tasks such as sentence similarity, paraphrase detection, and document retrieval.
Action 1: Finish[answer]
Question: What are cross encoders?
Thought 1:
Action 1: Since the context already provides a clear definition and explanation of cross encoders, there is no need to search for additional information. I can directly provide the answer. Action 1: Finish[Cross encoders are a type of neural network model used in natural language processing tasks. They are designed to encode pairs of sentences or documents and capture the semantic relationship between them. They are
Observation 1: Failed to parse action. Bad formatting or incorrect action name.
Thought 2: The action format was incorrect. Let me correct it.
Thought 2: Since the context already provides a clear definition and explanation of cross encoders, there is no need to search for additional information. I can directly provide the answer.
Action 2: Finish[Cross encoders are a type of neural network model used in natural language processing tasks. They are designed to encode pairs of sentences or documents and capture the semantic relationship between them. They are commonly used in tasks such as sentence similarity, paraphrase detection, and document retrieval.]
Initialize DSPy Program
[33]
Test uncompiled inference
[34]
Re-rankers in search engines are algorithms or models that are used to reorder or re-rank search results based on additional criteria or features. They can be used to improve the relevance and personalization of search results by considering factors such as user preferences, document features, metadata, and context.
Check the last call to the LLM
[35]
Answer questions based on the context.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ${answer}
---
Context:
[1] «They offer the advantage of further reasoning about the relevance of results without needing specialized training. Cross Encoders can be interfaced with Weaviate to re-rank search results, trading off performance for slower search speed. * **Metadata Rankers** are context-based re-rankers that use symbolic features to rank relevance. They take into account user and document features, such as age, gender, location, preferences, release year, genre, and box office, to predict the relevance of candidate documents. By incorporating metadata features, these rankers offer a more personalized and context-aware search experience.»
[2] «Taken directly from the paper, “Our findings indicate that cross-encoder re-rankers can efficiently be improved without additional computational burden and extra steps in the pipeline by explicitly adding the output of the first-stage ranker to the model input, and this effect is robust for different models and query types”. Taking this a bit further, [Dinh et al.](https://arxiv.org/abs/2206.06565) shows that most tabular machine learning tasks can be translated to text and benefit from transfer learning of text-based models. Many of these metadata rankers may also take in something like a collaborative filtering score that is based on this user’s history, as well as other users on the platform — another interesting feature to think of interfacing this way. The main point being, maybe we can just add these meta features to our [query, document] representation and keep the Zero-Shot party going. We recently had an interesting discussion about metadata ranking and future directions for ranking models broadly on our latest Weaviate podcast! 👉 Check it out [here](https://www.youtube.com/watch?v=aLY0q6V01G4)
## Score Rankers
Score rankers describe using either a classifier to detect things, or a regression model to score things, about our candidate documents to rank with.»
[3] «To end this article, let’s discuss a little further why ranking is so exciting for the most hyped pairing of LLMs and Search: Retrieval-Augmented Generation. ## Ranking for Retrieval-Augmented Generation
A lot of the recent successes of vector search can be attributed to their effectiveness as a tool for Large Language Models. So whereas the speed trade-off with rankers may be a major bottleneck for how humans use search, it might not be as much of a problem for how LLMs use search. Of course fast generation is preferred, but if you are paying for the result, quality may be more important than speed. Shi et al.»
Question: What are re-rankers in search engines?
Reasoning: Let's think step by step in order to produce the answer. We can find the answer by examining the given context.
Answer: Re-rankers in search engines are algorithms or models that are used to reorder or re-rank search results based on additional criteria or features. They can be used to improve the relevance and personalization of search results by considering factors such as user preferences, document features, metadata, and context.
4. DSPy Optimization
Evaluate our RAG Program before it is compiled
[38]
Example({'question': 'Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions)'}) (input_keys={'question'}) [39]
0%| | 0/10 [00:00<?, ?it/s]
Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer
Average Metric: 2.0 / 1 (200.0): 10%|█ | 1/10 [00:00<00:04, 2.03it/s]
Faithful: 1 Detail: 3 Overall: 5 Test Question: How can I retrieve the total object count in a class? Predicted Answer: predicted_answer
Average Metric: 3.4 / 2 (170.0): 20%|██ | 2/10 [00:00<00:03, 2.28it/s]
Faithful: 1 Detail: 4 Overall: 1 Test Question: How do I get the cosine similarity from Weaviate's certainty? Predicted Answer: predicted_answer
Average Metric: 5.0 / 3 (166.7): 30%|███ | 3/10 [00:01<00:03, 2.05it/s]
Faithful: 1 Detail: 5 Overall: 1
Average Metric: 7.6 / 4 (190.0): 40%|████ | 4/10 [00:02<00:04, 1.24it/s]
Test Question: The quality of my search results change depending on the specified limit. Why? How can I fix this? Predicted Answer: predicted_answer Faithful: 2 Detail: 5 Overall: 4 Test Question: Why did you use GraphQL instead of SPARQL? Predicted Answer: predicted_answer
Average Metric: 9.6 / 5 (192.0): 50%|█████ | 5/10 [00:03<00:04, 1.09it/s]
Faithful: 1 Detail: 3 Overall: 5 Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
Average Metric: 11.2 / 6 (186.7): 60%|█████▍ | 6/10 [00:04<00:03, 1.30it/s]
Faithful: 1 Detail: 1 Overall: 5 Test Question: What is best practice for updating data? Predicted Answer: predicted_answer
Average Metric: 14.6 / 7 (208.6): 70%|██████▎ | 7/10 [00:04<00:01, 1.56it/s]
Faithful: 5 Detail: 3 Overall: 4 Test Question: Can I connect my own module? Predicted Answer: predicted_answer
Average Metric: 18.6 / 8 (232.5): 80%|███████▏ | 8/10 [00:05<00:01, 1.76it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
Average Metric: 22.6 / 9 (251.1): 90%|████████ | 9/10 [00:05<00:00, 1.84it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Does Weaviate use Hnswlib? Predicted Answer: predicted_answer
Average Metric: 24.400000000000002 / 10 (244.0): 100%|█| 10/10 [00:06<00:00, 1
Faithful: 1 Detail: 2 Overall: 5 Average Metric: 24.400000000000002 / 10 (244.0%)
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/dspy/evaluate/evaluate.py:137: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead. df = df.applymap(truncate_cell)
244.0
Metric Analysis
The maximum value per rating is (5 + 5*2 + 5) / 5 = 4
4 * 10 test questions = 40
[40]
Answer questions based on the context.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ${answer}
---
Context:
[1] «There is no performance gain if we have more parallelization than there are available CPUs. However, each thread needs additional memory. So with 32 parallel imports, we had the worst of both worlds: High memory usage and no performance gains beyond 8. With the fix, even if you import from multiple clients, Weaviate automatically handles the parallelization to ensure that it does not exceed the number of CPU cores. As a result, you get the maximum performance without "unnecessary" memory usage. ### HNSW optimization
Next, we optimized memory allocations for the HNSW (vector) index.»
[2] «The past for vector searching definitely was not a “simpler time”, and the appeal of modern vector databases like Weaviate is pretty clear given this context. But while the future is here, it isn't yet perfect. Tools like Weaviate can seem like a magician's mystery box. Our users in turn ask us *exactly* how Weaviate does its magic; how it turns all of that data into vectors, and how to control the process. So let's take a look inside the magic box together in this post.»
[3] «:::
## Conclusions
We've managed to implement the indexing algorithm on DiskANN, and the resulting performance is good. From years of research & development, Weaviate has a highly optimized implementation of the HNSW algorithm. With the Vamana implementation, we achieved comparable in-memory results. There are still some challenges to overcome and questions to answer. For example:
* How do we proceed to the natural disk solution of Weaviate?»
Question: Does Weaviate use Hnswlib?
Reasoning: Let's think step by step in order to produce the answer. We can look for any mention of Hnswlib in the given context.
Answer: No information is provided in the given context about whether Weaviate uses Hnswlib or not.
[41]
Assess the quality of an answer to a question.
---
Follow the following format.
Context: The context for answering the question.
Assessed Question: The evaluation criterion.
Assessed Answer: The answer to the question.
Reasoning: Let's think step by step in order to ${produce the assessment_answer}. We ...
Assessment Answer: A rating between 1 and 5. Only output the rating and nothing else.
---
Context: N/A
Assessed Question: Is the assessed answer detailed?
Assessed Answer: No information is provided in the given context about whether Weaviate uses Hnswlib or not.
Reasoning: Let's think step by step in order to produce the assessment answer. We can see that the answer is not detailed. It simply states that there is no information provided in the given context, but it does not provide any additional information or insight into the topic at hand.
Assessment Answer: 2
Assess the quality of an answer to a question.
---
Follow the following format.
Context: The context for answering the question.
Assessed Question: The evaluation criterion.
Assessed Answer: The answer to the question.
Reasoning: Let's think step by step in order to ${produce the assessment_answer}. We ...
Assessment Answer: A rating between 1 and 5. Only output the rating and nothing else.
---
Context:
[1] «There is no performance gain if we have more parallelization than there are available CPUs. However, each thread needs additional memory. So with 32 parallel imports, we had the worst of both worlds: High memory usage and no performance gains beyond 8. With the fix, even if you import from multiple clients, Weaviate automatically handles the parallelization to ensure that it does not exceed the number of CPU cores. As a result, you get the maximum performance without "unnecessary" memory usage. ### HNSW optimization
Next, we optimized memory allocations for the HNSW (vector) index.»
[2] «The past for vector searching definitely was not a “simpler time”, and the appeal of modern vector databases like Weaviate is pretty clear given this context. But while the future is here, it isn't yet perfect. Tools like Weaviate can seem like a magician's mystery box. Our users in turn ask us *exactly* how Weaviate does its magic; how it turns all of that data into vectors, and how to control the process. So let's take a look inside the magic box together in this post.»
[3] «:::
## Conclusions
We've managed to implement the indexing algorithm on DiskANN, and the resulting performance is good. From years of research & development, Weaviate has a highly optimized implementation of the HNSW algorithm. With the Vamana implementation, we achieved comparable in-memory results. There are still some challenges to overcome and questions to answer. For example:
* How do we proceed to the natural disk solution of Weaviate?»
[4] «
### [Weaviate Academy](/developers/academy) & [Workshops](/learn/workshops)
Weaviate Academy and Workshops have had a fantastic year of learning and growth! We've been focusing on ensuring everyone has the chance to understand and use vector databases and get a grasp on Generative AI and data handling. Every week, [Zain](https://www.linkedin.com/in/zainhas/), [JP](https://www.linkedin.com/in/jphwang/), [Daniel](https://www.linkedin.com/in/malgamves/), and [Duda](https://www.linkedin.com/in/dudanogueira/) have been running introductory workshops on vector databases and Weaviate, which have been a hit. Plus, we're super excited about [JP Hwang](https://www.linkedin.com/in/jphwang/)'s initiative, the Weaviate Academy. It's a program that takes you from the basics all the way to production-level skills. Later in the year, we teamed up with [DeepLearningAI](https://www.deeplearning.ai/) to create a short course with [Sebastian](https://www.linkedin.com/in/sebawita/) and [Zain](https://www.linkedin.com/in/zainhas/), "[Vector Databases: from Embeddings to Applications with Weaviate](https://www.deeplearning.ai/short-courses/vector-databases-embeddings-applications/)." It’s been a year packed with learning.»
[5] «[Weaviate](/developers/weaviate/), an open-source vector database written in Go, can serve thousands of queries per second. Running Weaviate on [Sift1M](https://www.tensorflow.org/datasets/catalog/sift1m) (a 128-dimensional representation of objects) lets you serve queries in single-digit milliseconds. But how is this possible? 
*See the [benchmark](/developers/weaviate/benchmarks/ann) page for more stats.*
Weaviate does not look for the exact closest vectors in the store. Instead, it looks for approximate (close enough) elements.»
Assessed Question: Is the assessed text grounded in the context? Say no if it includes significant facts not in the context.
Assessed Answer: No information is provided in the given context about whether Weaviate uses Hnswlib or not.
Reasoning: Let's think step by step in order to produce the assessment answer. We need to determine if the assessed answer is grounded in the context provided. The context discusses Weaviate's optimization, its use of the HNSW algorithm, and its performance. However, there is no mention of Weaviate using Hnswlib. Therefore, the assessed answer is not grounded in the context as it introduces a fact not present in the context.
Assessment Answer: 1
Assess the quality of an answer to a question.
---
Follow the following format.
Context: The context for answering the question.
Assessed Question: The evaluation criterion.
Assessed Answer: The answer to the question.
Reasoning: Let's think step by step in order to ${produce the assessment_answer}. We ...
Assessment Answer: A rating between 1 and 5. Only output the rating and nothing else.
---
Context:
[1] «There is no performance gain if we have more parallelization than there are available CPUs. However, each thread needs additional memory. So with 32 parallel imports, we had the worst of both worlds: High memory usage and no performance gains beyond 8. With the fix, even if you import from multiple clients, Weaviate automatically handles the parallelization to ensure that it does not exceed the number of CPU cores. As a result, you get the maximum performance without "unnecessary" memory usage. ### HNSW optimization
Next, we optimized memory allocations for the HNSW (vector) index.»
[2] «The past for vector searching definitely was not a “simpler time”, and the appeal of modern vector databases like Weaviate is pretty clear given this context. But while the future is here, it isn't yet perfect. Tools like Weaviate can seem like a magician's mystery box. Our users in turn ask us *exactly* how Weaviate does its magic; how it turns all of that data into vectors, and how to control the process. So let's take a look inside the magic box together in this post.»
[3] «:::
## Conclusions
We've managed to implement the indexing algorithm on DiskANN, and the resulting performance is good. From years of research & development, Weaviate has a highly optimized implementation of the HNSW algorithm. With the Vamana implementation, we achieved comparable in-memory results. There are still some challenges to overcome and questions to answer. For example:
* How do we proceed to the natural disk solution of Weaviate?»
[4] «
### [Weaviate Academy](/developers/academy) & [Workshops](/learn/workshops)
Weaviate Academy and Workshops have had a fantastic year of learning and growth! We've been focusing on ensuring everyone has the chance to understand and use vector databases and get a grasp on Generative AI and data handling. Every week, [Zain](https://www.linkedin.com/in/zainhas/), [JP](https://www.linkedin.com/in/jphwang/), [Daniel](https://www.linkedin.com/in/malgamves/), and [Duda](https://www.linkedin.com/in/dudanogueira/) have been running introductory workshops on vector databases and Weaviate, which have been a hit. Plus, we're super excited about [JP Hwang](https://www.linkedin.com/in/jphwang/)'s initiative, the Weaviate Academy. It's a program that takes you from the basics all the way to production-level skills. Later in the year, we teamed up with [DeepLearningAI](https://www.deeplearning.ai/) to create a short course with [Sebastian](https://www.linkedin.com/in/sebawita/) and [Zain](https://www.linkedin.com/in/zainhas/), "[Vector Databases: from Embeddings to Applications with Weaviate](https://www.deeplearning.ai/short-courses/vector-databases-embeddings-applications/)." It’s been a year packed with learning.»
[5] «[Weaviate](/developers/weaviate/), an open-source vector database written in Go, can serve thousands of queries per second. Running Weaviate on [Sift1M](https://www.tensorflow.org/datasets/catalog/sift1m) (a 128-dimensional representation of objects) lets you serve queries in single-digit milliseconds. But how is this possible? 
*See the [benchmark](/developers/weaviate/benchmarks/ann) page for more stats.*
Weaviate does not look for the exact closest vectors in the store. Instead, it looks for approximate (close enough) elements.»
Assessed Question: Please rate how well this answer answers the question, `Does Weaviate use Hnswlib?` based on the context. `No information is provided in the given context about whether Weaviate uses Hnswlib or not.`
Assessed Answer: No information is provided in the given context about whether Weaviate uses Hnswlib or not.
Reasoning: Let's think step by step in order to produce the assessment answer. We need to assess if the answer provided is accurate based on the context given. The context talks about Weaviate, a vector database, and its optimization and performance. It mentions the use of the HNSW (vector) index and the implementation of the indexing algorithm on DiskANN. However, it does not mention the use of Hnswlib specifically. Therefore, the answer stating that no information is provided in the context about whether Weaviate uses Hnswlib or not is accurate.
Assessment Answer: 5
BootstrapFewShot
[42]
0%| | 0/20 [00:00<?, ?it/s]
Test Question: Why would I use Weaviate as my vector database? Predicted Answer: predicted_answer
5%|██▏ | 1/20 [00:19<06:15, 19.74s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: What is the difference between Weaviate and for example Elasticsearch? Predicted Answer: predicted_answer
10%|████▍ | 2/20 [00:40<06:02, 20.17s/it]
Faithful: No
Detail: 1
Overall: 5
Failed to run or to evaluate example Example({'question': 'What is the difference between Weaviate and for example Elasticsearch?'}) (input_keys={'question'}) with <function llm_metric at 0x281e95360> due to could not convert string to float: 'No'.
Test Question: Do you offer Weaviate as a managed service?
Predicted Answer: predicted_answer
15%|██████▌ | 3/20 [01:12<07:15, 25.64s/it]
Faithful: 1 Detail: 5 Overall: 1 Test Question: How should I configure the size of my instance? Predicted Answer: predicted_answer
20%|████████▊ | 4/20 [01:30<06:04, 22.79s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Do I need to know about Docker (Compose) to use Weaviate? Predicted Answer: predicted_answer
25%|███████████ | 5/20 [01:51<05:34, 22.32s/it]
Faithful: 5 Detail: 5 Overall: 5
0%| | 0/20 [00:00<?, ?it/s] 0%| | 0/20 [00:00<?, ?it/s]
Bootstrapped 4 full traces after 1 examples in round 2.
Inspect the compiled prompt
[48]
'Cross encoders are ranking models used for content-based re-ranking. They take a query and a document as input and output a score indicating the relevance of the document to the query. They can be interfaced with Weaviate to re-rank search results.'
[49]
Answer questions based on the context.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ${answer}
---
Context:
[1] «Then we will see how the text vectorization process can be tweaked, before wrapping up by discussing a few considerations also. ## Background
I often find myself saying that Weaviate makes it fast and easy to produce a vector database from text. But it can be easy to forget just how fast and how easy it can make things. It is true that even in the “old days” of say, five to ten years ago, producing a database with vector capabilities was technically possible. You *simply* had to (*inhales deeply*) develop a vectorization algorithm, vectorize the data, build a vector index, build a database with the underlying data, integrate the vector index with the database, then forward results from a vector index query to the database and combine the outputs from both (*exhales*).»
[2] «using a special algorithm, the database find the [closest](/blog/distance-metrics-in-vector-search) vectors to the given vector computed for the query. The quality of the search depends crucially on the quality of the model - this is the "secret sauce", as many models are [still closed source](https://www.semianalysis.com/p/google-we-have-no-moat-and-neither). The speed of the search depends crucially on Weaviate, which is open-source and [continuously improving its performance](/blog/weaviate-1-18-release). ## What exactly are vector embeddings? Vectors are numeric representations of data that capture certain features of the data.»
[3] «[Weaviate](/developers/weaviate/), an open-source vector database written in Go, can serve thousands of queries per second. Running Weaviate on [Sift1M](https://www.tensorflow.org/datasets/catalog/sift1m) (a 128-dimensional representation of objects) lets you serve queries in single-digit milliseconds. But how is this possible? 
*See the [benchmark](/developers/weaviate/benchmarks/ann) page for more stats.*
Weaviate does not look for the exact closest vectors in the store. Instead, it looks for approximate (close enough) elements.»
Question: Why would I use Weaviate as my vector database?
Reasoning: Let's think step by step in order to produce the answer. We can find the answer by analyzing the information provided in the context. The context mentions that Weaviate makes it fast and easy to produce a vector database from text. It also mentions that Weaviate can serve thousands of queries per second and can provide search results in single-digit milliseconds. Additionally, Weaviate looks for approximate (close enough) elements instead of exact closest vectors.
Answer: You would use Weaviate as your vector database because it is fast, easy to use, and can handle a large number of queries quickly. It also provides approximate search results, which can be useful in certain applications.
---
Context:
[1] «He developed an Autonomous Testing Agent to enhance software testing efficiency, harnessing the power of [SuperAGI](https://www.linkedin.com/company/superagi/) and Weaviate. Meanwhile, [BYTE](https://lablab.ai/event/cohere-coral-hackathon/byte/byte-ai-based-nutrition-app), an AI-based nutrition app, clinched the top spot at the [Coral Cohere Hackathon](https://lablab.ai/event/cohere-coral-hackathon)! Ayesha and Moneebah built this project to transform and personalize nutrition advice. They used Weaviate’s vector database for search and recommendation and multi-tenancy for data security. These projects offer just a glimpse of the boundless possibilities within the AI realm, pointing the way to a future where AI is more accessible, formidable, and transformative. So, what are you waiting for if you haven't already started building with Weaviate?»
[2] «For sure it might not be for everybody and every use case. But if you are using Weaviate at scale, in production, we believe enabling it will add significant value and encourage you to consider its use.»
[3] «We listened to your feedback, suggestions and use cases! So we made it our mission for the `1.15` release to design and implement an **elegant solution** with a great **Developer Experience (DX)**, which you will love 😍 to use for years to come. ### Announcement
Introducing **Weaviate Cloud-native backups**. 🎉
It allows you to make full database backups (or selected classes) straight to **S3**, **GCS** or the **local filesystem** with a single API call 🤩; and restore the data to a Weaviate instance of your choice with another API call. What is really great about this implementation is that you can create a backup without downtime on a running instance. The database stays fully operational (including receiving writes) while the backup is transferred to the remote storage.»
Question: Do you offer Weaviate as a managed service?
Reasoning: Let's think step by step in order to produce the answer. We need to look for information in the given context that indicates whether Weaviate is offered as a managed service.
Answer: Yes, Weaviate is offered as a managed service. Reasoning: In the context, it is mentioned that Weaviate Cloud-native backups have been introduced, allowing users to make full database backups and restore the data to a Weaviate instance of their choice with an API call. This indicates that Weaviate can be managed by the users themselves, suggesting that it is offered as a managed service.
---
Context:
[1] «First, modify the configuration file to enable a multi-node setup (e.g. 3), and add the `replicationConfig` setting to the collection definition like so:
```json
{
class: YOUR_CLASS_NAME,
... replicationConfig: {
factor: 3
}
}
```
:::tip Replication factor & nodes
The replication factor should be less than or equal to the number of nodes. :::
Once you’ve modified the configuration file and set the replication factor, you should have a multi-node setup. If you are keen to try running a multi-node setup yourself, follow the optional guide here. Or you can read ahead ;).»
[2] «## Recommendations & Wrap-up
As we mentioned before, all you need to configure to enable replication is this in the collection definition:
```json
{
class: 'YOUR_CLASS_NAME',
... replicationConfig: {
factor: 3,
}
}
```
But what replication factor would we recommend? That’s something of a subjective question, but our starting recommendation is 3. The reason is that odd numbers are preferred for consistency so that consensus can always be reached. Higher factors are also possible, but this is more of a measure to scale query throughput, rather than lead to more availability.»
[3] «* Should it be just an implementation of DiskANN? * Or should we explore the capabilities of HNSW and adjust it to work on disk? * How can we guarantee the excellent database UX – so valuable to many Weaviate users – while reaping the benefits of a disk-based solution? Stay tuned as we explore these challenges and questions. We will share our insights as we go.»
Question: How should I configure the size of my instance?
Reasoning: Let's think step by step in order to produce the answer. We need to find the information about configuring the size of the instance in the given context.
Answer: The given context does not provide any information about configuring the size of the instance.
---
Context:
[1] «:::
## Enabling replication in Weaviate
Just the simple act of enabling replication on a Weaviate cluster will provide huge benefits. Doing so might actually be simpler than you might imagine. ### How to enable replication on Weaviate Cloud Services (WCS)
Enabling replication on a Weaviate Cloud Services cluster is as simple as selecting the `Enable High Availability` button at cluster creation time. (Not available on sandboxes.)
This will enable a multi-node configuration in Weaviate and ensures that each class is configured with the appropriate replication factor. ### How to enable replication on self-deployed Weaviate
Enabling replication in a self-deployment setup such as a Docker or Kubernetes setup involves the following two steps.»
[2] «For sure it might not be for everybody and every use case. But if you are using Weaviate at scale, in production, we believe enabling it will add significant value and encourage you to consider its use.»
[3] «The data is persisted, so you can use it from future invocations, or you can [transfer it to another instance](/developers/weaviate/manage-data/read-all-objects/#restore-to-a-target-instance). You can learn more about running Weaviate locally from client code on the [Embedded Weaviate](/developers/weaviate/installation/embedded/) page. ## <i class="fa-solid fa-lightbulb"></i> Use cases
What can you do with Embedded Weaviate? Quite a few things!
First off, you can get started very quickly with Weaviate on your local machine, without having to explicitly download, install or instantiate a server. ### Jupyter notebooks
You can also use Embedded Weaviate from Jupyter notebooks, including on Google Colaboratory.»
Question: Do I need to know about Docker (Compose) to use Weaviate?
Reasoning: Let's think step by step in order to produce the answer. We need to determine if knowledge of Docker (Compose) is necessary to use Weaviate.
Answer: No, knowledge of Docker (Compose) is not necessary to use Weaviate. According to the given context, enabling replication in a self-deployment setup such as Docker or Kubernetes involves two steps, but it is not explicitly mentioned that knowledge of Docker (Compose) is required.
---
Context:
[1] «[Cross Encoders](#cross-encoders) (collapsing the use of Large Language Models for ranking into this category as well)
1. [Metadata Rankers](#metadata-rankers)
1. [Score Rankers](#score-rankers)
## Cross Encoders
Cross Encoders are one of the most well known ranking models for content-based re-ranking. There is quite a collection of pre-trained cross encoders available on [sentence transformers](https://www.sbert.net/docs/pretrained_cross-encoders.html). We are currently envisioning interfacing cross encoders with Weaviate using the following syntax.»
[2] «Thus, quality at the expense of speed, becomes more interesting. ### LLMs as Cross Encoders
So, let’s dive into the LLM hype a little more, how can we use LLMs for re-ranking? There are generally 2 ways to do this. The first strategy is identical to the cross encoder, we give the LLM the [query, document] input and prompt it to output a score of how relevant the document is to the query. The tricky thing with this is bounding the score.»
[3] «They offer the advantage of further reasoning about the relevance of results without needing specialized training. Cross Encoders can be interfaced with Weaviate to re-rank search results, trading off performance for slower search speed. * **Metadata Rankers** are context-based re-rankers that use symbolic features to rank relevance. They take into account user and document features, such as age, gender, location, preferences, release year, genre, and box office, to predict the relevance of candidate documents. By incorporating metadata features, these rankers offer a more personalized and context-aware search experience.»
Question: What do cross encoders do?
Reasoning: Let's think step by step in order to produce the answer. We need to find information about what cross encoders do in the given context.
Answer: Cross encoders are ranking models used for content-based re-ranking. They take a query and a document as input and output a score indicating the relevance of the document to the query. They can be interfaced with Weaviate to re-rank search results.
Evaluate the Compiled RAG Program
[52]
0%| | 0/10 [00:00<?, ?it/s]
Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer
Average Metric: 2.2 / 1 (220.0): 0%| | 0/10 [00:00<?, ?it/s] Average Metric: 2.2 / 1 (220.0): 10%|█ | 1/10 [00:00<00:03, 2.46it/s]
Faithful: 1 Detail: 4 Overall: 5 Test Question: How can I retrieve the total object count in a class? Predicted Answer: predicted_answer
Average Metric: 5.6 / 2 (280.0): 10%|█ | 1/10 [00:00<00:03, 2.46it/s] Average Metric: 5.6 / 2 (280.0): 20%|██ | 2/10 [00:00<00:03, 2.01it/s]
Faithful: 5 Detail: 2 Overall: 5 Test Question: How do I get the cosine similarity from Weaviate's certainty? Predicted Answer: predicted_answer
Average Metric: 7.199999999999999 / 3 (240.0): 20%|▏| 2/10 [00:01<00:03, 2.01 Average Metric: 7.199999999999999 / 3 (240.0): 30%|▎| 3/10 [00:01<00:03, 1.91
Faithful: 1 Detail: 1 Overall: 5 Test Question: The quality of my search results change depending on the specified limit. Why? How can I fix this? Predicted Answer: predicted_answer
Average Metric: 11.2 / 4 (280.0): 30%|██▋ | 3/10 [00:02<00:03, 1.91it/s] Average Metric: 11.2 / 4 (280.0): 40%|███▌ | 4/10 [00:02<00:03, 1.95it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Why did you use GraphQL instead of SPARQL? Predicted Answer: predicted_answer
Average Metric: 12.799999999999999 / 5 (256.0): 40%|▍| 4/10 [00:02<00:03, 1.9 Average Metric: 12.799999999999999 / 5 (256.0): 50%|▌| 5/10 [00:02<00:03, 1.4
Faithful: 1 Detail: 1 Overall: 5 Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
Average Metric: 14.6 / 6 (243.3): 50%|████▌ | 5/10 [00:13<00:03, 1.48it/s] Average Metric: 14.6 / 6 (243.3): 60%|█████▍ | 6/10 [00:13<00:15, 3.90s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: What is best practice for updating data? Predicted Answer: predicted_answer
Average Metric: 16.6 / 7 (237.1): 60%|█████▍ | 6/10 [00:35<00:15, 3.90s/it] Average Metric: 16.6 / 7 (237.1): 70%|██████▎ | 7/10 [00:35<00:30, 10.01s/it]
Faithful: 2 Detail: 1 Overall: 5 Test Question: Can I connect my own module? Predicted Answer: predicted_answer
Average Metric: 19.8 / 8 (247.5): 70%|██████▎ | 7/10 [01:03<00:30, 10.01s/it] Average Metric: 19.8 / 8 (247.5): 80%|███████▏ | 8/10 [01:03<00:31, 15.61s/it]
Faithful: 4 Detail: 5 Overall: 3 Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
Average Metric: 23.8 / 9 (264.4): 80%|███████▏ | 8/10 [01:22<00:31, 15.61s/it] Average Metric: 23.8 / 9 (264.4): 90%|████████ | 9/10 [01:22<00:16, 16.87s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Does Weaviate use Hnswlib? Predicted Answer: predicted_answer
Average Metric: 27.400000000000002 / 10 (274.0): 90%|▉| 9/10 [01:50<00:16, 16. Average Metric: 27.400000000000002 / 10 (274.0): 100%|█| 10/10 [01:50<00:00, 11
Faithful: 4 Detail: 5 Overall: 5 Average Metric: 27.400000000000002 / 10 (274.0%)
274.0
BootstrapFewShotWithRandomSearch
[ ]
[58]
Going to sample between 1 and 4 traces per predictor. Will attempt to train 2 candidate sets.
0%| | 0/20 [00:00<?, ?it/s]
Test Question: Why would I use Weaviate as my vector database? Predicted Answer: predicted_answer Test Question: What is the difference between Weaviate and for example Elasticsearch? Predicted Answer: predicted_answer
Average Metric: 0.0 / 1 (0.0): 0%| | 0/20 [00:00<?, ?it/s] Average Metric: 0.0 / 1 (0.0): 5%|▌ | 1/20 [00:00<00:08, 2.20it/s] Average Metric: 4.0 / 2 (200.0): 5%|▌ | 1/20 [00:00<00:08, 2.20it/s] Average Metric: 4.0 / 2 (200.0): 10%|█ | 2/20 [00:00<00:04, 3.98it/s]
Faithful: No Detail: 1 Overall: 5 Error for example in dev set: could not convert string to float: 'No' Faithful: 5 Detail: 5 Overall: 5 Test Question: How should I configure the size of my instance? Predicted Answer: predicted_answer Test Question: Do you offer Weaviate as a managed service? Predicted Answer: predicted_answer
Average Metric: 5.8 / 3 (193.3): 10%|█ | 2/20 [00:00<00:04, 3.98it/s] Average Metric: 5.8 / 3 (193.3): 15%|█▌ | 3/20 [00:00<00:05, 3.13it/s]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Do I need to know about Docker (Compose) to use Weaviate? Predicted Answer: predicted_answer
Average Metric: 9.8 / 4 (245.0): 15%|█▌ | 3/20 [00:01<00:05, 3.13it/s] Average Metric: 9.8 / 4 (245.0): 20%|██ | 4/20 [00:01<00:05, 2.70it/s] Average Metric: 11.4 / 5 (228.0): 20%|█▊ | 4/20 [00:01<00:05, 2.70it/s]
Faithful: 5 Detail: 5 Overall: 5 Faithful: 1 Detail: 5 Overall: 1
Average Metric: 15.4 / 6 (256.7): 25%|██▎ | 5/20 [00:01<00:05, 2.70it/s]
Test Question: Are there any 'best practices' or guidelines to consider when designing a schema? Predicted Answer: predicted_answer Test Question: What happens when the Weaviate Docker container restarts? Is my data in the Weaviate database lost? Predicted Answer: predicted_answer Faithful: 5 Detail: 5 Overall: 5
Average Metric: 15.4 / 6 (256.7): 30%|██▋ | 6/20 [00:01<00:04, 3.47it/s]
Test Question: Should I use references in my schema? Predicted Answer: predicted_answer
Average Metric: 19.0 / 7 (271.4): 30%|██▋ | 6/20 [00:02<00:04, 3.47it/s] Average Metric: 19.0 / 7 (271.4): 35%|███▏ | 7/20 [00:02<00:05, 2.59it/s]
Faithful: 4 Detail: 5 Overall: 5
Average Metric: 20.8 / 8 (260.0): 35%|███▏ | 7/20 [00:02<00:05, 2.59it/s] Average Metric: 20.8 / 8 (260.0): 40%|███▌ | 8/20 [00:02<00:04, 2.78it/s]
Test Question: Is it possible to create one-to-many relationships in the schema? Predicted Answer: predicted_answer Faithful: 1 Detail: 5 Overall: 2
Average Metric: 22.8 / 9 (253.3): 40%|███▌ | 8/20 [00:03<00:04, 2.78it/s] Average Metric: 22.8 / 9 (253.3): 45%|████ | 9/20 [00:03<00:03, 2.89it/s] Average Metric: 24.400000000000002 / 10 (244.0): 45%|▍| 9/20 [00:03<00:03, 2. Average Metric: 24.400000000000002 / 10 (244.0): 50%|▌| 10/20 [00:03<00:02, 3
Test Question: What is the difference between `text` and `string` and `valueText` and `valueString`? Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 5 Faithful: 1 Detail: 1 Overall: 5
Average Metric: 26.200000000000003 / 11 (238.2): 50%|▌| 10/20 [00:03<00:02, 3 Average Metric: 26.200000000000003 / 11 (238.2): 55%|▌| 11/20 [00:03<00:02, 3
Test Question: Do Weaviate classes have namespaces? Predicted Answer: predicted_answer Test Question: Are there restrictions on UUID formatting? Do I have to adhere to any standards? Predicted Answer: predicted_answer Faithful: 1 Detail: 2 Overall: 5
Average Metric: 28.200000000000003 / 12 (235.0): 55%|▌| 11/20 [00:03<00:02, 3 Average Metric: 28.200000000000003 / 12 (235.0): 60%|▌| 12/20 [00:03<00:01, 4
Faithful: 1 Detail: 3 Overall: 5 Test Question: If I do not specify a UUID during adding data objects, will Weaviate create one automatically? Predicted Answer: predicted_answer
Average Metric: 30.400000000000002 / 13 (233.8): 60%|▌| 12/20 [00:03<00:01, 4 Average Metric: 30.400000000000002 / 13 (233.8): 65%|▋| 13/20 [00:03<00:01, 4 Average Metric: 32.400000000000006 / 14 (231.4): 65%|▋| 13/20 [00:04<00:01, 4 Average Metric: 32.400000000000006 / 14 (231.4): 70%|▋| 14/20 [00:04<00:01, 4
Faithful: 2 Detail: 5 Overall: 2 Test Question: Can I use Weaviate to create a traditional knowledge graph? Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 5 Test Question: Why does Weaviate have a schema and not an ontology? Predicted Answer: predicted_answer Test Question: What is the difference between a Weaviate data schema, ontologies and taxonomies? Predicted Answer: predicted_answer
Average Metric: 36.00000000000001 / 15 (240.0): 70%|▋| 14/20 [00:04<00:01, 4. Average Metric: 36.00000000000001 / 15 (240.0): 75%|▊| 15/20 [00:04<00:01, 4. Average Metric: 38.400000000000006 / 16 (240.0): 75%|▊| 15/20 [00:04<00:01, 4
Faithful: 5 Detail: 5 Overall: 3 Faithful: 2 Detail: 5 Overall: 3 Test Question: How to deal with custom terminology? Predicted Answer: predicted_answer
Average Metric: 40.2 / 17 (236.5): 80%|█████▌ | 16/20 [00:04<00:00, 4.04it/s] Average Metric: 40.2 / 17 (236.5): 85%|█████▉ | 17/20 [00:04<00:00, 4.68it/s]
Test Question: How can you index data near-realtime without losing semantic meaning? Predicted Answer: predicted_answer Faithful: 1 Detail: 2 Overall: 5
Average Metric: 44.2 / 18 (245.6): 85%|█████▉ | 17/20 [00:04<00:00, 4.68it/s] Average Metric: 44.2 / 18 (245.6): 90%|██████▎| 18/20 [00:04<00:00, 4.97it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Why isn't there a text2vec-contextionary in my language? Predicted Answer: predicted_answer
Average Metric: 45.800000000000004 / 19 (241.1): 90%|▉| 18/20 [00:05<00:00, 4 Average Metric: 45.800000000000004 / 19 (241.1): 95%|▉| 19/20 [00:05<00:00, 4
Test Question: How do you deal with words that have multiple meanings? Predicted Answer: predicted_answer Faithful: 1 Detail: 1 Overall: 5
Average Metric: 48.6 / 20 (243.0): 95%|██████▋| 19/20 [00:05<00:00, 4.32it/s] Average Metric: 48.6 / 20 (243.0): 100%|███████| 20/20 [00:05<00:00, 3.70it/s] sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289e37940> ResourceWarning: Enable tracemalloc to get the object allocation traceback sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289e647c0> ResourceWarning: Enable tracemalloc to get the object allocation traceback
Faithful: 5 Detail: 2 Overall: 2 Average Metric: 48.6 / 20 (243.0%) Score: 243.0 for set: [0] New best score: 243.0 for seed -3 Scores so far: [243.0] Best score: 243.0
0%| | 0/20 [00:00<?, ?it/s] Average Metric: 4.0 / 1 (400.0): 0%| | 0/20 [00:00<?, ?it/s] Average Metric: 4.0 / 1 (400.0): 5%|▌ | 1/20 [00:00<00:07, 2.61it/s]
Test Question: Why would I use Weaviate as my vector database? Predicted Answer: predicted_answer Faithful: 5 Detail: 5 Overall: 5 Test Question: What is the difference between Weaviate and for example Elasticsearch? Predicted Answer: predicted_answer Test Question: Do you offer Weaviate as a managed service? Predicted Answer: predicted_answer
Average Metric: 5.6 / 2 (280.0): 5%|▌ | 1/20 [00:00<00:07, 2.61it/s] Average Metric: 5.6 / 2 (280.0): 10%|█ | 2/20 [00:00<00:07, 2.35it/s] Average Metric: 5.6 / 3 (186.7): 10%|█ | 2/20 [00:00<00:07, 2.35it/s]
Faithful: NoFaithful: 1 Detail: 5 Overall: 1 Detail: 1 Overall: 5 Error for example in dev set: could not convert string to float: 'No'
Average Metric: 7.3999999999999995 / 4 (185.0): 15%|▏| 3/20 [00:01<00:07, 2.3 Average Metric: 7.3999999999999995 / 4 (185.0): 20%|▏| 4/20 [00:01<00:04, 3.5
Test Question: How should I configure the size of my instance? Predicted Answer: predicted_answer Test Question: Do I need to know about Docker (Compose) to use Weaviate? Predicted Answer: predicted_answer Faithful: 1 Detail: 2 Overall: 5
Average Metric: 11.399999999999999 / 5 (228.0): 20%|▏| 4/20 [00:01<00:04, 3.5 Average Metric: 11.399999999999999 / 5 (228.0): 25%|▎| 5/20 [00:01<00:04, 3.3 Average Metric: 14.999999999999998 / 6 (250.0): 25%|▎| 5/20 [00:01<00:04, 3.3 Average Metric: 14.999999999999998 / 6 (250.0): 30%|▎| 6/20 [00:01<00:03, 4.0
Test Question: What happens when the Weaviate Docker container restarts? Is my data in the Weaviate database lost? Predicted Answer: predicted_answer Faithful: 5 Detail: 5 Overall: 5 Faithful: 4 Detail: 5 Overall: 5 Test Question: Are there any 'best practices' or guidelines to consider when designing a schema? Predicted Answer: predicted_answer
Average Metric: 19.0 / 7 (271.4): 30%|██▋ | 6/20 [00:02<00:03, 4.09it/s] Average Metric: 19.0 / 7 (271.4): 35%|███▏ | 7/20 [00:02<00:03, 3.71it/s]
Test Question: Should I use references in my schema?Faithful: 5 Detail: 5 Overall: 5 Predicted Answer: predicted_answer Test Question: Is it possible to create one-to-many relationships in the schema? Predicted Answer: predicted_answer
Average Metric: 20.8 / 8 (260.0): 35%|███▏ | 7/20 [00:02<00:03, 3.71it/s] Average Metric: 20.8 / 8 (260.0): 40%|███▌ | 8/20 [00:02<00:03, 3.71it/s] Average Metric: 22.8 / 9 (253.3): 40%|███▌ | 8/20 [00:02<00:03, 3.71it/s]
Faithful: 1 Detail: 5 Overall: 2 Faithful: 1 Detail: 3 Overall: 5
Average Metric: 24.400000000000002 / 10 (244.0): 45%|▍| 9/20 [00:02<00:02, 3. Average Metric: 24.400000000000002 / 10 (244.0): 50%|▌| 10/20 [00:02<00:02, 4
Test Question: What is the difference between `text` and `string` and `valueText` and `valueString`? Predicted Answer: predicted_answer Test Question: Do Weaviate classes have namespaces? Predicted Answer: predicted_answer Faithful: 1 Detail: 1 Overall: 5
Average Metric: 26.200000000000003 / 11 (238.2): 50%|▌| 10/20 [00:02<00:02, 4
Faithful: 1 Detail: 2 Overall: 5 Test Question: Are there restrictions on UUID formatting? Do I have to adhere to any standards? Predicted Answer: predicted_answer
Average Metric: 28.200000000000003 / 12 (235.0): 55%|▌| 11/20 [00:03<00:02, 4 Average Metric: 28.200000000000003 / 12 (235.0): 60%|▌| 12/20 [00:03<00:01, 4
Test Question: If I do not specify a UUID during adding data objects, will Weaviate create one automatically? Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 5
Average Metric: 30.400000000000002 / 13 (233.8): 60%|▌| 12/20 [00:03<00:01, 4
Faithful: 2 Detail: 5 Overall: 2 Test Question: Can I use Weaviate to create a traditional knowledge graph? Predicted Answer: predicted_answer
Average Metric: 32.400000000000006 / 14 (231.4): 65%|▋| 13/20 [00:03<00:01, 4 Average Metric: 32.400000000000006 / 14 (231.4): 70%|▋| 14/20 [00:03<00:01, 4 Average Metric: 36.00000000000001 / 15 (240.0): 70%|▋| 14/20 [00:03<00:01, 4.
Test Question: Why does Weaviate have a schema and not an ontology? Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 5 Faithful: 5 Detail: 5 Overall: 3 Test Question: What is the difference between a Weaviate data schema, ontologies and taxonomies? Predicted Answer: predicted_answer Test Question: How to deal with custom terminology? Predicted Answer: predicted_answer
Average Metric: 38.400000000000006 / 16 (240.0): 75%|▊| 15/20 [00:04<00:01, 4 Average Metric: 38.400000000000006 / 16 (240.0): 80%|▊| 16/20 [00:04<00:00, 4 Average Metric: 40.2 / 17 (236.5): 80%|█████▌ | 16/20 [00:04<00:00, 4.04it/s]
Faithful: 2 Detail: 5 Overall: 3 Faithful: 1 Detail: 5 Overall: 2
Average Metric: 44.2 / 18 (245.6): 85%|█████▉ | 17/20 [00:04<00:00, 4.04it/s] Average Metric: 44.2 / 18 (245.6): 90%|██████▎| 18/20 [00:04<00:00, 4.26it/s]
Test Question: How can you index data near-realtime without losing semantic meaning? Predicted Answer: predicted_answer Test Question: Why isn't there a text2vec-contextionary in my language? Predicted Answer: predicted_answer Faithful: 5 Detail: 5 Overall: 5
Average Metric: 45.800000000000004 / 19 (241.1): 90%|▉| 18/20 [00:04<00:00, 4
Faithful: 1 Detail: 1 Overall: 5 Test Question: How do you deal with words that have multiple meanings? Predicted Answer: predicted_answer
Average Metric: 48.6 / 20 (243.0): 95%|██████▋| 19/20 [00:04<00:00, 4.26it/s] Average Metric: 48.6 / 20 (243.0): 100%|███████| 20/20 [00:05<00:00, 3.99it/s] sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289e37460> ResourceWarning: Enable tracemalloc to get the object allocation traceback sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289e37e20> ResourceWarning: Enable tracemalloc to get the object allocation traceback
Faithful: 5 Detail: 2 Overall: 2 Average Metric: 48.6 / 20 (243.0%) Score: 243.0 for set: [4] Scores so far: [243.0, 243.0] Best score: 243.0
0%| | 0/20 [00:00<?, ?it/s]
Test Question: Why would I use Weaviate as my vector database? Predicted Answer: predicted_answer
5%|██▏ | 1/20 [00:00<00:07, 2.38it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: What is the difference between Weaviate and for example Elasticsearch? Predicted Answer: predicted_answer
10%|████▍ | 2/20 [00:00<00:08, 2.07it/s]
Faithful: No
Detail: 1
Overall: 5
Failed to run or to evaluate example Example({'question': 'What is the difference between Weaviate and for example Elasticsearch?'}) (input_keys={'question'}) with <function llm_metric at 0x281e95360> due to could not convert string to float: 'No'.
15%|██████▌ | 3/20 [00:01<00:07, 2.25it/s]
Test Question: Do you offer Weaviate as a managed service? Predicted Answer: predicted_answer Faithful: 1 Detail: 5 Overall: 1
20%|████████▊ | 4/20 [00:01<00:07, 2.23it/s]
Test Question: How should I configure the size of my instance? Predicted Answer: predicted_answer Faithful: 1 Detail: 2 Overall: 5 Test Question: Do I need to know about Docker (Compose) to use Weaviate? Predicted Answer: predicted_answer
25%|███████████ | 5/20 [00:02<00:06, 2.24it/s]
Faithful: 5 Detail: 5 Overall: 5 Bootstrapped 4 full traces after 6 examples in round 0.
0%| | 0/20 [00:00<?, ?it/s] Average Metric: 0.0 / 1 (0.0): 0%| | 0/20 [00:01<?, ?it/s] Average Metric: 0.0 / 1 (0.0): 5%|▌ | 1/20 [00:01<00:35, 1.87s/it]
Test Question: What is the difference between Weaviate and for example Elasticsearch? Predicted Answer: predicted_answer Faithful: No Detail: 1 Overall: 5 Error for example in dev set: could not convert string to float: 'No' Test Question: Why would I use Weaviate as my vector database? Predicted Answer: predicted_answer
Average Metric: 4.0 / 2 (200.0): 5%|▌ | 1/20 [00:02<00:35, 1.87s/it] Average Metric: 4.0 / 2 (200.0): 10%|█ | 2/20 [00:02<00:21, 1.21s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: How should I configure the size of my instance? Predicted Answer: predicted_answer Test Question: Do you offer Weaviate as a managed service? Predicted Answer: predicted_answer
Average Metric: 5.8 / 3 (193.3): 10%|█ | 2/20 [00:04<00:21, 1.21s/it] Average Metric: 5.8 / 3 (193.3): 15%|█▌ | 3/20 [00:04<00:27, 1.61s/it] Average Metric: 7.4 / 4 (185.0): 15%|█▌ | 3/20 [00:04<00:27, 1.61s/it]
Faithful: 1 Detail: 2 Overall: 5 Faithful: 1 Detail: 5 Overall: 1
Average Metric: 16.000000000000004 / 12 (133.3): 60%|▌| 12/20 [19:00<12:40, 95 Average Metric: 12.799999999999999 / 5 (256.0): 50%|▌| 5/10 [09:08<09:08, 109. Average Metric: 11.4 / 5 (228.0): 20%|█▊ | 4/20 [00:06<00:25, 1.61s/it] Average Metric: 11.4 / 5 (228.0): 25%|██▎ | 5/20 [00:06<00:18, 1.26s/it]
Test Question: Do I need to know about Docker (Compose) to use Weaviate? Predicted Answer: predicted_answer Faithful: 5 Detail: 5 Overall: 5 Test Question: What happens when the Weaviate Docker container restarts? Is my data in the Weaviate database lost? Predicted Answer: predicted_answer Test Question: Are there any 'best practices' or guidelines to consider when designing a schema? Predicted Answer: predicted_answer
Average Metric: 15.4 / 6 (256.7): 25%|██▎ | 5/20 [00:35<00:18, 1.26s/it] Average Metric: 15.4 / 6 (256.7): 30%|██▋ | 6/20 [00:35<02:06, 9.05s/it]
Faithful: 5 Detail: 5 Overall: 5
Average Metric: 19.2 / 7 (274.3): 30%|██▋ | 6/20 [00:35<02:06, 9.05s/it] Average Metric: 19.2 / 7 (274.3): 35%|███▏ | 7/20 [00:35<01:27, 6.70s/it]
Faithful: 5 Detail: 4 Overall: 5 Test Question: Should I use references in my schema? Predicted Answer: predicted_answer Test Question: Is it possible to create one-to-many relationships in the schema? Predicted Answer: predicted_answer
Average Metric: 22.0 / 8 (275.0): 35%|███▏ | 7/20 [00:59<01:27, 6.70s/it] Average Metric: 22.0 / 8 (275.0): 40%|███▌ | 8/20 [00:59<02:20, 11.67s/it]
Faithful: 2 Detail: 5 Overall: 5 Test Question: What is the difference between `text` and `string` and `valueText` and `valueString`? Predicted Answer: predicted_answer
Average Metric: 26.0 / 9 (288.9): 40%|███▌ | 8/20 [01:02<02:20, 11.67s/it] Average Metric: 26.0 / 9 (288.9): 45%|████ | 9/20 [01:02<01:40, 9.10s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Do Weaviate classes have namespaces? Predicted Answer: predicted_answer
Average Metric: 27.8 / 10 (278.0): 45%|███▌ | 9/20 [01:04<01:40, 9.10s/it] Average Metric: 27.8 / 10 (278.0): 50%|███▌ | 10/20 [01:04<01:09, 6.99s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Are there restrictions on UUID formatting? Do I have to adhere to any standards? Predicted Answer: predicted_answer
Average Metric: 29.400000000000002 / 11 (267.3): 50%|▌| 10/20 [01:24<01:09, 6 Average Metric: 29.400000000000002 / 11 (267.3): 55%|▌| 11/20 [01:24<01:37, 10
Faithful: 1 Detail: 1 Overall: 5
Average Metric: 31.8 / 12 (265.0): 55%|███▊ | 11/20 [01:26<01:37, 10.79s/it] Average Metric: 31.8 / 12 (265.0): 60%|████▏ | 12/20 [01:26<01:05, 8.20s/it]
Faithful: 1 Detail: 5 Overall: 5 Test Question: If I do not specify a UUID during adding data objects, will Weaviate create one automatically? Predicted Answer: predicted_answer Test Question: Can I use Weaviate to create a traditional knowledge graph? Predicted Answer: predicted_answer
Average Metric: 34.0 / 13 (261.5): 60%|████▏ | 12/20 [01:51<01:05, 8.20s/it] Average Metric: 34.0 / 13 (261.5): 65%|████▌ | 13/20 [01:51<01:33, 13.35s/it]
Faithful: 1 Detail: 5 Overall: 4 Test Question: Why does Weaviate have a schema and not an ontology? Predicted Answer: predicted_answer
Average Metric: 37.2 / 14 (265.7): 65%|████▌ | 13/20 [01:55<01:33, 13.35s/it] Average Metric: 37.2 / 14 (265.7): 70%|████▉ | 14/20 [01:55<01:02, 10.49s/it]
Faithful: 4 Detail: 5 Overall: 3 Test Question: What is the difference between a Weaviate data schema, ontologies and taxonomies? Predicted Answer: predicted_answer
Average Metric: 38.800000000000004 / 15 (258.7): 70%|▋| 14/20 [02:09<01:02, 10 Average Metric: 38.800000000000004 / 15 (258.7): 75%|▊| 15/20 [02:09<00:57, 11
Faithful: 1 Detail: 1 Overall: 5 Test Question: How to deal with custom terminology? Predicted Answer: predicted_answer
Average Metric: 40.6 / 16 (253.8): 75%|█████▎ | 15/20 [02:11<00:57, 11.56s/it] Average Metric: 40.6 / 16 (253.8): 80%|█████▌ | 16/20 [02:11<00:34, 8.56s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: How can you index data near-realtime without losing semantic meaning? Predicted Answer: predicted_answer
Average Metric: 42.2 / 17 (248.2): 80%|█████▌ | 16/20 [02:15<00:34, 8.56s/it] Average Metric: 42.2 / 17 (248.2): 85%|█████▉ | 17/20 [02:15<00:22, 7.39s/it]
Faithful: 1 Detail: 1 Overall: 5 Test Question: Why isn't there a text2vec-contextionary in my language? Predicted Answer: predicted_answer
Average Metric: 46.2 / 18 (256.7): 85%|█████▉ | 17/20 [02:32<00:22, 7.39s/it] Average Metric: 46.2 / 18 (256.7): 90%|██████▎| 18/20 [02:32<00:20, 10.26s/it]
Faithful: 5 Detail: 5 Overall: 5
Average Metric: 48.0 / 19 (252.6): 90%|██████▎| 18/20 [02:37<00:20, 10.26s/it] Average Metric: 48.0 / 19 (252.6): 95%|██████▋| 19/20 [02:37<00:08, 8.55s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: How do you deal with words that have multiple meanings? Predicted Answer: predicted_answer
Average Metric: 52.0 / 20 (260.0): 95%|██████▋| 19/20 [03:05<00:08, 8.55s/it] Average Metric: 52.0 / 20 (260.0): 100%|███████| 20/20 [03:05<00:00, 9.29s/it] sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289e37a00> ResourceWarning: Enable tracemalloc to get the object allocation traceback sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289e37e80> ResourceWarning: Enable tracemalloc to get the object allocation traceback
Faithful: 5 Detail: 5 Overall: 5 Average Metric: 52.0 / 20 (260.0%) Score: 260.0 for set: [4] New best score: 260.0 for seed -1 Scores so far: [243.0, 243.0, 260.0] Best score: 260.0 Average of max per entry across top 1 scores: 2.6 Average of max per entry across top 2 scores: 2.75 Average of max per entry across top 3 scores: 2.75 Average of max per entry across top 5 scores: 2.75 Average of max per entry across top 8 scores: 2.75 Average of max per entry across top 9999 scores: 2.75
0%| | 0/20 [00:00<?, ?it/s]
Test Question: Do Weaviate classes have namespaces? Predicted Answer: predicted_answer
5%|██▏ | 1/20 [00:00<00:09, 1.91it/s]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Why isn't there a text2vec-contextionary in my language? Predicted Answer: predicted_answer
10%|████▍ | 2/20 [00:01<00:11, 1.63it/s]
Faithful: 1 Detail: 1 Overall: 5 Test Question: How to deal with custom terminology? Predicted Answer: predicted_answer
15%|██████▌ | 3/20 [00:01<00:09, 1.85it/s]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Why does Weaviate have a schema and not an ontology? Predicted Answer: predicted_answer
20%|████████▊ | 4/20 [00:02<00:08, 1.90it/s]
Faithful: 5 Detail: 5 Overall: 3 Bootstrapped 4 full traces after 5 examples in round 0.
0%| | 0/20 [00:00<?, ?it/s]
Test Question: What is the difference between Weaviate and for example Elasticsearch? Predicted Answer: predicted_answer Test Question: Why would I use Weaviate as my vector database? Predicted Answer: predicted_answer
Average Metric: 1.6 / 1 (160.0): 5%|▌ | 1/20 [00:18<05:58, 18.86s/it]
Faithful: 2 Detail: 2 Overall: 2 Test Question: Do you offer Weaviate as a managed service? Predicted Answer: predicted_answer
Average Metric: 5.6 / 2 (280.0): 10%|█ | 2/20 [00:22<02:59, 9.99s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: How should I configure the size of my instance? Predicted Answer: predicted_answer
Average Metric: 7.6 / 3 (253.3): 15%|█▌ | 3/20 [00:41<03:58, 14.04s/it]
Faithful: 1 Detail: 3 Overall: 5 Test Question: Do I need to know about Docker (Compose) to use Weaviate? Predicted Answer: predicted_answer
Average Metric: 11.0 / 4 (275.0): 20%|█▊ | 4/20 [00:44<02:35, 9.71s/it]
Faithful: 5 Detail: 2 Overall: 5 Test Question: What happens when the Weaviate Docker container restarts? Is my data in the Weaviate database lost? Predicted Answer: predicted_answer
Average Metric: 14.6 / 5 (292.0): 25%|██▎ | 5/20 [01:03<03:16, 13.12s/it]
Faithful: 5 Detail: 3 Overall: 5 Test Question: Are there any 'best practices' or guidelines to consider when designing a schema? Predicted Answer: predicted_answer
Average Metric: 17.4 / 6 (290.0): 30%|██▋ | 6/20 [01:10<02:34, 11.05s/it]
Faithful: 2 Detail: 5 Overall: 5 Test Question: Should I use references in my schema? Predicted Answer: predicted_answer
Average Metric: 20.799999999999997 / 7 (297.1): 35%|▎| 7/20 [01:27<02:46, 12.8
Faithful: 5 Detail: 2 Overall: 5 Test Question: Is it possible to create one-to-many relationships in the schema? Predicted Answer: predicted_answer
Average Metric: 24.599999999999998 / 8 (307.5): 40%|▍| 8/20 [01:36<02:18, 11.5
Faithful: 5 Detail: 4 Overall: 5 Test Question: What is the difference between `text` and `string` and `valueText` and `valueString`? Predicted Answer: predicted_answer
Average Metric: 28.2 / 9 (313.3): 45%|████ | 9/20 [01:53<02:27, 13.44s/it]
Faithful: 5 Detail: 3 Overall: 5 Test Question: Do Weaviate classes have namespaces? Predicted Answer: predicted_answer
Average Metric: 30.0 / 10 (300.0): 50%|███▌ | 10/20 [01:54<01:36, 9.69s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Are there restrictions on UUID formatting? Do I have to adhere to any standards? Predicted Answer: predicted_answer
Average Metric: 31.8 / 11 (289.1): 55%|███▊ | 11/20 [02:04<01:26, 9.59s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: If I do not specify a UUID during adding data objects, will Weaviate create one automatically? Predicted Answer: predicted_answer
Average Metric: 35.4 / 12 (295.0): 60%|████▏ | 12/20 [02:15<01:21, 10.20s/it]
Faithful: 5 Detail: 3 Overall: 5 Test Question: Can I use Weaviate to create a traditional knowledge graph? Predicted Answer: predicted_answer
Average Metric: 39.0 / 13 (300.0): 65%|████▌ | 13/20 [02:33<01:26, 12.42s/it]
Faithful: 5 Detail: 3 Overall: 5 Test Question: Why does Weaviate have a schema and not an ontology? Predicted Answer: predicted_answer
Average Metric: 42.6 / 14 (304.3): 70%|████▉ | 14/20 [02:36<00:58, 9.68s/it]
Faithful: 5 Detail: 5 Overall: 3 Test Question: What is the difference between a Weaviate data schema, ontologies and taxonomies? Predicted Answer: predicted_answer
Average Metric: 44.2 / 15 (294.7): 75%|█████▎ | 15/20 [03:01<01:11, 14.31s/it]
Faithful: 1 Detail: 1 Overall: 5 Test Question: How to deal with custom terminology? Predicted Answer: predicted_answer
Average Metric: 46.0 / 16 (287.5): 80%|█████▌ | 16/20 [03:03<00:42, 10.59s/it]
Faithful: 1 Detail: 2 Overall: 5
Average Metric: 49.2 / 17 (289.4): 85%|█████▉ | 17/20 [03:04<00:22, 7.47s/it]
Faithful: 5 Detail: 1 Overall: 5 Test Question: How can you index data near-realtime without losing semantic meaning? Predicted Answer: predicted_answer Test Question: Why isn't there a text2vec-contextionary in my language? Predicted Answer: predicted_answer
Average Metric: 50.800000000000004 / 18 (282.2): 90%|▉| 18/20 [03:06<00:12, 6
Faithful: 1 Detail: 1 Overall: 5 Test Question: How do you deal with words that have multiple meanings? Predicted Answer: predicted_answer
Average Metric: 54.400000000000006 / 19 (286.3): 95%|▉| 19/20 [03:27<00:10, 10
Faithful: 5 Detail: 3 Overall: 5
Average Metric: 57.60000000000001 / 20 (288.0): 100%|█| 20/20 [03:29<00:00, 10. sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289d07700> ResourceWarning: Enable tracemalloc to get the object allocation traceback sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289ca6b60> ResourceWarning: Enable tracemalloc to get the object allocation traceback
Faithful: 5 Detail: 1 Overall: 5 Average Metric: 57.60000000000001 / 20 (288.0%) Score: 288.0 for set: [4] New best score: 288.0 for seed 0 Scores so far: [243.0, 243.0, 260.0, 288.0] Best score: 288.0 Average of max per entry across top 1 scores: 2.8800000000000003 Average of max per entry across top 2 scores: 3.06 Average of max per entry across top 3 scores: 3.11 Average of max per entry across top 5 scores: 3.11 Average of max per entry across top 8 scores: 3.11 Average of max per entry across top 9999 scores: 3.11
0%| | 0/20 [00:00<?, ?it/s]
Test Question: Are there restrictions on UUID formatting? Do I have to adhere to any standards? Predicted Answer: predicted_answer
5%|██▏ | 1/20 [00:00<00:09, 1.91it/s]
Faithful: 1 Detail: 3 Overall: 5 Test Question: What happens when the Weaviate Docker container restarts? Is my data in the Weaviate database lost? Predicted Answer: predicted_answer
10%|████▍ | 2/20 [00:00<00:08, 2.12it/s]
Faithful: 4 Detail: 5 Overall: 5 Bootstrapped 2 full traces after 3 examples in round 0.
0%| | 0/20 [00:00<?, ?it/s]
Test Question: What is the difference between Weaviate and for example Elasticsearch? Predicted Answer: predicted_answer Test Question: Why would I use Weaviate as my vector database? Predicted Answer: predicted_answer
Average Metric: 4.0 / 1 (400.0): 5%|▌ | 1/20 [00:19<06:12, 19.59s/it]
Faithful: 5 Detail: 5 Overall: 5
Average Metric: 6.0 / 2 (300.0): 10%|█ | 2/20 [00:20<02:39, 8.87s/it]
Faithful: 1 Detail: 5 Overall: 3 Test Question: Do you offer Weaviate as a managed service? Predicted Answer: predicted_answer Test Question: How should I configure the size of my instance? Predicted Answer: predicted_answer
Average Metric: 7.8 / 3 (260.0): 15%|█▌ | 3/20 [00:36<03:22, 11.89s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Do I need to know about Docker (Compose) to use Weaviate? Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 3
Average Metric: 13.0 / 5 (260.0): 25%|██▎ | 5/20 [00:41<01:30, 6.01s/it]
Test Question: What happens when the Weaviate Docker container restarts? Is my data in the Weaviate database lost? Predicted Answer: predicted_answer Faithful: 4 Detail: 5 Overall: 5
Average Metric: 14.8 / 6 (246.7): 30%|██▋ | 6/20 [00:41<00:56, 4.05s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Are there any 'best practices' or guidelines to consider when designing a schema? Predicted Answer: predicted_answer Test Question: Should I use references in my schema? Predicted Answer: predicted_answer
Average Metric: 18.8 / 7 (268.6): 35%|███▏ | 7/20 [01:04<02:12, 10.22s/it]
Faithful: 5 Detail: 5 Overall: 5
Average Metric: 22.8 / 8 (285.0): 40%|███▌ | 8/20 [01:04<01:26, 7.19s/it]
Faithful: 5 Detail: 5 Overall: 5
Average Metric: 24.8 / 9 (275.6): 40%|███▌ | 8/20 [01:06<01:26, 7.19s/it]
Test Question: Is it possible to create one-to-many relationships in the schema? Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 5
Average Metric: 24.8 / 9 (275.6): 45%|████ | 9/20 [01:06<00:58, 5.35s/it]
Test Question: What is the difference between `text` and `string` and `valueText` and `valueString`? Predicted Answer: predicted_answer Test Question: Do Weaviate classes have namespaces? Predicted Answer: predicted_answer
Average Metric: 26.8 / 10 (268.0): 50%|███▌ | 10/20 [01:26<01:38, 9.89s/it]
Faithful: 1 Detail: 3 Overall: 5 Test Question: Are there restrictions on UUID formatting? Do I have to adhere to any standards? Predicted Answer: predicted_answer
Average Metric: 28.8 / 11 (261.8): 55%|███▊ | 11/20 [01:34<01:23, 9.31s/it]
Faithful: 1 Detail: 3 Overall: 5 Test Question: If I do not specify a UUID during adding data objects, will Weaviate create one automatically? Predicted Answer: predicted_answer
Average Metric: 30.400000000000002 / 12 (253.3): 60%|▌| 12/20 [01:52<01:36, 12
Faithful: 1 Detail: 1 Overall: 5
Average Metric: 32.800000000000004 / 13 (252.3): 65%|▋| 13/20 [01:52<00:59, 8
Faithful: 1 Detail: 5 Overall: 5 Test Question: Can I use Weaviate to create a traditional knowledge graph? Predicted Answer: predicted_answer Test Question: Why does Weaviate have a schema and not an ontology? Predicted Answer: predicted_answer
Average Metric: 34.400000000000006 / 14 (245.7): 70%|▋| 14/20 [02:14<01:14, 12
Faithful: 1 Detail: 1 Overall: 5 Test Question: What is the difference between a Weaviate data schema, ontologies and taxonomies? Predicted Answer: predicted_answer
Average Metric: 36.800000000000004 / 15 (245.3): 75%|▊| 15/20 [02:26<01:02, 12
Faithful: 2 Detail: 5 Overall: 3 Test Question: How to deal with custom terminology? Predicted Answer: predicted_answer
Average Metric: 36.800000000000004 / 16 (230.0): 80%|▊| 16/20 [02:35<00:45, 11
Error for example in dev set: 'NoneType' object is not iterable Test Question: How can you index data near-realtime without losing semantic meaning? Predicted Answer: predicted_answer
Average Metric: 36.800000000000004 / 17 (216.5): 85%|▊| 17/20 [02:40<00:28, 9
Faithful: No Detail: 1 Overall: 5 Error for example in dev set: could not convert string to float: 'No' Test Question: Why isn't there a text2vec-contextionary in my language? Predicted Answer: predicted_answer
Average Metric: 38.6 / 18 (214.4): 90%|██████▎| 18/20 [03:00<00:25, 12.67s/it]
Faithful: 1 Detail: 2 Overall: 5
Average Metric: 42.2 / 19 (222.1): 95%|██████▋| 19/20 [03:02<00:09, 9.44s/it]
Faithful: 5 Detail: 5 Overall: 3 Test Question: How do you deal with words that have multiple meanings? Predicted Answer: predicted_answer
Average Metric: 46.2 / 20 (231.0): 100%|███████| 20/20 [03:34<00:00, 10.74s/it]
Faithful: 5 Detail: 5 Overall: 5 Average Metric: 46.2 / 20 (231.0%) Score: 231.0 for set: [4] Scores so far: [243.0, 243.0, 260.0, 288.0, 231.0] Best score: 288.0 Average of max per entry across top 1 scores: 2.8800000000000003 Average of max per entry across top 2 scores: 3.06 Average of max per entry across top 3 scores: 3.11 Average of max per entry across top 5 scores: 3.1399999999999997 Average of max per entry across top 8 scores: 3.1399999999999997 Average of max per entry across top 9999 scores: 3.1399999999999997 5 candidate programs found.
sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289ca6e60> ResourceWarning: Enable tracemalloc to get the object allocation traceback sys:1: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUSH) at 0x289ca48e0> ResourceWarning: Enable tracemalloc to get the object allocation traceback
[59]
Prediction( , answer='Cross encoders are ranking models used for content-based re-ranking. They are used to give a score indicating the relevance of a document to a query.' ,)
[60]
Answer questions based on the context.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ${answer}
---
Context:
[1] «The schema is the place to define, among other things, the data type and vectorizer to be used, as well as cross-references between classes. As a corollary, the vectorization process can be modified for each class by setting the relevant schema options. In fact, you can [define the data schema](/developers/weaviate/manage-data/collections) for each class individually. All this means that you can also use the schema to tweak Weaviate's vectorization behavior. The relevant variables for vectorization are `dataType` and those listed under `moduleConfig` at both the class level and property level.»
[2] «
### [Weaviate Academy](/developers/academy) & [Workshops](/learn/workshops)
Weaviate Academy and Workshops have had a fantastic year of learning and growth! We've been focusing on ensuring everyone has the chance to understand and use vector databases and get a grasp on Generative AI and data handling. Every week, [Zain](https://www.linkedin.com/in/zainhas/), [JP](https://www.linkedin.com/in/jphwang/), [Daniel](https://www.linkedin.com/in/malgamves/), and [Duda](https://www.linkedin.com/in/dudanogueira/) have been running introductory workshops on vector databases and Weaviate, which have been a hit. Plus, we're super excited about [JP Hwang](https://www.linkedin.com/in/jphwang/)'s initiative, the Weaviate Academy. It's a program that takes you from the basics all the way to production-level skills. Later in the year, we teamed up with [DeepLearningAI](https://www.deeplearning.ai/) to create a short course with [Sebastian](https://www.linkedin.com/in/sebawita/) and [Zain](https://www.linkedin.com/in/zainhas/), "[Vector Databases: from Embeddings to Applications with Weaviate](https://www.deeplearning.ai/short-courses/vector-databases-embeddings-applications/)." It’s been a year packed with learning.»
[3] «---
title: Weaviate 1.15 release
slug: weaviate-1-15-release
authors: [connor, erika, laura, sebastian]
date: 2022-09-07
tags: ['release']
image: ./img/hero.png
description: "Weaviate 1.15 introduces Cloud-native Backups, Memory Optimizations, faster Filtered Aggregations and Ordered Imports, new Distance Metrics and new Weaviate modules."
---
<!-- truncate -->
We are happy to announce the release of Weaviate 1.15, which is packed with great features, significant performance improvements, new distance metrics and modules, and many smaller improvements and fixes. ## The brief
If you like your content brief and to the point, here is the TL;DR of this release:
1. [☁️Cloud-native backups](#cloud-native-backups) - allows you to configure your environment to create backups - of selected classes or the whole database - straight into AWS S3, GCS or local filesystem
1. [Reduced memory usage](#reduced-memory-usage) - we found new ways to optimize memory usage, reducing RAM usage by 10-30%. 1.»
Question: Do Weaviate classes have namespaces?
Reasoning: Let's think step by step in order to produce the answer. We need to examine the given context to determine if Weaviate classes have namespaces.
Answer: The given context does not provide any information about whether Weaviate classes have namespaces.
---
Context:
[1] «You can even run transformer models locally with [`text2vec-transformers`](/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-transformers), and modules such as [`multi2vec-clip`](/developers/weaviate/modules/retriever-vectorizer-modules/multi2vec-clip) can convert images and text to vectors using a CLIP model. But they all perform the same core task—which is to represent the “meaning” of the original data as a set of numbers. And that’s why semantic search works so well. import WhatNext from '/_includes/what-next.mdx'
<WhatNext />»
[2] «## How are vector embeddings generated? The magic of vector search resides primarily in how the embeddings are generated for each entity and the query, and secondarily in how to efficiently search within very large datasets (see our [“Why is Vector Search so Fast”](/blog/why-is-vector-search-so-fast) article for the latter). As we mentioned, vector embeddings can be generated for various media types such as text, images, audio and others. For text, vectorization techniques have evolved tremendously over the last decade, from the venerable [word2vec](https://en.wikipedia.org/wiki/Word2vec) ([2013](https://code.google.com/archive/p/word2vec/)), to the state-of-the-art transformer models era, spurred by the release of [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)) in [2018](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html). ### Word-level dense vector models (word2vec, GloVe, etc.)
[word2vec](https://wiki.pathmind.com/word2vec) is a [family of model architectures](https://www.tensorflow.org/tutorials/text/word2vec) that introduced the idea of “dense” vectors in language processing, in which all values are non-zero.»
[3] «However, this can be done by aggregating vectors of constituent words, which is often done by incorporating weightings such that certain words are weighted more heavily than others. However, word2vec still suffers from important limitations:
* it doesn’t address words with multiple meanings (polysemantic): “run”, “set”, “go”, or “take” each have [over 300 meanings](https://www.insider.com/words-with-the-most-definitions-2019-1) (!)
* it doesn’t address words with ambiguous meanings: “to consult” can be its own antonym, like [many other words](http://www.dailywritingtips.com/75-contronyms-words-with-contradictory-meanings/)
Which takes us to the next, state-of-the-art, models. ### Transformer models (BERT, ELMo, and others)
The current state-of-the-art models are based on what’s called a “transformer” architecture as introduced in [this paper](https://arxiv.org/abs/1706.03762). [Transformer models](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) such as BERT and its successors improve search accuracy, [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall) by looking at every word’s context to create full contextual embeddings (though [the exact mechanism of BERT’s success is not fully understood](https://aclanthology.org/D19-1445/)). Unlike word2vec embeddings which are context-agnostic, transformer-generated embeddings take the entire input text into account—each occurrence of a word has its own embedding that is modified by the surrounding text.»
Question: Why isn't there a text2vec-contextionary in my language?
Reasoning: Let's think step by step in order to produce the answer. We need to understand the context of the given information. The context mentions different techniques for generating vector embeddings for text, such as word2vec and transformer models like BERT. It also mentions that transformer models take the entire input text into account and generate context-aware embeddings. However, there is no specific mention of a "text2vec-contextionary" in the given context. Therefore, we can conclude that the reason there isn't a text2vec-contextionary in your language is not provided in the given information.
Answer: The reason why there isn't a text2vec-contextionary in your language is not provided in the given information.
---
Context:
[1] «One technique is to prompt it with:
`please output a relevance score on a scale of 1 to 100.`
I think the second strategy is a bit more interesting, in which we put as many documents as we can in the input and ask the LLM to rank them. The key to making this work is the emergence of LLMs to follow instructions, especially with formatting their output. By prompting this ranking with “please output the ranking as a dictionary of IDs with the key equal to the rank and the value equal to the document id”. Also interesting is the question around how many documents we can rank like this and how expensive it is. For example, if we want to re-rank 100 documents, but can only fit 5 in the input at a time, we are going to need to construct some kind of tournament-style decomposition of the ranking task.»
[2] «What I love about all this work, like self-ask, chain of thought, we're developing new querying languages. This is like us inventing SQL, except that we didn't design the database. The database came into being and we have to figure out how to interact with it. That example I mentioned about the IPython interaction, like that's a, again, it's a new querying language. And I honestly thought the most potent part of the self-ask wasn't even necessarily the self-ask part.»
[3] «have published “Large Language Models are easily distracted by irrelevant context”, highlighting how problematic bad precision in search can be for retrieval-augmented generation. The recent developments in LLM agent tooling such as LangChain, LlamaIndex, and recent projects such as AutoGPT or Microsoft’s Semantic Kernel are paving the way towards letting LLMs run for a while to complete complex tasks. By ranking each handoff from search to prompt, we can achieve better results in each intermediate task. Thus when we leave an LLM running overnight to research the future of ranking models, we can expect a better final result in the morning!
import WhatNext from '/_includes/what-next.mdx'
<WhatNext />»
Question: How to deal with custom terminology?
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the given context to understand how to deal with custom terminology.
Answer: The given context does not provide any information about how to deal with custom terminology. Therefore, we cannot determine the answer based on the given context.
---
Context:
[1] «The schema is the place to define, among other things, the data type and vectorizer to be used, as well as cross-references between classes. As a corollary, the vectorization process can be modified for each class by setting the relevant schema options. In fact, you can [define the data schema](/developers/weaviate/manage-data/collections) for each class individually. All this means that you can also use the schema to tweak Weaviate's vectorization behavior. The relevant variables for vectorization are `dataType` and those listed under `moduleConfig` at both the class level and property level.»
[2] «Let's recap exactly what Weaviate does. ### Text vectorization in Weaviate
import VectorizationBehavior from '/_includes/vectorization.behavior.mdx';
<VectorizationBehavior/>
Now that we understand this, you might be asking - is it possible to customize the vectorization process? The answer is, yes, of course. ## Tweaking text2vec vectorization in Weaviate
Some of you might have noticed that we have not done anything at all with the schema so far. This meant that the schema used is one generated by the auto-schema feature and thus the vectorizations were carried out using default options.»
[3] «The past for vector searching definitely was not a “simpler time”, and the appeal of modern vector databases like Weaviate is pretty clear given this context. But while the future is here, it isn't yet perfect. Tools like Weaviate can seem like a magician's mystery box. Our users in turn ask us *exactly* how Weaviate does its magic; how it turns all of that data into vectors, and how to control the process. So let's take a look inside the magic box together in this post.»
Question: Why does Weaviate have a schema and not an ontology?
Reasoning: Let's think step by step in order to produce the answer. We can gather information from the given context. In [1], it is mentioned that the schema is used to define the data type, vectorizer, and cross-references between classes in Weaviate. Additionally, the schema can be used to tweak Weaviate's vectorization behavior. In [2], it is stated that the schema used in Weaviate is generated by the auto-schema feature.
Answer: Weaviate has a schema instead of an ontology because the schema is used to define data types, vectorizers, and cross-references between classes, and it allows for customization of the vectorization process.
---
Context:
[1] «[Cross Encoders](#cross-encoders) (collapsing the use of Large Language Models for ranking into this category as well)
1. [Metadata Rankers](#metadata-rankers)
1. [Score Rankers](#score-rankers)
## Cross Encoders
Cross Encoders are one of the most well known ranking models for content-based re-ranking. There is quite a collection of pre-trained cross encoders available on [sentence transformers](https://www.sbert.net/docs/pretrained_cross-encoders.html). We are currently envisioning interfacing cross encoders with Weaviate using the following syntax.»
[2] «Thus, quality at the expense of speed, becomes more interesting. ### LLMs as Cross Encoders
So, let’s dive into the LLM hype a little more, how can we use LLMs for re-ranking? There are generally 2 ways to do this. The first strategy is identical to the cross encoder, we give the LLM the [query, document] input and prompt it to output a score of how relevant the document is to the query. The tricky thing with this is bounding the score.»
[3] «They offer the advantage of further reasoning about the relevance of results without needing specialized training. Cross Encoders can be interfaced with Weaviate to re-rank search results, trading off performance for slower search speed. * **Metadata Rankers** are context-based re-rankers that use symbolic features to rank relevance. They take into account user and document features, such as age, gender, location, preferences, release year, genre, and box office, to predict the relevance of candidate documents. By incorporating metadata features, these rankers offer a more personalized and context-aware search experience.»
Question: What do cross encoders do?
Reasoning: Let's think step by step in order to produce the answer. We can gather information from the given context. In [1], it is mentioned that cross encoders are ranking models used for content-based re-ranking. They are often used for re-ranking search results. In [2], it is stated that cross encoders are used to give the LLM the query and document input and prompt it to output a score of how relevant the document is to the query.
Answer: Cross encoders are ranking models used for content-based re-ranking. They are used to give a score indicating the relevance of a document to a query.
[61]
0%| | 0/10 [00:00<?, ?it/s]
Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer
Average Metric: 3.6 / 1 (360.0): 10%|█ | 1/10 [00:21<03:12, 21.35s/it]
Faithful: 5 Detail: 3 Overall: 5 Test Question: How can I retrieve the total object count in a class? Predicted Answer: predicted_answer
Average Metric: 5.4 / 2 (270.0): 20%|██ | 2/10 [00:44<02:58, 22.35s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: How do I get the cosine similarity from Weaviate's certainty? Predicted Answer: predicted_answer
Average Metric: 7.2 / 3 (240.0): 30%|███ | 3/10 [01:11<02:50, 24.30s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: The quality of my search results change depending on the specified limit. Why? How can I fix this? Predicted Answer: predicted_answer
Average Metric: 9.0 / 4 (225.0): 40%|████ | 4/10 [01:36<02:28, 24.73s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Why did you use GraphQL instead of SPARQL? Predicted Answer: predicted_answer
Average Metric: 10.6 / 5 (212.0): 50%|████▌ | 5/10 [01:38<01:22, 16.43s/it]
Faithful: 1 Detail: 1 Overall: 5 Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
Average Metric: 12.8 / 6 (213.3): 60%|█████▍ | 6/10 [02:01<01:15, 18.86s/it]
Faithful: 1 Detail: 4 Overall: 5 Test Question: What is best practice for updating data? Predicted Answer: predicted_answer
Average Metric: 16.0 / 7 (228.6): 70%|██████▎ | 7/10 [02:24<01:00, 20.19s/it]
Faithful: 5 Detail: 1 Overall: 5 Test Question: Can I connect my own module? Predicted Answer: predicted_answer
Average Metric: 19.6 / 8 (245.0): 80%|███████▏ | 8/10 [02:48<00:42, 21.48s/it]
Faithful: 5 Detail: 3 Overall: 5 Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
Average Metric: 23.200000000000003 / 9 (257.8): 90%|▉| 9/10 [03:28<00:27, 27.2
Faithful: 5 Detail: 3 Overall: 5 Test Question: Does Weaviate use Hnswlib? Predicted Answer: predicted_answer
Average Metric: 26.400000000000002 / 10 (264.0): 100%|█| 10/10 [03:41<00:00, 22
Faithful: 5 Detail: 3 Overall: 3 Average Metric: 26.400000000000002 / 10 (264.0%)
264.0
BayesianSignatureOptimizer
[62]
0%| | 0/10 [00:00<?, ?it/s]
Test Question: What is best practice for updating data? Predicted Answer: predicted_answer
10%|████▍ | 1/10 [00:00<00:03, 2.36it/s]
Faithful: 5 Detail: 3 Overall: 4 Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
20%|████████▊ | 2/10 [00:00<00:03, 2.65it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Does Weaviate use Hnswlib? Predicted Answer: predicted_answer
30%|█████████████▏ | 3/10 [00:01<00:03, 2.07it/s]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Can I connect my own module? Predicted Answer: predicted_answer
40%|█████████████████▌ | 4/10 [00:01<00:02, 2.13it/s]
Faithful: 5 Detail: 5 Overall: 5 Bootstrapped 4 full traces after 5 examples in round 0.
0%| | 0/10 [00:00<?, ?it/s]
Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
10%|████▍ | 1/10 [00:01<00:10, 1.20s/it]
Faithful: 1 Detail: 1 Overall: 5 Test Question: Does Weaviate use Hnswlib? Predicted Answer: predicted_answer
20%|████████▊ | 2/10 [00:01<00:06, 1.28it/s]
Faithful: 1 Detail: 2 Overall: 5 Test Question: The quality of my search results change depending on the specified limit. Why? How can I fix this? Predicted Answer: predicted_answer
30%|█████████████▏ | 3/10 [00:02<00:04, 1.66it/s]
Faithful: 2 Detail: 5 Overall: 4
40%|█████████████████▌ | 4/10 [00:02<00:03, 1.62it/s]
Test Question: Why did you use GraphQL instead of SPARQL? Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 5 Bootstrapped 4 full traces after 5 examples in round 0.
10%|████▍ | 1/10 [00:00<00:03, 2.57it/s]
Test Question: How can I retrieve the total object count in a class? Predicted Answer: predicted_answer Faithful: 1 Detail: 4 Overall: 1 Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
20%|████████▊ | 2/10 [00:00<00:03, 2.53it/s]
Faithful: 1 Detail: 1 Overall: 5 Test Question: What is best practice for updating data? Predicted Answer: predicted_answer
30%|█████████████▏ | 3/10 [00:01<00:02, 2.66it/s]
Faithful: 5 Detail: 3 Overall: 4
40%|█████████████████▌ | 4/10 [00:01<00:02, 2.60it/s]
Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 5 Bootstrapped 4 full traces after 5 examples in round 0.
0%| | 0/10 [00:00<?, ?it/s]
Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
10%|████▍ | 1/10 [00:00<00:04, 2.03it/s]
Faithful: 5 Detail: 5 Overall: 5
20%|████████▊ | 2/10 [00:00<00:03, 2.07it/s]
Test Question: How do I get the cosine similarity from Weaviate's certainty? Predicted Answer: predicted_answer Faithful: 1 Detail: 5 Overall: 1 Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer
30%|█████████████▏ | 3/10 [00:01<00:03, 2.11it/s]
Faithful: 1 Detail: 3 Overall: 5 Test Question: Can I connect my own module? Predicted Answer: predicted_answer
40%|█████████████████▌ | 4/10 [00:01<00:02, 2.12it/s] [I 2024-02-11 14:46:53,175] A new study created in memory with name: no-name-5e566642-02dd-4eee-8691-53989ee359f0
Faithful: 5 Detail: 5 Overall: 5 Bootstrapped 4 full traces after 5 examples in round 0.
0%| | 0/10 [00:00<?, ?it/s]
Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer
Average Metric: 2.0 / 1 (200.0): 10%|█ | 1/10 [00:00<00:07, 1.28it/s]
Faithful: 1 Detail: 3 Overall: 5 Test Question: How can I retrieve the total object count in a class? Predicted Answer: predicted_answer
Average Metric: 3.8 / 2 (190.0): 20%|██ | 2/10 [00:19<01:30, 11.34s/it]
Faithful: 1 Detail: 5 Overall: 2
Average Metric: 5.4 / 3 (180.0): 30%|███ | 3/10 [00:20<00:44, 6.40s/it]
Test Question: How do I get the cosine similarity from Weaviate's certainty? Predicted Answer: predicted_answer Faithful: 1 Detail: 5 Overall: 1 Test Question: The quality of my search results change depending on the specified limit. Why? How can I fix this? Predicted Answer: predicted_answer
Average Metric: 7.4 / 4 (185.0): 40%|████ | 4/10 [00:44<01:20, 13.37s/it]
Faithful: 1 Detail: 5 Overall: 3 Test Question: Why did you use GraphQL instead of SPARQL? Predicted Answer: predicted_answer
Average Metric: 11.4 / 5 (228.0): 50%|████▌ | 5/10 [01:04<01:19, 15.82s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
Average Metric: 13.0 / 6 (216.7): 60%|█████▍ | 6/10 [01:29<01:16, 19.08s/it]
Faithful: 1 Detail: 4 Overall: 2 Test Question: What is best practice for updating data? Predicted Answer: predicted_answer
Average Metric: 17.0 / 7 (242.9): 70%|██████▎ | 7/10 [02:02<01:11, 23.69s/it]
Faithful: 5 Detail: 5 Overall: 5
Average Metric: 21.0 / 8 (262.5): 80%|███████▏ | 8/10 [02:03<00:32, 16.28s/it]
Test Question: Can I connect my own module? Predicted Answer: predicted_answer Faithful: 5 Detail: 5 Overall: 5 Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
Average Metric: 25.0 / 9 (277.8): 90%|████████ | 9/10 [02:03<00:11, 11.33s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Does Weaviate use Hnswlib? Predicted Answer: predicted_answer
Average Metric: 29.0 / 10 (290.0): 100%|███████| 10/10 [02:27<00:00, 14.72s/it]
[I 2024-02-11 14:49:20,377] Trial 0 finished with value: 290.0 and parameters: {'10883682544_predictor_instruction': 1, '10883682544_predictor_demos': 4}. Best is trial 0 with value: 290.0.
Faithful: 5
Detail: 5
Overall: 5
Average Metric: 29.0 / 10 (290.0%)
Assess the context and answer the given questions that are predominantly about software usage, process optimization, and troubleshooting. Focus on providing accurate information related to tech or software-related queries.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Best Answer: ${answer}
---
Context:
[1] «Some models, such as [CLIP](https://openai.com/blog/clip/), are capable of vectorizing multiple data types (images and text in this case) into one vector space, so that an image can be searched by its content using only text. ## Vector embeddings with Weaviate
For this reason, Weaviate is configured to support many different vectorizer models and vectorizer service providers. You can even [bring your own vectors](/developers/weaviate/starter-guides/custom-vectors), for example if you already have a vectorization pipeline available, or if none of the publicly available models are suitable for you. For one, Weaviate supports using any Hugging Face models through our `text2vec-hugginface` module, so that you can [choose one of the many sentence transformers published on Hugging Face](/blog/how-to-choose-a-sentence-transformer-from-hugging-face). Or, you can use other very popular vectorization APIs such as OpenAI or Cohere through the [`text2vec-openai`](/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) or [`text2vec-cohere`](/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-cohere) modules.»
[2] «You can even run transformer models locally with [`text2vec-transformers`](/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-transformers), and modules such as [`multi2vec-clip`](/developers/weaviate/modules/retriever-vectorizer-modules/multi2vec-clip) can convert images and text to vectors using a CLIP model. But they all perform the same core task—which is to represent the “meaning” of the original data as a set of numbers. And that’s why semantic search works so well. import WhatNext from '/_includes/what-next.mdx'
<WhatNext />»
[3] «Each `text2vec-*` module uses an external API (like `text2vec-openai` or `text2vec-huggingface`) or a local instance like `text2vec-transformers` to produce a vector for each object. Let's try vectorizing data with the `text2vec-cohere` module. We will be using data from `tiny_jeopardy.csv` [available here](https://github.com/weaviate/weaviate-examples/tree/main/text2vec-behind-curtain) containing questions from the game show Jeopardy. We'll just use a few (20) questions here, but the [full dataset on Kaggle](https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions) includes 200k+ questions. Load the data into a Pandas dataframe, then populate Weaviate like this:
```python
client.batch.configure(batch_size=100) # Configure batch
with client.batch as batch:
for i, row in df.iterrows():
properties = {
"question": row.Question,
"answer": row.Answer
}
batch.add_data_object(properties, "Question")
```
This should add a series of `Question` objects with text properties like this:
```text
{'question': 'In 1963, live on "The Art Linkletter Show", this company served its billionth burger',
'answer': "McDonald's"}
```
Since we use the `text2vec-cohere` module to vectorize our data, we can query Weaviate to find data objects most similar to any input text.»
Question: Can I train my own text2vec-contextionary vectorizer module?
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the information provided in the context to determine if it is possible to train a custom text2vec-contextionary vectorizer module.
Best Answer: No, it is not mentioned in the context that you can train your own text2vec-contextionary vectorizer module. The context only mentions the availability of different vectorizer models and service providers, such as Hugging Face, OpenAI, and Cohere, that can be used with Weaviate. It also mentions the option to bring your own vectors if you already have a vectorization pipeline available. However, there is no mention of training a custom text2vec-contextionary vectorizer module.
---
Context:
[1] «For example, if we broke down this blog post into **chapters** in Weaviate, with **title** and **content** properties. We could run a query to summarize the *"New distance metrics"* chapter like this:
```graphql
{
Get {
Chapter(
where: {
operator: Equal
path: "title"
valueText: "New distance metrics"
}
) {
title
_additional{
summary(
properties: ["content"],
) {
property
result
}
}
}
}
}
```
Which would return the following result:
```graphql
{
"data": {
"Get": {
"Chapters": [
{
"_additional": {
"summary": [
{
"property": "content",
"result": "Weaviate 1.15 adds two new distance metrics - Hamming
distance and Manhattan distance. In total, you can now choose
between five various distance metrics to support your datasets. Check out the metrics documentation page, for the full overview
of all the available metrics in Weaviate."
}
]
},
"title": "New distance metrics"
}
]
}
},
"errors": null
}
```
Head to the [Summarization Module docs page](/developers/weaviate/modules/reader-generator-modules/sum-transformers) to learn more. ### Hugging Face Module
The Hugging Face module (`text2vec-huggingface`) opens up doors to over 600 [Hugging Face sentence similarity models](https://huggingface.co/models?pipeline_tag=sentence-similarity), ready to be used in Weaviate as a vectorization module.»
[2] «Weaviate generates vector embeddings using [modules](/developers/weaviate/modules/retriever-vectorizer-modules) (OpenAI, Cohere, Google PaLM etc.), and conveniently stores both objects and vectors in the same database. For example, vectorizing the two words above might result in:
```text
cat = [1.5, -0.4, 7.2, 19.6, 3.1, ..., 20.2]
kitty = [1.5, -0.4, 7.2, 19.5, 3.2, ..., 20.8]
```
These two vectors have a very high similarity. In contrast, vectors for “banjo” or “comedy” would not be very similar to either of these vectors. To this extent, vectors capture the semantic similarity of words. Now that you’ve seen what vectors are, and that they can represent meaning to some extent, you might have further questions.»
[3] «And instead of the `text2vec-cohere` module, we will go straight to the Cohere API. Concatenating the text from the object:
```python
str_in = ' '.join([i for i in properties.values()])
```
We see:
```text
'In 1963, live on "The Art Linkletter Show", this company served its billionth burger McDonald\'s'
```
Then, use the Cohere API to generate the vector like so, where `cohere_key` is the API key (keep it secret!), and `model` is the vectorizer. ```python
import cohere
co = cohere.Client(cohere_key)
co_resp = co.embed([str_in], model="embed-multilingual-v2.0")
```
Then we run a `nearVector` based query to find the best matching object to this vector:
```python
client.query.get(
"Question",
["question", "answer"]
).with_limit(2).with_near_vector(
{'vector': co_resp.embeddings[0]}
).with_additional(['distance']).do()
```
Interestingly, we get a distance of `0.0181` - small, but not zero. In other words - we did something differently to Weaviate!
Let's go through the differences one by one, which hopefully will help you to see Weaviate's default behavior. First, Weaviate sorts properties alphabetically (a-z) before concatenation.»
Question: How do I get the cosine similarity from Weaviate's certainty?
Reasoning: Let's think step by step in order to produce the answer. We need to understand how Weaviate calculates certainty and how it relates to cosine similarity.
Best Answer: Weaviate does not directly provide the cosine similarity from its certainty value. The certainty value in Weaviate represents the confidence level of the result, but it is not equivalent to cosine similarity. Cosine similarity is a measure of the similarity between two vectors, while certainty in Weaviate is a measure of confidence in the result. To calculate cosine similarity, you would need to compare the vector embeddings of the objects using a specific algorithm or library.
---
Context:
[1] «Developers who want to build AI-powered applications can now skip the tedious process of complex training strategies. Now you can simply take models off-the-shelf and plug them into your apps. Applying a ranking model to hybrid search results is a promising approach to keep pushing the frontier of zero-shot AI. Imagine we want to retrieve information about the Weaviate Ref2Vec feature. If our application is using the Cohere embedding model, it has never seen this term or concept.»
[2] «These embeddings better reflect the polysemantic nature of words, which can only be disambiguated when they are considered in context. Some of the potential downsides include:
* increased compute requirements: fine-tuning transformer models is much slower (on the order of hours vs. minutes)
* increased memory requirements: context-sensitivity greatly increases memory requirements, which often leads to limited possible input lengths
Despite these downsides, transformer models have been wildly successful. Countless text vectorizer models have proliferated over the recent past. Plus, many more vectorizer models exist for other data types such as audio, video and images, to name a few.»
[3] «Further these approaches are well positioned to generalize to Recommendation. In Recommendation, instead of taking a [query, document] as input to a cross-encoder, we take as input a [user description, document] pair. For example, we can ask users to describe their preferences. Further, we could combine these in trios of [user description, query, item] for LLM, or more lightweight cross-encoder, ranking. There is a bonus 3rd idea where we use the log probabilities concatenating the query with the document.»
Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions)
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the given context to see if there is any mention of support for multiple versions of the query/document embedding models.
Best Answer: No specific information is provided in the given context about support for multiple versions of the query/document embedding models to co-exist at a given time.
---
Context:
[1] «Now, that's a lot of new models. 😉
#### How this works
The way the module works, Weaviate coordinates the efforts around data imports, data updates, queries, etc. and delegates requests to the Hugging Face Inference API. You need a `Hugging Face API Token` to use the Hugging Face module. You can [request it here](https://huggingface.co/login?next=%2Fsettings%2Ftokens).»
[2] «What I love about all this work, like self-ask, chain of thought, we're developing new querying languages. This is like us inventing SQL, except that we didn't design the database. The database came into being and we have to figure out how to interact with it. That example I mentioned about the IPython interaction, like that's a, again, it's a new querying language. And I honestly thought the most potent part of the self-ask wasn't even necessarily the self-ask part.»
[3] «Developers who want to build AI-powered applications can now skip the tedious process of complex training strategies. Now you can simply take models off-the-shelf and plug them into your apps. Applying a ranking model to hybrid search results is a promising approach to keep pushing the frontier of zero-shot AI. Imagine we want to retrieve information about the Weaviate Ref2Vec feature. If our application is using the Cohere embedding model, it has never seen this term or concept.»
Question: Can I connect my own module?
Reasoning: Let's think step by step in order to produce the answer. We need to understand if it is possible to connect a custom module to Weaviate.
Best Answer: Yes, it is possible to connect your own module to Weaviate. Weaviate coordinates the efforts around data imports, data updates, queries, etc. and delegates requests to the Hugging Face Inference API. This means that you can use your own module by integrating it with Weaviate.
---
Context:
[1] «There is no performance gain if we have more parallelization than there are available CPUs. However, each thread needs additional memory. So with 32 parallel imports, we had the worst of both worlds: High memory usage and no performance gains beyond 8. With the fix, even if you import from multiple clients, Weaviate automatically handles the parallelization to ensure that it does not exceed the number of CPU cores. As a result, you get the maximum performance without "unnecessary" memory usage. ### HNSW optimization
Next, we optimized memory allocations for the HNSW (vector) index.»
[2] «The past for vector searching definitely was not a “simpler time”, and the appeal of modern vector databases like Weaviate is pretty clear given this context. But while the future is here, it isn't yet perfect. Tools like Weaviate can seem like a magician's mystery box. Our users in turn ask us *exactly* how Weaviate does its magic; how it turns all of that data into vectors, and how to control the process. So let's take a look inside the magic box together in this post.»
[3] «:::
## Conclusions
We've managed to implement the indexing algorithm on DiskANN, and the resulting performance is good. From years of research & development, Weaviate has a highly optimized implementation of the HNSW algorithm. With the Vamana implementation, we achieved comparable in-memory results. There are still some challenges to overcome and questions to answer. For example:
* How do we proceed to the natural disk solution of Weaviate?»
Question: Does Weaviate use Hnswlib?
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the given context to determine if Weaviate uses Hnswlib.
Best Answer: Yes, Weaviate uses Hnswlib. The context mentions that Weaviate has a highly optimized implementation of the HNSW (Hierarchical Navigable Small World) algorithm, which is the algorithm used by Hnswlib. Additionally, the context mentions the implementation of the indexing algorithm on DiskANN, which is a disk-based variant of HNSW.
0%| | 0/10 [00:00<?, ?it/s]
Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer
Average Metric: 3.8 / 1 (380.0): 10%|█ | 1/10 [00:21<03:09, 21.09s/it]
Faithful: 5 Detail: 4 Overall: 5 Test Question: How can I retrieve the total object count in a class? Predicted Answer: predicted_answer
Average Metric: 7.199999999999999 / 2 (360.0): 20%|▏| 2/10 [00:21<01:11, 8.96
Faithful: 5 Detail: 2 Overall: 5 Test Question: How do I get the cosine similarity from Weaviate's certainty? Predicted Answer: predicted_answer
Average Metric: 8.799999999999999 / 3 (293.3): 30%|▎| 3/10 [00:22<00:35, 5.10
Faithful: 1 Detail: 1 Overall: 5
Average Metric: 11.399999999999999 / 4 (285.0): 40%|▍| 4/10 [00:22<00:19, 3.2
Test Question: The quality of my search results change depending on the specified limit. Why? How can I fix this? Predicted Answer: predicted_answer Faithful: 2 Detail: 5 Overall: 4
Average Metric: 13.399999999999999 / 5 (268.0): 50%|▌| 5/10 [00:22<00:11, 2.2
Test Question: Why did you use GraphQL instead of SPARQL? Predicted Answer: predicted_answer Faithful: 1 Detail: 3 Overall: 5 Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
Average Metric: 14.999999999999998 / 6 (250.0): 60%|▌| 6/10 [00:23<00:06, 1.6
Faithful: 1 Detail: 1 Overall: 5 Test Question: What is best practice for updating data? Predicted Answer: predicted_answer
Average Metric: 16.599999999999998 / 7 (237.1): 70%|▋| 7/10 [00:45<00:25, 8.4
Faithful: 1 Detail: 1 Overall: 5 Test Question: Can I connect my own module? Predicted Answer: predicted_answer
Average Metric: 20.2 / 8 (252.5): 80%|███████▏ | 8/10 [01:08<00:25, 12.92s/it]
Faithful: 5 Detail: 5 Overall: 3 Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
Average Metric: 24.0 / 9 (266.7): 90%|████████ | 9/10 [01:33<00:16, 16.71s/it]
Faithful: 5 Detail: 4 Overall: 5 Test Question: Does Weaviate use Hnswlib? Predicted Answer: predicted_answer
Average Metric: 27.8 / 10 (278.0): 100%|███████| 10/10 [01:53<00:00, 11.34s/it]
[I 2024-02-11 14:51:13,743] Trial 1 finished with value: 278.0 and parameters: {'10883682544_predictor_instruction': 3, '10883682544_predictor_demos': 2}. Best is trial 0 with value: 290.0.
Faithful: 5
Detail: 4
Overall: 5
Average Metric: 27.8 / 10 (278.0%)
You are presented with several contexts or data scenarios, typically related to software usage or tech process optimization. Based on this, you will be asked questions to clarify or retrieve certain details. Your task is to provide an accurate and specific response, which will assist in a tech-oriented customer service role, such as creating FAQs, supporting a chatbot, or enhancing Natural Language Processing to understand user queries more efficiently.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Relevant Information: ${answer}
---
Context:
[1] «Finally, we can jump on a bike to reach our local destination. For a better understanding, consider the below graphic, which shows a graph with all the connections generated using 1000 objects in two dimensions. <img
src={require('./img/vamana-graph.png').default}
alt="Vamana graph with 1000 objects"
style={{ maxWidth: "50%" }}
/>
If we iterate over it in steps – we can analyze how Vamana navigates through the graph. <img
src={require('./img/vamana-graph-animated.gif').default}
alt="Vamana graph - animated in 3/6/9 steps"
style={{ maxWidth: "50%" }}
/>
In the **first step**, you can see that the entry point for the search is in the center, and then the long-range connections allow jumping to the edges. This means that when a query comes, it will quickly move in the appropriate direction.<br/>
The **second**, **third**, and **final steps** highlight the nodes reachable within **three**, **six**, and **nine** hops from the entry node.»
[2] «It was that Ofir did such a phenomenal job of figuring out a way to measure the complexity of the knowledge that was extracted from the model. He gave us a benchmark, a ladder to climb, a way to measure whether we could retrieve certain kinds of information from models. And I think that's going to open the door to a ton more benchmarks. And you know what happens when there's a benchmark. We optimize the hell out of that benchmark and it moves science forward… [ truncated for visibility ] |
| Hybrid Only | Or, at least being able to ask follow up questions when it’s unclear about and that’s surprisingly not that difficult to do with these current systems, as long as you’re halfway decent at prompting, you can build up these follow up systems and train them over the course of a couple 1,000 examples to perform really, really well, at least to cove r90, 95% of questions that you might get.»
[3] «The data is persisted, so you can use it from future invocations, or you can [transfer it to another instance](/developers/weaviate/manage-data/read-all-objects/#restore-to-a-target-instance). You can learn more about running Weaviate locally from client code on the [Embedded Weaviate](/developers/weaviate/installation/embedded/) page. ## <i class="fa-solid fa-lightbulb"></i> Use cases
What can you do with Embedded Weaviate? Quite a few things!
First off, you can get started very quickly with Weaviate on your local machine, without having to explicitly download, install or instantiate a server. ### Jupyter notebooks
You can also use Embedded Weaviate from Jupyter notebooks, including on Google Colaboratory.»
Question: What is the best way to iterate through objects? Can I do paginated API calls?
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the given context to find information about iterating through objects and paginated API calls.
Relevant Information: The context does not provide any information about the best way to iterate through objects or paginated API calls.
---
Context:
[1] «There is no performance gain if we have more parallelization than there are available CPUs. However, each thread needs additional memory. So with 32 parallel imports, we had the worst of both worlds: High memory usage and no performance gains beyond 8. With the fix, even if you import from multiple clients, Weaviate automatically handles the parallelization to ensure that it does not exceed the number of CPU cores. As a result, you get the maximum performance without "unnecessary" memory usage. ### HNSW optimization
Next, we optimized memory allocations for the HNSW (vector) index.»
[2] «The past for vector searching definitely was not a “simpler time”, and the appeal of modern vector databases like Weaviate is pretty clear given this context. But while the future is here, it isn't yet perfect. Tools like Weaviate can seem like a magician's mystery box. Our users in turn ask us *exactly* how Weaviate does its magic; how it turns all of that data into vectors, and how to control the process. So let's take a look inside the magic box together in this post.»
[3] «:::
## Conclusions
We've managed to implement the indexing algorithm on DiskANN, and the resulting performance is good. From years of research & development, Weaviate has a highly optimized implementation of the HNSW algorithm. With the Vamana implementation, we achieved comparable in-memory results. There are still some challenges to overcome and questions to answer. For example:
* How do we proceed to the natural disk solution of Weaviate?»
Question: Does Weaviate use Hnswlib?
Reasoning: Let's think step by step in order to produce the answer. We can look for any mention of Hnswlib in the given context.
Relevant Information: No information is provided in the given context about whether Weaviate uses Hnswlib or not.
---
Context:
[1] «|
By re-ranking the results we are able to get the clip where Jonathan Frankle describes the benchmarks created by Ofir Press et al. in the self-ask paper! This result was originally placed at #6 with Hybrid Search only. This is a great opportunity to preview the discussion of how LLMs use search versus humans. When humans search, we are used to scrolling through the results a bit to see the one that makes sense. In contrast, language models are constrained by input length; we can only give so many results to the input of the LLM.»
[2] «To end this article, let’s discuss a little further why ranking is so exciting for the most hyped pairing of LLMs and Search: Retrieval-Augmented Generation. ## Ranking for Retrieval-Augmented Generation
A lot of the recent successes of vector search can be attributed to their effectiveness as a tool for Large Language Models. So whereas the speed trade-off with rankers may be a major bottleneck for how humans use search, it might not be as much of a problem for how LLMs use search. Of course fast generation is preferred, but if you are paying for the result, quality may be more important than speed. Shi et al.»
[3] «have published “Large Language Models are easily distracted by irrelevant context”, highlighting how problematic bad precision in search can be for retrieval-augmented generation. The recent developments in LLM agent tooling such as LangChain, LlamaIndex, and recent projects such as AutoGPT or Microsoft’s Semantic Kernel are paving the way towards letting LLMs run for a while to complete complex tasks. By ranking each handoff from search to prompt, we can achieve better results in each intermediate task. Thus when we leave an LLM running overnight to research the future of ranking models, we can expect a better final result in the morning!
import WhatNext from '/_includes/what-next.mdx'
<WhatNext />»
Question: The quality of my search results change depending on the specified limit. Why? How can I fix this?
Reasoning: Let's think step by step in order to produce the answer. We can refer to the given context to understand why the quality of search results changes depending on the specified limit. The context mentions that language models are constrained by input length, meaning they can only provide a limited number of results based on the input. This limitation is not present when humans search, as they can scroll through multiple results. To fix this, we can adjust the specified limit to allow for more results to be generated by the language model.
Relevant Information: The quality of search results changes depending on the specified limit because language models are constrained by input length and can only provide a limited number of results. To fix this, we can adjust the specified limit to allow for more results to be generated by the language model.
---
Context:
[1] «HNSW, on the other hand, implements the same idea a bit differently. Instead of having all information together on a flat graph, it has a hierarchical representation distributed across multiple layers. The top layers only contain long-range connections, and as you dive deeper into the layers, your query is routed to the appropriate region where you can look more locally for your answer. So your search starts making only big jumps across the top layers until it finally looks for the closest points locally in the bottom layers. ## Performance comparison
So, how do they perform?»
[2] «If we make one final trip to the supermarket to understand what a guava is, we could look at the things around it—other fruit. Somewhat further away we might find guava juice or tinned guavas, but there's really no reason to look four aisles over for guava-flavored cat food. Using ANN allowed high-dimensional searches with near-perfect accuracy to be returned in milliseconds instead of hours. To be practical, vector databases also needed something prosaically called CRUD support. That stands for "create, read, update and delete," and solving that technical challenge meant that the complex process of indexing the database could be done once, rather than being repeated from scratch whenever the database was updated.»
[3] «It was that Ofir did such a phenomenal job of figuring out a way to measure the complexity of the knowledge that was extracted from the model. He gave us a benchmark, a ladder to climb, a way to measure whether we could retrieve certain kinds of information from models. And I think that's going to open the door to a ton more benchmarks. And you know what happens when there's a benchmark. We optimize the hell out of that benchmark and it moves science forward… [ truncated for visibility ] |
| Hybrid Only | Or, at least being able to ask follow up questions when it’s unclear about and that’s surprisingly not that difficult to do with these current systems, as long as you’re halfway decent at prompting, you can build up these follow up systems and train them over the course of a couple 1,000 examples to perform really, really well, at least to cove r90, 95% of questions that you might get.»
Question: Why did you use GraphQL instead of SPARQL?
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the given context to find any information related to the use of GraphQL instead of SPARQL.
Relevant Information: The given context does not provide any information about the use of GraphQL instead of SPARQL. Therefore, we cannot determine the reason for using GraphQL over SPARQL based on the given context.
---
Context:
[1] «There is no performance gain if we have more parallelization than there are available CPUs. However, each thread needs additional memory. So with 32 parallel imports, we had the worst of both worlds: High memory usage and no performance gains beyond 8. With the fix, even if you import from multiple clients, Weaviate automatically handles the parallelization to ensure that it does not exceed the number of CPU cores. As a result, you get the maximum performance without "unnecessary" memory usage. ### HNSW optimization
Next, we optimized memory allocations for the HNSW (vector) index.»
[2] «The past for vector searching definitely was not a “simpler time”, and the appeal of modern vector databases like Weaviate is pretty clear given this context. But while the future is here, it isn't yet perfect. Tools like Weaviate can seem like a magician's mystery box. Our users in turn ask us *exactly* how Weaviate does its magic; how it turns all of that data into vectors, and how to control the process. So let's take a look inside the magic box together in this post.»
[3] «:::
## Conclusions
We've managed to implement the indexing algorithm on DiskANN, and the resulting performance is good. From years of research & development, Weaviate has a highly optimized implementation of the HNSW algorithm. With the Vamana implementation, we achieved comparable in-memory results. There are still some challenges to overcome and questions to answer. For example:
* How do we proceed to the natural disk solution of Weaviate?»
Question: Does Weaviate use Hnswlib?
Reasoning: Let's think step by step in order to produce the answer. We can look for any mention of Hnswlib in the given context.
Relevant Information: The given context mentions that Weaviate has a highly optimized implementation of the HNSW algorithm, but it does not specify whether Weaviate uses Hnswlib or not. Therefore, we cannot determine if Weaviate uses Hnswlib based on the given context.
0%| | 0/10 [00:00<?, ?it/s]
Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer
Average Metric: 2.0 / 1 (200.0): 10%|█ | 1/10 [00:00<00:04, 1.99it/s]
Faithful: 1 Detail: 3 Overall: 5
Average Metric: 3.4 / 2 (170.0): 20%|██ | 2/10 [00:01<00:04, 1.79it/s]
Test Question: How can I retrieve the total object count in a class? Predicted Answer: predicted_answer Faithful: 1 Detail: 4 Overall: 1 Test Question: How do I get the cosine similarity from Weaviate's certainty? Predicted Answer: predicted_answer
Average Metric: 5.0 / 3 (166.7): 30%|███ | 3/10 [00:01<00:03, 1.94it/s]
Faithful: 1 Detail: 5 Overall: 1
Average Metric: 7.6 / 4 (190.0): 40%|████ | 4/10 [00:02<00:03, 1.98it/s]
Test Question: The quality of my search results change depending on the specified limit. Why? How can I fix this? Predicted Answer: predicted_answer Faithful: 2 Detail: 5 Overall: 4 Test Question: Why did you use GraphQL instead of SPARQL? Predicted Answer: predicted_answer
Average Metric: 9.6 / 5 (192.0): 50%|█████ | 5/10 [00:02<00:02, 1.96it/s]
Faithful: 1 Detail: 3 Overall: 5 Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
Average Metric: 11.2 / 6 (186.7): 60%|█████▍ | 6/10 [00:03<00:02, 1.98it/s]
Faithful: 1 Detail: 1 Overall: 5
Average Metric: 14.6 / 7 (208.6): 70%|██████▎ | 7/10 [00:03<00:01, 2.14it/s]
Test Question: What is best practice for updating data? Predicted Answer: predicted_answer Faithful: 5 Detail: 3 Overall: 4
Average Metric: 18.6 / 8 (232.5): 80%|███████▏ | 8/10 [00:03<00:00, 2.33it/s]
Test Question: Can I connect my own module? Predicted Answer: predicted_answer Faithful: 5 Detail: 5 Overall: 5 Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
Average Metric: 22.6 / 9 (251.1): 90%|████████ | 9/10 [00:04<00:00, 2.35it/s]
Faithful: 5 Detail: 5 Overall: 5
Average Metric: 24.400000000000002 / 10 (244.0): 100%|█| 10/10 [00:04<00:00, 2
[I 2024-02-11 14:51:18,463] Trial 2 finished with value: 244.0 and parameters: {'10883682544_predictor_instruction': 0, '10883682544_predictor_demos': 0}. Best is trial 0 with value: 290.0.
Test Question: Does Weaviate use Hnswlib?
Predicted Answer: predicted_answer
Faithful: 1
Detail: 2
Overall: 5
Average Metric: 24.400000000000002 / 10 (244.0%)
Answer questions based on the context.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ${answer}
---
Context:
[1] «There is no performance gain if we have more parallelization than there are available CPUs. However, each thread needs additional memory. So with 32 parallel imports, we had the worst of both worlds: High memory usage and no performance gains beyond 8. With the fix, even if you import from multiple clients, Weaviate automatically handles the parallelization to ensure that it does not exceed the number of CPU cores. As a result, you get the maximum performance without "unnecessary" memory usage. ### HNSW optimization
Next, we optimized memory allocations for the HNSW (vector) index.»
[2] «The past for vector searching definitely was not a “simpler time”, and the appeal of modern vector databases like Weaviate is pretty clear given this context. But while the future is here, it isn't yet perfect. Tools like Weaviate can seem like a magician's mystery box. Our users in turn ask us *exactly* how Weaviate does its magic; how it turns all of that data into vectors, and how to control the process. So let's take a look inside the magic box together in this post.»
[3] «:::
## Conclusions
We've managed to implement the indexing algorithm on DiskANN, and the resulting performance is good. From years of research & development, Weaviate has a highly optimized implementation of the HNSW algorithm. With the Vamana implementation, we achieved comparable in-memory results. There are still some challenges to overcome and questions to answer. For example:
* How do we proceed to the natural disk solution of Weaviate?»
Question: Does Weaviate use Hnswlib?
Reasoning: Let's think step by step in order to produce the answer. We can look for any mention of Hnswlib in the given context.
Answer: No information is provided in the given context about whether Weaviate uses Hnswlib or not.
[63]
Prediction( , answer='Cross encoders are ranking models used for content-based re-ranking. They are designed to determine the relevance of a document to a query. Cross encoders take a [query, document] input and output a score indicating the relevance of the document to the query. They can be used to re-rank search results and provide a more personalized and context-aware search experience.' ,)
Check this out!!
Below you can see how the BayesianSignatureOptimizer jointly (1) optimizes the task instruction to:
Assess the context and answer the given questions that are predominantly about software usage, process optimization, and troubleshooting. Focus on providing accurate information related to tech or software-related queries.
As well as sourcing input-output examples for the prompt!
[64]
Assess the context and answer the given questions that are predominantly about software usage, process optimization, and troubleshooting. Focus on providing accurate information related to tech or software-related queries.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Best Answer: ${answer}
---
Context:
[1] «Some models, such as [CLIP](https://openai.com/blog/clip/), are capable of vectorizing multiple data types (images and text in this case) into one vector space, so that an image can be searched by its content using only text. ## Vector embeddings with Weaviate
For this reason, Weaviate is configured to support many different vectorizer models and vectorizer service providers. You can even [bring your own vectors](/developers/weaviate/starter-guides/custom-vectors), for example if you already have a vectorization pipeline available, or if none of the publicly available models are suitable for you. For one, Weaviate supports using any Hugging Face models through our `text2vec-hugginface` module, so that you can [choose one of the many sentence transformers published on Hugging Face](/blog/how-to-choose-a-sentence-transformer-from-hugging-face). Or, you can use other very popular vectorization APIs such as OpenAI or Cohere through the [`text2vec-openai`](/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) or [`text2vec-cohere`](/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-cohere) modules.»
[2] «You can even run transformer models locally with [`text2vec-transformers`](/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-transformers), and modules such as [`multi2vec-clip`](/developers/weaviate/modules/retriever-vectorizer-modules/multi2vec-clip) can convert images and text to vectors using a CLIP model. But they all perform the same core task—which is to represent the “meaning” of the original data as a set of numbers. And that’s why semantic search works so well. import WhatNext from '/_includes/what-next.mdx'
<WhatNext />»
[3] «Each `text2vec-*` module uses an external API (like `text2vec-openai` or `text2vec-huggingface`) or a local instance like `text2vec-transformers` to produce a vector for each object. Let's try vectorizing data with the `text2vec-cohere` module. We will be using data from `tiny_jeopardy.csv` [available here](https://github.com/weaviate/weaviate-examples/tree/main/text2vec-behind-curtain) containing questions from the game show Jeopardy. We'll just use a few (20) questions here, but the [full dataset on Kaggle](https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions) includes 200k+ questions. Load the data into a Pandas dataframe, then populate Weaviate like this:
```python
client.batch.configure(batch_size=100) # Configure batch
with client.batch as batch:
for i, row in df.iterrows():
properties = {
"question": row.Question,
"answer": row.Answer
}
batch.add_data_object(properties, "Question")
```
This should add a series of `Question` objects with text properties like this:
```text
{'question': 'In 1963, live on "The Art Linkletter Show", this company served its billionth burger',
'answer': "McDonald's"}
```
Since we use the `text2vec-cohere` module to vectorize our data, we can query Weaviate to find data objects most similar to any input text.»
Question: Can I train my own text2vec-contextionary vectorizer module?
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the information provided in the context to determine if it is possible to train a custom text2vec-contextionary vectorizer module.
Best Answer: No, it is not mentioned in the context that you can train your own text2vec-contextionary vectorizer module. The context only mentions the availability of different vectorizer models and service providers, such as Hugging Face, OpenAI, and Cohere, that can be used with Weaviate. It also mentions the option to bring your own vectors if you already have a vectorization pipeline available. However, there is no mention of training a custom text2vec-contextionary vectorizer module.
---
Context:
[1] «For example, if we broke down this blog post into **chapters** in Weaviate, with **title** and **content** properties. We could run a query to summarize the *"New distance metrics"* chapter like this:
```graphql
{
Get {
Chapter(
where: {
operator: Equal
path: "title"
valueText: "New distance metrics"
}
) {
title
_additional{
summary(
properties: ["content"],
) {
property
result
}
}
}
}
}
```
Which would return the following result:
```graphql
{
"data": {
"Get": {
"Chapters": [
{
"_additional": {
"summary": [
{
"property": "content",
"result": "Weaviate 1.15 adds two new distance metrics - Hamming
distance and Manhattan distance. In total, you can now choose
between five various distance metrics to support your datasets. Check out the metrics documentation page, for the full overview
of all the available metrics in Weaviate."
}
]
},
"title": "New distance metrics"
}
]
}
},
"errors": null
}
```
Head to the [Summarization Module docs page](/developers/weaviate/modules/reader-generator-modules/sum-transformers) to learn more. ### Hugging Face Module
The Hugging Face module (`text2vec-huggingface`) opens up doors to over 600 [Hugging Face sentence similarity models](https://huggingface.co/models?pipeline_tag=sentence-similarity), ready to be used in Weaviate as a vectorization module.»
[2] «Weaviate generates vector embeddings using [modules](/developers/weaviate/modules/retriever-vectorizer-modules) (OpenAI, Cohere, Google PaLM etc.), and conveniently stores both objects and vectors in the same database. For example, vectorizing the two words above might result in:
```text
cat = [1.5, -0.4, 7.2, 19.6, 3.1, ..., 20.2]
kitty = [1.5, -0.4, 7.2, 19.5, 3.2, ..., 20.8]
```
These two vectors have a very high similarity. In contrast, vectors for “banjo” or “comedy” would not be very similar to either of these vectors. To this extent, vectors capture the semantic similarity of words. Now that you’ve seen what vectors are, and that they can represent meaning to some extent, you might have further questions.»
[3] «And instead of the `text2vec-cohere` module, we will go straight to the Cohere API. Concatenating the text from the object:
```python
str_in = ' '.join([i for i in properties.values()])
```
We see:
```text
'In 1963, live on "The Art Linkletter Show", this company served its billionth burger McDonald\'s'
```
Then, use the Cohere API to generate the vector like so, where `cohere_key` is the API key (keep it secret!), and `model` is the vectorizer. ```python
import cohere
co = cohere.Client(cohere_key)
co_resp = co.embed([str_in], model="embed-multilingual-v2.0")
```
Then we run a `nearVector` based query to find the best matching object to this vector:
```python
client.query.get(
"Question",
["question", "answer"]
).with_limit(2).with_near_vector(
{'vector': co_resp.embeddings[0]}
).with_additional(['distance']).do()
```
Interestingly, we get a distance of `0.0181` - small, but not zero. In other words - we did something differently to Weaviate!
Let's go through the differences one by one, which hopefully will help you to see Weaviate's default behavior. First, Weaviate sorts properties alphabetically (a-z) before concatenation.»
Question: How do I get the cosine similarity from Weaviate's certainty?
Reasoning: Let's think step by step in order to produce the answer. We need to understand how Weaviate calculates certainty and how it relates to cosine similarity.
Best Answer: Weaviate does not directly provide the cosine similarity from its certainty value. The certainty value in Weaviate represents the confidence level of the result, but it is not equivalent to cosine similarity. Cosine similarity is a measure of the similarity between two vectors, while certainty in Weaviate is a measure of confidence in the result. To calculate cosine similarity, you would need to compare the vector embeddings of the objects using a specific algorithm or library.
---
Context:
[1] «Developers who want to build AI-powered applications can now skip the tedious process of complex training strategies. Now you can simply take models off-the-shelf and plug them into your apps. Applying a ranking model to hybrid search results is a promising approach to keep pushing the frontier of zero-shot AI. Imagine we want to retrieve information about the Weaviate Ref2Vec feature. If our application is using the Cohere embedding model, it has never seen this term or concept.»
[2] «These embeddings better reflect the polysemantic nature of words, which can only be disambiguated when they are considered in context. Some of the potential downsides include:
* increased compute requirements: fine-tuning transformer models is much slower (on the order of hours vs. minutes)
* increased memory requirements: context-sensitivity greatly increases memory requirements, which often leads to limited possible input lengths
Despite these downsides, transformer models have been wildly successful. Countless text vectorizer models have proliferated over the recent past. Plus, many more vectorizer models exist for other data types such as audio, video and images, to name a few.»
[3] «Further these approaches are well positioned to generalize to Recommendation. In Recommendation, instead of taking a [query, document] as input to a cross-encoder, we take as input a [user description, document] pair. For example, we can ask users to describe their preferences. Further, we could combine these in trios of [user description, query, item] for LLM, or more lightweight cross-encoder, ranking. There is a bonus 3rd idea where we use the log probabilities concatenating the query with the document.»
Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions)
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the given context to see if there is any mention of support for multiple versions of the query/document embedding models.
Best Answer: No specific information is provided in the given context about support for multiple versions of the query/document embedding models to co-exist at a given time.
---
Context:
[1] «Now, that's a lot of new models. 😉
#### How this works
The way the module works, Weaviate coordinates the efforts around data imports, data updates, queries, etc. and delegates requests to the Hugging Face Inference API. You need a `Hugging Face API Token` to use the Hugging Face module. You can [request it here](https://huggingface.co/login?next=%2Fsettings%2Ftokens).»
[2] «What I love about all this work, like self-ask, chain of thought, we're developing new querying languages. This is like us inventing SQL, except that we didn't design the database. The database came into being and we have to figure out how to interact with it. That example I mentioned about the IPython interaction, like that's a, again, it's a new querying language. And I honestly thought the most potent part of the self-ask wasn't even necessarily the self-ask part.»
[3] «Developers who want to build AI-powered applications can now skip the tedious process of complex training strategies. Now you can simply take models off-the-shelf and plug them into your apps. Applying a ranking model to hybrid search results is a promising approach to keep pushing the frontier of zero-shot AI. Imagine we want to retrieve information about the Weaviate Ref2Vec feature. If our application is using the Cohere embedding model, it has never seen this term or concept.»
Question: Can I connect my own module?
Reasoning: Let's think step by step in order to produce the answer. We need to understand if it is possible to connect a custom module to Weaviate.
Best Answer: Yes, it is possible to connect your own module to Weaviate. Weaviate coordinates the efforts around data imports, data updates, queries, etc. and delegates requests to the Hugging Face Inference API. This means that you can use your own module by integrating it with Weaviate.
---
Context:
[1] «[Cross Encoders](#cross-encoders) (collapsing the use of Large Language Models for ranking into this category as well)
1. [Metadata Rankers](#metadata-rankers)
1. [Score Rankers](#score-rankers)
## Cross Encoders
Cross Encoders are one of the most well known ranking models for content-based re-ranking. There is quite a collection of pre-trained cross encoders available on [sentence transformers](https://www.sbert.net/docs/pretrained_cross-encoders.html). We are currently envisioning interfacing cross encoders with Weaviate using the following syntax.»
[2] «Thus, quality at the expense of speed, becomes more interesting. ### LLMs as Cross Encoders
So, let’s dive into the LLM hype a little more, how can we use LLMs for re-ranking? There are generally 2 ways to do this. The first strategy is identical to the cross encoder, we give the LLM the [query, document] input and prompt it to output a score of how relevant the document is to the query. The tricky thing with this is bounding the score.»
[3] «They offer the advantage of further reasoning about the relevance of results without needing specialized training. Cross Encoders can be interfaced with Weaviate to re-rank search results, trading off performance for slower search speed. * **Metadata Rankers** are context-based re-rankers that use symbolic features to rank relevance. They take into account user and document features, such as age, gender, location, preferences, release year, genre, and box office, to predict the relevance of candidate documents. By incorporating metadata features, these rankers offer a more personalized and context-aware search experience.»
Question: What do cross encoders do?
Reasoning: Let's think step by step in order to produce the answer. We need to analyze the given context to understand what cross encoders do.
Best Answer: Cross encoders are ranking models used for content-based re-ranking. They are designed to determine the relevance of a document to a query. Cross encoders take a [query, document] input and output a score indicating the relevance of the document to the query. They can be used to re-rank search results and provide a more personalized and context-aware search experience.
[65]
0%| | 0/10 [00:00<?, ?it/s]
Test Question: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions) Predicted Answer: predicted_answer
Average Metric: 2.0 / 1 (200.0): 10%|█ | 1/10 [00:00<00:02, 3.09it/s]
Faithful: 1 Detail: 3 Overall: 5 Test Question: How can I retrieve the total object count in a class? Predicted Answer: predicted_answer
Average Metric: 3.8 / 2 (190.0): 20%|██ | 2/10 [00:00<00:02, 2.90it/s]
Faithful: 1 Detail: 5 Overall: 2 Test Question: How do I get the cosine similarity from Weaviate's certainty? Predicted Answer: predicted_answer
Average Metric: 5.4 / 3 (180.0): 30%|███ | 3/10 [00:01<00:03, 2.24it/s]
Faithful: 1 Detail: 5 Overall: 1 Test Question: The quality of my search results change depending on the specified limit. Why? How can I fix this? Predicted Answer: predicted_answer
Average Metric: 7.4 / 4 (185.0): 40%|████ | 4/10 [00:01<00:02, 2.26it/s]
Faithful: 1 Detail: 5 Overall: 3 Test Question: Why did you use GraphQL instead of SPARQL? Predicted Answer: predicted_answer
Average Metric: 11.4 / 5 (228.0): 50%|████▌ | 5/10 [00:02<00:02, 1.90it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: What is the best way to iterate through objects? Can I do paginated API calls? Predicted Answer: predicted_answer
Average Metric: 13.0 / 6 (216.7): 60%|█████▍ | 6/10 [00:02<00:02, 1.86it/s]
Faithful: 1 Detail: 4 Overall: 2 Test Question: What is best practice for updating data? Predicted Answer: predicted_answer
Average Metric: 17.0 / 7 (242.9): 70%|██████▎ | 7/10 [00:03<00:01, 1.80it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Can I connect my own module? Predicted Answer: predicted_answer
Average Metric: 21.0 / 8 (262.5): 80%|███████▏ | 8/10 [00:04<00:01, 1.81it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Can I train my own text2vec-contextionary vectorizer module? Predicted Answer: predicted_answer
Average Metric: 25.0 / 9 (277.8): 90%|████████ | 9/10 [00:04<00:00, 2.02it/s]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Does Weaviate use Hnswlib? Predicted Answer: predicted_answer
Average Metric: 29.0 / 10 (290.0): 100%|███████| 10/10 [00:04<00:00, 2.05it/s]
Faithful: 5 Detail: 5 Overall: 5 Average Metric: 29.0 / 10 (290.0%)
290.0
Test Set Eval
[66]
[67]
0%| | 0/14 [00:00<?, ?it/s]
Test Question: Are all ANN algorithms potential candidates to become an indexation plugin in Weaviate? Predicted Answer: predicted_answer
Average Metric: 3.2 / 1 (320.0): 7%|▋ | 1/14 [00:28<06:09, 28.41s/it]
Faithful: 3 Detail: 5 Overall: 5 Test Question: Does Weaviate use pre- or post-filtering ANN index search? Predicted Answer: predicted_answer
Average Metric: 7.2 / 2 (360.0): 14%|█▍ | 2/14 [00:55<05:31, 27.64s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: How does Weaviate's vector and scalar filtering work? Predicted Answer: predicted_answer
Average Metric: 7.2 / 3 (240.0): 21%|██▏ | 3/14 [01:17<04:37, 25.22s/it]
Faithful: No Detail: 1 Overall: 5 Error for example in dev set: could not convert string to float: 'No' Test Question: What would you say is more important for query speed in Weaviate: More CPU power, or more RAM? Predicted Answer: predicted_answer
Average Metric: 8.4 / 4 (210.0): 29%|██▊ | 4/14 [01:39<03:59, 23.94s/it]
Faithful: 1 Detail: 2 Overall: 2 Test Question: Data import takes long / is slow, what is causing this and what can I do? Predicted Answer: predicted_answer
Average Metric: 12.2 / 5 (244.0): 36%|███▏ | 5/14 [02:07<03:47, 25.33s/it]
Faithful: 5 Detail: 5 Overall: 4 Test Question: How can slow queries be optimized? Predicted Answer: predicted_answer
Average Metric: 16.2 / 6 (270.0): 43%|███▊ | 6/14 [02:32<03:21, 25.15s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: When scalar and vector search are combined, will the scalar filter happen before or after the nearest neighbor (vector) search? Predicted Answer: predicted_answer
Average Metric: 17.599999999999998 / 7 (251.4): 50%|▌| 7/14 [02:51<02:42, 23.2
Faithful: 2 Detail: 2 Overall: 1 Test Question: Regarding "filtered vector search": Since this is a two-phase pipeline, how big can that list of IDs get? Do you know how that size might affect query performance? Predicted Answer: predicted_answer
Average Metric: 19.799999999999997 / 8 (247.5): 57%|▌| 8/14 [03:14<02:18, 23.1
Faithful: 1 Detail: 4 Overall: 5 Test Question: My Weaviate setup is using more memory than what I think is reasonable. How can I debug this? Predicted Answer: predicted_answer
Average Metric: 23.799999999999997 / 9 (264.4): 64%|▋| 9/14 [03:39<01:58, 23.6
Faithful: 5 Detail: 5 Overall: 5 Test Question: How can I print a stack trace of Weaviate? Predicted Answer: predicted_answer
Average Metric: 26.999999999999996 / 10 (270.0): 71%|▋| 10/14 [04:08<01:40, 25
Faithful: 5 Detail: 1 Overall: 5 Test Question: Can I request a feature in Weaviate? Predicted Answer: predicted_answer
Average Metric: 30.399999999999995 / 11 (276.4): 79%|▊| 11/14 [04:28<01:10, 23
Faithful: 5 Detail: 3 Overall: 4 Test Question: What is Weaviate's consistency model in a distributed setup? Predicted Answer: predicted_answer
Average Metric: 33.99999999999999 / 12 (283.3): 86%|▊| 12/14 [04:50<00:46, 23.
Faithful: 5 Detail: 3 Overall: 5 Test Question: With your aggregations I could not see how to do time buckets, is this possible? Predicted Answer: predicted_answer
Average Metric: 37.99999999999999 / 13 (292.3): 93%|▉| 13/14 [05:10<00:22, 22.
Faithful: 5 Detail: 5 Overall: 5 Test Question: How can I run the latest master branch with Docker Compose? Predicted Answer: predicted_answer
Average Metric: 37.99999999999999 / 14 (271.4): 100%|█| 14/14 [05:27<00:00, 23.
Faithful: No Detail: 1 Overall: 5 Error for example in dev set: could not convert string to float: 'No' Average Metric: 37.99999999999999 / 14 (271.4%)
271.43
[68]
0%| | 0/14 [00:00<?, ?it/s]
Test Question: Are all ANN algorithms potential candidates to become an indexation plugin in Weaviate? Predicted Answer: predicted_answer
Average Metric: 3.6 / 1 (360.0): 7%|▋ | 1/14 [00:28<06:07, 28.25s/it]
Faithful: 4 Detail: 5 Overall: 5 Test Question: Does Weaviate use pre- or post-filtering ANN index search? Predicted Answer: predicted_answer
Average Metric: 7.6 / 2 (380.0): 14%|█▍ | 2/14 [00:54<05:24, 27.04s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: How does Weaviate's vector and scalar filtering work? Predicted Answer: predicted_answer
Average Metric: 9.2 / 3 (306.7): 21%|██▏ | 3/14 [01:15<04:26, 24.21s/it]
Faithful: 1 Detail: 1 Overall: 5 Test Question: What would you say is more important for query speed in Weaviate: More CPU power, or more RAM? Predicted Answer: predicted_answer
Average Metric: 11.6 / 4 (290.0): 29%|██▌ | 4/14 [01:39<04:03, 24.38s/it]
Faithful: 2 Detail: 5 Overall: 3 Test Question: Data import takes long / is slow, what is causing this and what can I do? Predicted Answer: predicted_answer
Average Metric: 15.0 / 5 (300.0): 36%|███▏ | 5/14 [02:03<03:38, 24.23s/it]
Faithful: 4 Detail: 5 Overall: 4 Test Question: How can slow queries be optimized? Predicted Answer: predicted_answer
Average Metric: 19.0 / 6 (316.7): 43%|███▊ | 6/14 [02:24<03:03, 22.95s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: When scalar and vector search are combined, will the scalar filter happen before or after the nearest neighbor (vector) search? Predicted Answer: predicted_answer
Average Metric: 20.8 / 7 (297.1): 50%|████▌ | 7/14 [02:45<02:36, 22.30s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: Regarding "filtered vector search": Since this is a two-phase pipeline, how big can that list of IDs get? Do you know how that size might affect query performance? Predicted Answer: predicted_answer
Average Metric: 23.0 / 8 (287.5): 57%|█████▏ | 8/14 [03:07<02:12, 22.15s/it]
Faithful: 1 Detail: 4 Overall: 5 Test Question: My Weaviate setup is using more memory than what I think is reasonable. How can I debug this? Predicted Answer: predicted_answer
Average Metric: 27.0 / 9 (300.0): 64%|█████▊ | 9/14 [03:45<02:16, 27.30s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: How can I print a stack trace of Weaviate? Predicted Answer: predicted_answer
Average Metric: 28.6 / 10 (286.0): 71%|█████ | 10/14 [04:12<01:49, 27.26s/it]
Faithful: 1 Detail: 1 Overall: 5 Test Question: Can I request a feature in Weaviate? Predicted Answer: predicted_answer
Average Metric: 32.6 / 11 (296.4): 79%|█████▌ | 11/14 [04:35<01:17, 25.70s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: What is Weaviate's consistency model in a distributed setup? Predicted Answer: predicted_answer
Average Metric: 36.6 / 12 (305.0): 86%|██████ | 12/14 [05:01<00:51, 25.97s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: With your aggregations I could not see how to do time buckets, is this possible? Predicted Answer: predicted_answer
Average Metric: 38.4 / 13 (295.4): 93%|██████▌| 13/14 [05:24<00:24, 24.91s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: How can I run the latest master branch with Docker Compose? Predicted Answer: predicted_answer
Average Metric: 40.0 / 14 (285.7): 100%|███████| 14/14 [05:47<00:00, 24.83s/it]
Faithful: 1 Detail: 1 Overall: 5 Average Metric: 40.0 / 14 (285.7%)
285.71
[69]
0%| | 0/14 [00:00<?, ?it/s]
Test Question: Are all ANN algorithms potential candidates to become an indexation plugin in Weaviate? Predicted Answer: predicted_answer
Average Metric: 3.6 / 1 (360.0): 7%|▋ | 1/14 [00:22<04:47, 22.08s/it]
Faithful: 5 Detail: 3 Overall: 5 Test Question: Does Weaviate use pre- or post-filtering ANN index search? Predicted Answer: predicted_answer
Average Metric: 5.4 / 2 (270.0): 14%|█▍ | 2/14 [00:55<05:41, 28.48s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: How does Weaviate's vector and scalar filtering work? Predicted Answer: predicted_answer
Average Metric: 7.2 / 3 (240.0): 21%|██▏ | 3/14 [01:21<05:05, 27.77s/it]
Faithful: 1 Detail: 2 Overall: 5 Test Question: What would you say is more important for query speed in Weaviate: More CPU power, or more RAM? Predicted Answer: predicted_answer
Average Metric: 11.2 / 4 (280.0): 29%|██▌ | 4/14 [01:47<04:28, 26.83s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Data import takes long / is slow, what is causing this and what can I do? Predicted Answer: predicted_answer
Average Metric: 12.0 / 5 (240.0): 36%|███▏ | 5/14 [02:15<04:05, 27.26s/it]
Faithful: 1 Detail: 1 Overall: 1 Test Question: How can slow queries be optimized? Predicted Answer: predicted_answer
Average Metric: 13.6 / 6 (226.7): 43%|███▊ | 6/14 [02:40<03:32, 26.58s/it]
Faithful: 1 Detail: 1 Overall: 5 Test Question: When scalar and vector search are combined, will the scalar filter happen before or after the nearest neighbor (vector) search? Predicted Answer: predicted_answer
Average Metric: 17.4 / 7 (248.6): 50%|████▌ | 7/14 [03:01<02:53, 24.78s/it]
Faithful: 5 Detail: 4 Overall: 5 Test Question: Regarding "filtered vector search": Since this is a two-phase pipeline, how big can that list of IDs get? Do you know how that size might affect query performance? Predicted Answer: predicted_answer
Average Metric: 19.599999999999998 / 8 (245.0): 57%|▌| 8/14 [03:23<02:23, 23.9
Faithful: 1 Detail: 4 Overall: 5 Test Question: My Weaviate setup is using more memory than what I think is reasonable. How can I debug this? Predicted Answer: predicted_answer
Average Metric: 21.2 / 9 (235.6): 64%|█████▊ | 9/14 [03:51<02:05, 25.07s/it]
Faithful: 1 Detail: 1 Overall: 5 Test Question: How can I print a stack trace of Weaviate? Predicted Answer: predicted_answer
Average Metric: 24.0 / 10 (240.0): 71%|█████ | 10/14 [04:16<01:40, 25.04s/it]
Faithful: 5 Detail: 1 Overall: 3 Test Question: Can I request a feature in Weaviate? Predicted Answer: predicted_answer
Average Metric: 27.4 / 11 (249.1): 79%|█████▌ | 11/14 [04:43<01:16, 25.59s/it]
Faithful: 5 Detail: 2 Overall: 5 Test Question: What is Weaviate's consistency model in a distributed setup? Predicted Answer: predicted_answer
Average Metric: 30.599999999999998 / 12 (255.0): 86%|▊| 12/14 [05:07<00:50, 25
Faithful: 5 Detail: 1 Overall: 5 Test Question: With your aggregations I could not see how to do time buckets, is this possible? Predicted Answer: predicted_answer
Average Metric: 34.199999999999996 / 13 (263.1): 93%|▉| 13/14 [05:30<00:24, 24
Faithful: 5 Detail: 3 Overall: 5 Test Question: How can I run the latest master branch with Docker Compose? Predicted Answer: predicted_answer
Average Metric: 35.8 / 14 (255.7): 100%|███████| 14/14 [05:51<00:00, 25.14s/it]
Faithful: 1 Detail: 1 Overall: 5 Average Metric: 35.8 / 14 (255.7%)
255.71
[70]
0%| | 0/14 [00:00<?, ?it/s]
Test Question: Are all ANN algorithms potential candidates to become an indexation plugin in Weaviate? Predicted Answer: predicted_answer
Average Metric: 3.2 / 1 (320.0): 7%|▋ | 1/14 [00:34<07:25, 34.28s/it]
Faithful: 3 Detail: 5 Overall: 5 Test Question: Does Weaviate use pre- or post-filtering ANN index search? Predicted Answer: predicted_answer
Average Metric: 6.0 / 2 (300.0): 14%|█▍ | 2/14 [00:59<05:45, 28.75s/it]
Faithful: 2 Detail: 5 Overall: 5 Test Question: How does Weaviate's vector and scalar filtering work? Predicted Answer: predicted_answer
Average Metric: 9.4 / 3 (313.3): 21%|██▏ | 3/14 [01:23<04:51, 26.51s/it]
Faithful: 5 Detail: 3 Overall: 4 Test Question: What would you say is more important for query speed in Weaviate: More CPU power, or more RAM? Predicted Answer: predicted_answer
Average Metric: 13.4 / 4 (335.0): 29%|██▌ | 4/14 [01:48<04:22, 26.28s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Data import takes long / is slow, what is causing this and what can I do? Predicted Answer: predicted_answer
Average Metric: 17.4 / 5 (348.0): 36%|███▏ | 5/14 [02:11<03:43, 24.84s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: How can slow queries be optimized? Predicted Answer: predicted_answer
Average Metric: 20.799999999999997 / 6 (346.7): 43%|▍| 6/14 [02:36<03:20, 25.0
Faithful: 5 Detail: 3 Overall: 4 Test Question: When scalar and vector search are combined, will the scalar filter happen before or after the nearest neighbor (vector) search? Predicted Answer: predicted_answer
Average Metric: 23.199999999999996 / 7 (331.4): 50%|▌| 7/14 [03:04<03:01, 25.8
Faithful: 2 Detail: 5 Overall: 3 Test Question: Regarding "filtered vector search": Since this is a two-phase pipeline, how big can that list of IDs get? Do you know how that size might affect query performance? Predicted Answer: predicted_answer
Average Metric: 26.999999999999996 / 8 (337.5): 57%|▌| 8/14 [03:31<02:37, 26.3
Faithful: 5 Detail: 5 Overall: 4 Test Question: My Weaviate setup is using more memory than what I think is reasonable. How can I debug this? Predicted Answer: predicted_answer
Average Metric: 30.799999999999997 / 9 (342.2): 64%|▋| 9/14 [03:56<02:09, 25.8
Faithful: 5 Detail: 4 Overall: 5 Test Question: How can I print a stack trace of Weaviate? Predicted Answer: predicted_answer
Average Metric: 34.8 / 10 (348.0): 71%|█████ | 10/14 [04:22<01:43, 25.98s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: Can I request a feature in Weaviate? Predicted Answer: predicted_answer
Average Metric: 38.8 / 11 (352.7): 79%|█████▌ | 11/14 [04:44<01:14, 24.75s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: What is Weaviate's consistency model in a distributed setup? Predicted Answer: predicted_answer
Average Metric: 42.8 / 12 (356.7): 86%|██████ | 12/14 [05:08<00:48, 24.38s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: With your aggregations I could not see how to do time buckets, is this possible? Predicted Answer: predicted_answer
Average Metric: 46.8 / 13 (360.0): 93%|██████▌| 13/14 [05:31<00:24, 24.08s/it]
Faithful: 5 Detail: 5 Overall: 5 Test Question: How can I run the latest master branch with Docker Compose? Predicted Answer: predicted_answer
Average Metric: 48.4 / 14 (345.7): 100%|███████| 14/14 [05:55<00:00, 25.37s/it]
Faithful: 1 Detail: 5 Overall: 1 Average Metric: 48.4 / 14 (345.7%)
345.71
[ ]