Getting Started With Azure Ai Search And Openai
Azure AI Search as a vector database for OpenAI embeddings
This notebook provides step by step instuctions on using Azure AI Search (f.k.a Azure Cognitive Search) as a vector database with OpenAI embeddings. Azure AI Search is a cloud search service that gives developers infrastructure, APIs, and tools for building a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications.
Prerequistites:
For the purposes of this exercise you must have the following:
Import required libraries
Configure OpenAI settings
This section guides you through setting up authentication for Azure OpenAI, allowing you to securely interact with the service using either Azure Active Directory (AAD) or an API key. Before proceeding, ensure you have your Azure OpenAI endpoint and credentials ready. For detailed instructions on setting up AAD with Azure OpenAI, refer to the official documentation.
Configure Azure AI Search Vector Store settings
This section explains how to set up the Azure AI Search client for integrating with the Vector Store feature. You can locate your Azure AI Search service details in the Azure Portal or programmatically via the Search Management SDK.
Load data
'vector_database_wikipedia_articles_embedded.zip'
Create an index
This code snippet demonstrates how to define and create a search index using the SearchIndexClient from the Azure AI Search Python SDK. The index incorporates both vector search and semantic ranker capabilities. For more details, visit our documentation on how to Create a Vector Index
azure-ai-search-openai-cookbook-demo created
Uploading Data to Azure AI Search Index
The following code snippet outlines the process of uploading a batch of documents—specifically, Wikipedia articles with pre-computed embeddings—from a pandas DataFrame to an Azure AI Search index. For a detailed guide on data import strategies and best practices, refer to Data Import in Azure AI Search.
Uploaded 25000 documents in total
If your dataset didn't already contain pre-computed embeddings, you can create embeddings by using the below function using the openai python library. You'll also notice the same function and model are being used to generate query embeddings for performing vector searches.
Content: April is the fourth month of the year in the Julian and Gregorian calendars, and comes between March Content vector generated
Perform a vector similarity search
Title: Documenta Score: 0.8599451 URL: https://simple.wikipedia.org/wiki/Documenta Title: Museum of Modern Art Score: 0.85260946 URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art Title: Expressionism Score: 0.852354 URL: https://simple.wikipedia.org/wiki/Expressionism
Perform a Hybrid Search
Hybrid search combines the capabilities of traditional keyword-based search with vector-based similarity search to provide more relevant and contextual results. This approach is particularly useful when dealing with complex queries that benefit from understanding the semantic meaning behind the text.
The provided code snippet demonstrates how to execute a hybrid search query:
Title: Wars of Scottish Independence Score: 0.03306011110544205 URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence Title: Battle of Bannockburn Score: 0.022253260016441345 URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn Title: Scottish Score: 0.016393441706895828 URL: https://simple.wikipedia.org/wiki/Scottish
Perform a Hybrid Search with Reranking (powered by Bing)
Semantic ranker measurably improves search relevance by using language understanding to rerank search results. Additionally, you can get extractive captions, answers, and highlights.
Semantic Answer: Advancements During the industrial revolution, new technology brought many changes. For example:<em> Canals</em> were built to allow heavy goods to be moved easily where they were needed. The steam engine became the main source of power. It replaced horses and human labor. Cheap iron and steel became mass-produced. Semantic Answer Score: 0.90478515625 Title: Industrial Revolution Reranker Score: 3.408700942993164 URL: https://simple.wikipedia.org/wiki/Industrial%20Revolution Caption: Advancements During the industrial revolution, new technology brought many changes. For example: Canals were built to allow heavy goods to be moved easily where they were needed. The steam engine became the main source of power. It replaced horses and human labor. Cheap iron and steel became mass-produced. Title: Printing Reranker Score: 1.603400707244873 URL: https://simple.wikipedia.org/wiki/Printing Caption: Machines to speed printing, cheaper paper, automatic stitching and binding all arrived in the 19th century during the industrial revolution. What had once been done by a few men by hand was now done by limited companies on huge machines. The result was much lower prices, and a much wider readership. Title: Industrialisation Reranker Score: 1.3238357305526733 URL: https://simple.wikipedia.org/wiki/Industrialisation Caption: <em>Industrialisation</em> (or<em> industrialization)</em> is a process that happens in countries when they start to use machines to do work that was once done by people.<em> Industrialisation changes</em> the things people do.<em> Industrialisation</em> caused towns to grow larger. Many people left farming to take higher paid jobs in factories in towns.