Notebooks
M
Microsoft Aoai Solution

Aoai Solution

llmsgenerativeaichatgpt08-building-search-applicationsmicrosoft-for-beginnerssemantic-searchgptopenaiAItransformersprompt-engineeringdall-eazurePythongenerative-ailanguage-modelmicrosoft-generative-ai-for-beginners
alph-notebooks/microsoft-generative-ai-for-beginners / aoai-solution.ipynb

In order to run the following notebooks, if you haven't done yet, you need to deploy a model that uses text-embedding-ada-002 as base model and set the deployment name inside .env file as AZURE_OPENAI_EMBEDDINGS_ENDPOINT

[ ]

Next, we are going to load the Embedding Index into a Pandas Dataframe. The Embedding Index is stored in a JSON file called embedding_index_3m.json. The Embedding Index contains the Embeddings for each of the YouTube transcripts up until late Oct 2023.

[ ]

Next, we are going to create a function called get_videos that will search the Embedding Index for the query. The function will return the top 5 videos that are most similar to the query. The function works as follows:

  1. First, a copy of the Embedding Index is created.
  2. Next, the Embedding for the query is calculated using the OpenAI Embedding API.
  3. Then a new column is created in the Embedding Index called similarity. The similarity column contains the cosine similarity between the query Embedding and the Embedding for each video segment.
  4. Next, the Embedding Index is filtered by the similarity column. The Embedding Index is filtered to only include videos that have a cosine similarity greater than or equal to 0.75.
  5. Finally, the Embedding Index is sorted by the similarity column and the top 5 videos are returned.
[ ]

This function is very simple, it just prints out the results of the search query.

[ ]
  1. First, the Embedding Index is loaded into a Pandas Dataframe.
  2. Next, the user is prompted to enter a query.
  3. Then the get_videos function is called to search the Embedding Index for the query.
  4. Finally, the display_results function is called to display the results to the user.
  5. The user is then prompted to enter another query. This process continues until the user enters exit.

You will be prompted to enter a query. Enter a query and press enter. The application will return a list of videos that are relevant to the query. The application will also return a link to the place in the video where the answer to the question is located.

Here are some queries to try out:

  • What is Azure Machine Learning?
  • How do convolutional neural networks work?
  • What is a neural network?
  • Can I use Jupyter Notebooks with Azure Machine Learning?
  • What is ONNX?
[ ]