Notebooks
W
Weaviate Media Search Bind

Media Search Bind

vector-searchvector-databaseretrieval-augmented-generationllm-frameworksweaviate-featuresfunction-callingweaviate-recipesmedia-searchPythongenerative-ai
alph-notebooks/weaviate-recipes / media_search_bind.ipynb

Open In Colab

Image Audio Video Search with Meta AI ImageBind

This recipe demonstrates how build multi-modal search (image, audio, video) Meta AI ImageBind model (multi2vec-bind).

ImageBind allows us to search through text, image, audio and video.

This recipe will focus on searching through image, audio, video (skipping searching through text):

Weaviate Setup

The ImageBind model is only available with local Weaviate deployments with Docker or Kubernetes.

ImageBind is not supported with Weaviate Cloud Services (WCS).

Steps to deploy Weaviate locally with ImageBind

  1. Get a docker compose file.

    Run the following command in your terminal:

    curl -o docker-compose.yml "https://configuration.weaviate.io/v2/docker-compose/docker-compose.yml?bind_model=imagebind&generative_cohere=false&generative_openai=false&generative_palm=false&media_type=bind&modules=modules&ref2vec_centroid=false&reranker_cohere=false&reranker_transformers=false&runtime=docker-compose&weaviate_version=v1.21.8&weaviate_volume=named-volume"

    This will download docker-compose.yml file for you.

  2. Run Weaviate+Bind with Docker Compose

    If you are new to Docker Compose, here are instructions on how to install it.

    To start the docker image defined in the docker-compose.yml file, call:

    docker compose up

    Note #1 - the first time you run the command, Docker will download a ~6GB image.

    Note #2 – to shut down a running docker image, press CMD+C or CTRL+C.

Dependencies

[ ]

Configuration

[5]
True

Create Animals collection

The collection has the following key characteristics:

  1. Name: "Animals"
  2. Vectorizer: multi2vec-clip
  3. Image property: "img" - Weaviate will use values in "img" property to generate vectors. Note, you can call it anything you want.
[2]
Successfully created Animals collection.

Helper function to delete a specific resource

Only use, if you need to delete a specific type of resource

[ ]

Import Media

For every object, we will store:

  • name - the file name
  • path - path to the file, so that we could display returned images at query time.
  • (one of the following) media:
    • image - a base64 representation of the image file, Weaviate will use it to generate a vector - see imageFields.
    • audio - a base64 representation of the audio file, Weaviate will use it to generate a vector - see audioFields.
    • video - a base64 representation of the video file, Weaviate will use it to generate a vector - see videoFields.
[8]

Import images

[4]
Adding cat1.jpg
Adding cat2.jpg
Adding cat3.jpg
Adding dog1.jpg
Adding dog2.jpg
Adding dog3.jpg
Adding meerkat1.jpg
Adding meerkat2.jpg
Adding meerkat3.jpg

Import Audio

[9]
Adding mixkit-crowd-laugh-424.wav
Adding mixkit-big-thunder-with-rain-1291.wav
Adding mixkit-little-birds-singing-in-the-trees-17.wav
Adding mixkit-sick-man-sneeze-2213.wav
Adding mixkit-jungle-ape-sound-2419.wav
Adding mixkit-rooster-crowing-in-the-morning-2462.wav
Adding mixkit-dog-barking-twice-1.wav
Adding mixkit-small-group-cheer-and-applause-518.wav
Adding mixkit-rain-and-thunder-storm-2390.wav
Adding mixkit-cow-moo-1744.wav
Adding mixkit-cartoon-kitty-begging-meow-92.wav

Import Video

[19]
Adding cat-clean.mp4
Adding cat-play.mp4
Adding dog-high-five.mp4
Adding dog-with-stick.mp4
Adding meerkat-dig.mp4
Adding meerkat-watch.mp4

Check number of objects in the Animals collection

[21]
{'data': {'Aggregate': {'Animals': [{'meta': {'count': 26}}]}}}

Query examples

[11]

Text to Media search

[22]
[
  {
    "mediaType": "video",
    "name": "cat-play.mp4",
    "path": "./source/video/cat-play.mp4"
  },
  {
    "mediaType": "video",
    "name": "cat-clean.mp4",
    "path": "./source/video/cat-clean.mp4"
  },
  {
    "mediaType": "video",
    "name": "dog-with-stick.mp4",
    "path": "./source/video/dog-with-stick.mp4"
  }
]

Image to Media search

[16]
Output
[23]
[
  {
    "mediaType": "image",
    "name": "cat1.jpg",
    "path": "./source/image/cat1.jpg"
  },
  {
    "mediaType": "image",
    "name": "cat2.jpg",
    "path": "./source/image/cat2.jpg"
  },
  {
    "mediaType": "image",
    "name": "cat3.jpg",
    "path": "./source/image/cat3.jpg"
  },
  {
    "mediaType": "audio",
    "name": "mixkit-cartoon-kitty-begging-meow-92.wav",
    "path": "./source/audio/mixkit-cartoon-kitty-begging-meow-92.wav"
  },
  {
    "mediaType": "video",
    "name": "cat-clean.mp4",
    "path": "./source/video/cat-clean.mp4"
  }
]
Output

Audio to Media search

[20]
[24]
[
  {
    "mediaType": "video",
    "name": "dog-with-stick.mp4",
    "path": "./source/video/dog-with-stick.mp4"
  },
  {
    "mediaType": "video",
    "name": "dog-high-five.mp4",
    "path": "./source/video/dog-high-five.mp4"
  },
  {
    "mediaType": "audio",
    "name": "mixkit-dog-barking-twice-1.wav",
    "path": "./source/audio/mixkit-dog-barking-twice-1.wav"
  },
  {
    "mediaType": "image",
    "name": "dog2.jpg",
    "path": "./source/image/dog2.jpg"
  },
  {
    "mediaType": "image",
    "name": "dog3.jpg",
    "path": "./source/image/dog3.jpg"
  }
]

Video to Media search

[25]
[26]
[
  {
    "mediaType": "video",
    "name": "meerkat-watch.mp4",
    "path": "./source/video/meerkat-watch.mp4"
  },
  {
    "mediaType": "video",
    "name": "meerkat-dig.mp4",
    "path": "./source/video/meerkat-dig.mp4"
  },
  {
    "mediaType": "image",
    "name": "meerkat3.jpg",
    "path": "./source/image/meerkat3.jpg"
  }
]