Text To SQL

mistral-cookbookmistralfunction_calling

Text to SQL on multi-tables database

image.png

In this cookbook we will show you how to :

  • Use the function calling capabilities of Mistral models
  • Build a text2SQL architecture that scales more efficiently than a naive approach where all schemas are integrally injected in the system prompt
  • Evaluate your system with Mistral models and leveraging the DeepEval framework

Imports

[1]
Collecting mistralai
  Downloading mistralai-1.3.1-py3-none-any.whl.metadata (28 kB)
Requirement already satisfied: langchain in /usr/local/lib/python3.11/dist-packages (0.3.14)
Collecting deepeval
  Downloading deepeval-2.1.9-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: eval-type-backport>=0.2.0 in /usr/local/lib/python3.11/dist-packages (from mistralai) (0.2.2)
Requirement already satisfied: httpx>=0.27.0 in /usr/local/lib/python3.11/dist-packages (from mistralai) (0.28.1)
Collecting jsonpath-python>=1.0.6 (from mistralai)
  Downloading jsonpath_python-1.0.6-py3-none-any.whl.metadata (12 kB)
Requirement already satisfied: pydantic>=2.9.0 in /usr/local/lib/python3.11/dist-packages (from mistralai) (2.10.5)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from mistralai) (2.8.2)
Collecting typing-inspect>=0.9.0 (from mistralai)
  Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)
Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.11/dist-packages (from langchain) (6.0.2)
Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.11/dist-packages (from langchain) (2.0.37)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.11/dist-packages (from langchain) (3.11.11)
Requirement already satisfied: langchain-core<0.4.0,>=0.3.29 in /usr/local/lib/python3.11/dist-packages (from langchain) (0.3.29)
Requirement already satisfied: langchain-text-splitters<0.4.0,>=0.3.3 in /usr/local/lib/python3.11/dist-packages (from langchain) (0.3.5)
Requirement already satisfied: langsmith<0.3,>=0.1.17 in /usr/local/lib/python3.11/dist-packages (from langchain) (0.2.10)
Requirement already satisfied: numpy<2,>=1.22.4 in /usr/local/lib/python3.11/dist-packages (from langchain) (1.26.4)
Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.11/dist-packages (from langchain) (2.32.3)
Requirement already satisfied: tenacity!=8.4.0,<10,>=8.1.0 in /usr/local/lib/python3.11/dist-packages (from langchain) (9.0.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.11/dist-packages (from deepeval) (4.67.1)
Requirement already satisfied: pytest in /usr/local/lib/python3.11/dist-packages (from deepeval) (8.3.4)
Requirement already satisfied: tabulate in /usr/local/lib/python3.11/dist-packages (from deepeval) (0.9.0)
Requirement already satisfied: typer in /usr/local/lib/python3.11/dist-packages (from deepeval) (0.15.1)
Requirement already satisfied: rich in /usr/local/lib/python3.11/dist-packages (from deepeval) (13.9.4)
Requirement already satisfied: protobuf in /usr/local/lib/python3.11/dist-packages (from deepeval) (4.25.5)
Requirement already satisfied: sentry-sdk in /usr/local/lib/python3.11/dist-packages (from deepeval) (2.19.2)
Collecting pytest-repeat (from deepeval)
  Downloading pytest_repeat-0.9.3-py3-none-any.whl.metadata (4.9 kB)
Collecting pytest-xdist (from deepeval)
  Downloading pytest_xdist-3.6.1-py3-none-any.whl.metadata (4.3 kB)
Collecting portalocker (from deepeval)
  Downloading portalocker-3.1.1-py3-none-any.whl.metadata (8.6 kB)
Collecting langchain_openai (from deepeval)
  Downloading langchain_openai-0.3.1-py3-none-any.whl.metadata (2.7 kB)
Collecting langchain-community (from deepeval)
  Downloading langchain_community-0.3.15-py3-none-any.whl.metadata (2.9 kB)
Collecting docx2txt~=0.8 (from deepeval)
  Downloading docx2txt-0.8.tar.gz (2.8 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: importlib-metadata>=6.0.2 in /usr/local/lib/python3.11/dist-packages (from deepeval) (8.5.0)
Requirement already satisfied: opentelemetry-api<2.0.0,>=1.24.0 in /usr/local/lib/python3.11/dist-packages (from deepeval) (1.29.0)
Requirement already satisfied: opentelemetry-sdk<2.0.0,>=1.24.0 in /usr/local/lib/python3.11/dist-packages (from deepeval) (1.29.0)
Collecting opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0 (from deepeval)
  Downloading opentelemetry_exporter_otlp_proto_grpc-1.29.0-py3-none-any.whl.metadata (2.2 kB)
Collecting grpcio==1.67.1 (from deepeval)
  Downloading grpcio-1.67.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.9 kB)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.11/dist-packages (from deepeval) (1.6.0)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (2.4.4)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.2)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (24.3.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.1.0)
Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (0.2.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.18.3)
Requirement already satisfied: anyio in /usr/local/lib/python3.11/dist-packages (from httpx>=0.27.0->mistralai) (3.7.1)
Requirement already satisfied: certifi in /usr/local/lib/python3.11/dist-packages (from httpx>=0.27.0->mistralai) (2024.12.14)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.11/dist-packages (from httpx>=0.27.0->mistralai) (1.0.7)
Requirement already satisfied: idna in /usr/local/lib/python3.11/dist-packages (from httpx>=0.27.0->mistralai) (3.10)
Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.11/dist-packages (from httpcore==1.*->httpx>=0.27.0->mistralai) (0.14.0)
Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.11/dist-packages (from importlib-metadata>=6.0.2->deepeval) (3.21.0)
Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.11/dist-packages (from langchain-core<0.4.0,>=0.3.29->langchain) (1.33)
Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.11/dist-packages (from langchain-core<0.4.0,>=0.3.29->langchain) (24.2)
Requirement already satisfied: typing-extensions>=4.7 in /usr/local/lib/python3.11/dist-packages (from langchain-core<0.4.0,>=0.3.29->langchain) (4.12.2)
Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.11/dist-packages (from langsmith<0.3,>=0.1.17->langchain) (3.10.14)
Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from langsmith<0.3,>=0.1.17->langchain) (1.0.0)
Requirement already satisfied: deprecated>=1.2.6 in /usr/local/lib/python3.11/dist-packages (from opentelemetry-api<2.0.0,>=1.24.0->deepeval) (1.2.15)
Requirement already satisfied: googleapis-common-protos~=1.52 in /usr/local/lib/python3.11/dist-packages (from opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0->deepeval) (1.66.0)
Collecting opentelemetry-exporter-otlp-proto-common==1.29.0 (from opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0->deepeval)
  Downloading opentelemetry_exporter_otlp_proto_common-1.29.0-py3-none-any.whl.metadata (1.8 kB)
Collecting opentelemetry-proto==1.29.0 (from opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.24.0->deepeval)
  Downloading opentelemetry_proto-1.29.0-py3-none-any.whl.metadata (2.3 kB)
Collecting protobuf (from deepeval)
  Downloading protobuf-5.29.3-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Requirement already satisfied: opentelemetry-semantic-conventions==0.50b0 in /usr/local/lib/python3.11/dist-packages (from opentelemetry-sdk<2.0.0,>=1.24.0->deepeval) (0.50b0)
Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.11/dist-packages (from pydantic>=2.9.0->mistralai) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.2 in /usr/local/lib/python3.11/dist-packages (from pydantic>=2.9.0->mistralai) (2.27.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.8.2->mistralai) (1.17.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2->langchain) (3.4.1)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2->langchain) (2.3.0)
Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.11/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.1.1)
Collecting mypy-extensions>=0.3.0 (from typing-inspect>=0.9.0->mistralai)
  Downloading mypy_extensions-1.0.0-py3-none-any.whl.metadata (1.1 kB)
Collecting dataclasses-json<0.7,>=0.5.7 (from langchain-community->deepeval)
  Downloading dataclasses_json-0.6.7-py3-none-any.whl.metadata (25 kB)
Collecting httpx-sse<0.5.0,>=0.4.0 (from langchain-community->deepeval)
  Downloading httpx_sse-0.4.0-py3-none-any.whl.metadata (9.0 kB)
Collecting langchain
  Downloading langchain-0.3.15-py3-none-any.whl.metadata (7.1 kB)
Collecting langchain-core<0.4.0,>=0.3.29 (from langchain)
  Downloading langchain_core-0.3.31-py3-none-any.whl.metadata (6.3 kB)
Collecting pydantic-settings<3.0.0,>=2.4.0 (from langchain-community->deepeval)
  Downloading pydantic_settings-2.7.1-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: openai<2.0.0,>=1.58.1 in /usr/local/lib/python3.11/dist-packages (from langchain_openai->deepeval) (1.59.6)
Collecting tiktoken<1,>=0.7 (from langchain_openai->deepeval)
  Downloading tiktoken-0.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: iniconfig in /usr/local/lib/python3.11/dist-packages (from pytest->deepeval) (2.0.0)
Requirement already satisfied: pluggy<2,>=1.5 in /usr/local/lib/python3.11/dist-packages (from pytest->deepeval) (1.5.0)
Collecting execnet>=2.1 (from pytest-xdist->deepeval)
  Downloading execnet-2.1.1-py3-none-any.whl.metadata (2.9 kB)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.11/dist-packages (from rich->deepeval) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.11/dist-packages (from rich->deepeval) (2.18.0)
Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.11/dist-packages (from typer->deepeval) (8.1.8)
Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.11/dist-packages (from typer->deepeval) (1.5.4)
Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain-community->deepeval)
  Downloading marshmallow-3.25.1-py3-none-any.whl.metadata (7.3 kB)
Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.11/dist-packages (from deprecated>=1.2.6->opentelemetry-api<2.0.0,>=1.24.0->deepeval) (1.17.0)
Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.11/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.4.0,>=0.3.29->langchain) (3.0.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.11/dist-packages (from markdown-it-py>=2.2.0->rich->deepeval) (0.1.2)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.11/dist-packages (from openai<2.0.0,>=1.58.1->langchain_openai->deepeval) (1.9.0)
Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.11/dist-packages (from openai<2.0.0,>=1.58.1->langchain_openai->deepeval) (0.8.2)
Requirement already satisfied: sniffio in /usr/local/lib/python3.11/dist-packages (from openai<2.0.0,>=1.58.1->langchain_openai->deepeval) (1.3.1)
Collecting python-dotenv>=0.21.0 (from pydantic-settings<3.0.0,>=2.4.0->langchain-community->deepeval)
  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.11/dist-packages (from tiktoken<1,>=0.7->langchain_openai->deepeval) (2024.11.6)
Downloading mistralai-1.3.1-py3-none-any.whl (261 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 261.3/261.3 kB 6.7 MB/s eta 0:00:00
Downloading deepeval-2.1.9-py3-none-any.whl (588 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 588.2/588.2 kB 16.6 MB/s eta 0:00:00
Downloading grpcio-1.67.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 38.7 MB/s eta 0:00:00
Downloading jsonpath_python-1.0.6-py3-none-any.whl (7.6 kB)
Downloading opentelemetry_exporter_otlp_proto_grpc-1.29.0-py3-none-any.whl (18 kB)
Downloading opentelemetry_exporter_otlp_proto_common-1.29.0-py3-none-any.whl (18 kB)
Downloading opentelemetry_proto-1.29.0-py3-none-any.whl (55 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 2.9 MB/s eta 0:00:00
Downloading protobuf-5.29.3-cp38-abi3-manylinux2014_x86_64.whl (319 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 319.7/319.7 kB 18.0 MB/s eta 0:00:00
Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
Downloading langchain_community-0.3.15-py3-none-any.whl (2.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 30.3 MB/s eta 0:00:00
Downloading langchain-0.3.15-py3-none-any.whl (1.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 31.7 MB/s eta 0:00:00
Downloading langchain_core-0.3.31-py3-none-any.whl (412 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 412.2/412.2 kB 18.7 MB/s eta 0:00:00
Downloading langchain_openai-0.3.1-py3-none-any.whl (54 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 kB 3.5 MB/s eta 0:00:00
Downloading portalocker-3.1.1-py3-none-any.whl (19 kB)
Downloading pytest_repeat-0.9.3-py3-none-any.whl (4.2 kB)
Downloading pytest_xdist-3.6.1-py3-none-any.whl (46 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.1/46.1 kB 2.2 MB/s eta 0:00:00
Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB)
Downloading execnet-2.1.1-py3-none-any.whl (40 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.6/40.6 kB 1.8 MB/s eta 0:00:00
Downloading httpx_sse-0.4.0-py3-none-any.whl (7.8 kB)
Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Downloading pydantic_settings-2.7.1-py3-none-any.whl (29 kB)
Downloading tiktoken-0.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 29.4 MB/s eta 0:00:00
Downloading marshmallow-3.25.1-py3-none-any.whl (49 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.6/49.6 kB 2.6 MB/s eta 0:00:00
Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Building wheels for collected packages: docx2txt
  Building wheel for docx2txt (setup.py) ... done
  Created wheel for docx2txt: filename=docx2txt-0.8-py3-none-any.whl size=3960 sha256=7b726a0cbe5588eb27a3c2b8518f75ee7ca8ac8380573c834740aee90d33bdcd
  Stored in directory: /root/.cache/pip/wheels/0f/0e/7a/3094a4ceefe657bff7e12dd9592a9d5b6487ef4338ace0afa6
Successfully built docx2txt
Installing collected packages: docx2txt, python-dotenv, protobuf, portalocker, mypy-extensions, marshmallow, jsonpath-python, httpx-sse, grpcio, execnet, typing-inspect, tiktoken, pytest-xdist, pytest-repeat, opentelemetry-proto, pydantic-settings, opentelemetry-exporter-otlp-proto-common, mistralai, dataclasses-json, langchain-core, opentelemetry-exporter-otlp-proto-grpc, langchain_openai, langchain, langchain-community, deepeval
  Attempting uninstall: protobuf
    Found existing installation: protobuf 4.25.5
    Uninstalling protobuf-4.25.5:
      Successfully uninstalled protobuf-4.25.5
  Attempting uninstall: grpcio
    Found existing installation: grpcio 1.69.0
    Uninstalling grpcio-1.69.0:
      Successfully uninstalled grpcio-1.69.0
  Attempting uninstall: langchain-core
    Found existing installation: langchain-core 0.3.29
    Uninstalling langchain-core-0.3.29:
      Successfully uninstalled langchain-core-0.3.29
  Attempting uninstall: langchain
    Found existing installation: langchain 0.3.14
    Uninstalling langchain-0.3.14:
      Successfully uninstalled langchain-0.3.14
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow 2.17.1 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 5.29.3 which is incompatible.
Successfully installed dataclasses-json-0.6.7 deepeval-2.1.9 docx2txt-0.8 execnet-2.1.1 grpcio-1.67.1 httpx-sse-0.4.0 jsonpath-python-1.0.6 langchain-0.3.15 langchain-community-0.3.15 langchain-core-0.3.31 langchain_openai-0.3.1 marshmallow-3.25.1 mistralai-1.3.1 mypy-extensions-1.0.0 opentelemetry-exporter-otlp-proto-common-1.29.0 opentelemetry-exporter-otlp-proto-grpc-1.29.0 opentelemetry-proto-1.29.0 portalocker-3.1.1 protobuf-5.29.3 pydantic-settings-2.7.1 pytest-repeat-0.9.3 pytest-xdist-3.6.1 python-dotenv-1.0.1 tiktoken-0.8.0 typing-inspect-0.9.0
[5]

Load the Chinook database

"Chinook is a sample database available for SQL Server, Oracle, MySQL, etc. It can be created by running a single SQL script. Chinook database is an alternative to the Northwind database, being ideal for demos and testing ORM tools targeting single and multiple database servers."

To run this notebook you will need to download the Chinook datase. You will find more information about this database by clicking on this github link.

To create the Chinook.db in the same directory as this notebook you have several options :

  • You can download and build the database via the command line :
curl -s https://raw.githubusercontent.com/lerocha/chinook-database/master/ChinookDatabase/DataSources/Chinook_Sqlite.sql | sqlite3 Chinook.db
  • Another strategy consists in running the following script https://raw.githubusercontent.com/lerocha/chinook-database/master/ChinookDatabase/DataSources/Chinook_Sqlite.sql.

Firstly save the script to a folder/directory on your computer. Then create a database called Chinook with sqlite3 Chinook.db. Ultimately run the script with the command .read Chinook_Sqlite.sql

image.png

[3]
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  sqlite3-doc
The following NEW packages will be installed:
  sqlite3
0 upgraded, 1 newly installed, 0 to remove and 49 not upgraded.
Need to get 768 kB of archives.
After this operation, 1,873 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 sqlite3 amd64 3.37.2-2ubuntu0.3 [768 kB]
Fetched 768 kB in 1s (799 kB/s)
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78, <> line 1.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin: 
Selecting previously unselected package sqlite3.
(Reading database ... 124565 files and directories currently installed.)
Preparing to unpack .../sqlite3_3.37.2-2ubuntu0.3_amd64.deb ...
Unpacking sqlite3 (3.37.2-2ubuntu0.3) ...
Setting up sqlite3 (3.37.2-2ubuntu0.3) ...
Processing triggers for man-db (2.10.2-1) ...
[4]
[7]
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> .tables
Album          Employee       InvoiceLine    PlaylistTrack
Artist         Genre          MediaType      Track        
Customer       Invoice        Playlist     
sqlite> SELECT * FROM Artist LIMIT 5;
1|AC/DC
2|Accept
3|Aerosmith
4|Alanis Morissette
5|Alice In Chains
sqlite> .exit

Set up clients

[12]
Type your API Key··········

Interract with the Chinook database

We are defining two functions :

  • run_sql_query that runs sql code on Chinook
  • get_sql_schema_of_table that returns the schema of a table specified as input
[22]
"[(1, 'AC/DC'), (2, 'Accept'), (3, 'Aerosmith'), (4, 'Alanis Morissette'), (5, 'Alice In Chains'), (6, 'Antônio Carlos Jobim'), (7, 'Apocalyptica'), (8, 'Audioslave'), (9, 'BackBeat'), (10, 'Billy Cobham')]"
[23]

Build agent

[24]

Test the agent

Let's test the agent and ask a few random questions of increasing complexity

[25]
User: What is the oldest player in the NBA?

Assistant: I'm sorry, but I can only answer questions related to the Chinook database, which is a digital media store. I don't have information about the NBA or any other topics outside of the Chinook database. Please ask another question.

[28]
User: What are the genre in the store?

Tool: Getting SQL schema of table Genre

Tool: Running code SELECT Name FROM Genre

Assistant: Here are the genres available in the store:

1. Rock
2. Jazz
3. Metal
4. Alternative & Punk
5. Rock And Roll
6. Blues
7. Latin
8. Reggae
9. Pop
10. Soundtrack
11. Bossa Nova
12. Easy Listening
13. Heavy Metal
14. R&B/Soul
15. Electronica/Dance
16. World
17. Hip Hop/Rap
18. Science Fiction
19. TV Shows
20. Sci Fi & Fantasy
21. Drama
22. Comedy
23. Alternative
24. Classical
25. Opera

[29]
User: What are the albums of the rock band U2?

Tool: Getting SQL schema of table Album

Tool: Getting SQL schema of table Artist

Tool: Running code SELECT Title FROM Album WHERE ArtistId = (SELECT ArtistId FROM Artist WHERE Name = 'U2');

Assistant: The albums of the rock band U2 are :

- Achtung Baby
- All That You Can't Leave Behind
- B-Sides 1980-1990
- How To Dismantle An Atomic Bomb
- Pop
- Rattle And Hum
- The Best Of 1980-1990
- War
- Zooropa
- Instant Karma: The Amnesty International Campaign to Save Darfur

[31]
User: What is the shortest song that the rock band U2 ever composed?

Tool: Getting SQL schema of table Artist

Tool: Getting SQL schema of table Album

Tool: Getting SQL schema of table Track

Tool: Running code SELECT Name, Milliseconds FROM Track WHERE Composer = 'U2' ORDER BY Milliseconds ASC LIMIT 1;

Assistant: The shortest song that the rock band U2 ever composed is "Trash, Trampoline And The Party Girl".

[33]
User: Which track from U2 is the most sold?

Tool: Getting SQL schema of table Track

Tool: Getting SQL schema of table Artist

Tool: Getting SQL schema of table InvoiceLine

Tool: Running code SELECT Track.Name, SUM(InvoiceLine.Quantity) as TotalSold    FROM Track    JOIN InvoiceLine ON Track.TrackId = InvoiceLine.TrackId    JOIN Album ON Track.AlbumId = Album.AlbumId    JOIN Artist ON Album.ArtistId = Artist.ArtistId WHERE Artist.Name = 'U2'    GROUP BY Track.Name    ORDER BY TotalSold DESC    LIMIT 1;

Assistant: The most sold track from U2 is "Sweetest Thing" with 3 copies sold.

[34]
User: Which consumer bought the biggest amound of U2 songs?

Tool: Getting SQL schema of table Track

Tool: Getting SQL schema of table InvoiceLine

Tool: Getting SQL schema of table Invoice

Tool: Getting SQL schema of table Customer

Tool: Getting SQL schema of table Artist

Tool: Running code SELECT Customer.CustomerId, Customer.FirstName, Customer.LastName, SUM(InvoiceLine.Quantity) as total_quantity FROM Customer JOIN Invoice ON Customer.CustomerId = Invoice.CustomerId JOIN InvoiceLine ON Invoice.InvoiceId = InvoiceLine.InvoiceId JOIN Track ON InvoiceLine.TrackId = Track.TrackId JOIN Album ON Track.AlbumId = Album.AlbumId JOIN Artist ON Album.ArtistId = Artist.ArtistId WHERE Artist.Name = 'U2' GROUP BY Customer.CustomerId ORDER BY total_quantity DESC LIMIT 1;

Assistant: The consumer who bought the biggest amount of U2 songs is Madalena Sampaio. She bought 9 songs.

[35]
User: List all artist that have a color in their name

Tool: Getting SQL schema of table Artist

Tool: Running code SELECT Name FROM Artist WHERE Name LIKE '%red%' OR Name LIKE '%blue%' OR Name LIKE '%green%' OR Name LIKE '%yellow%' OR Name LIKE '%black%' OR Name LIKE '%white%' OR Name LIKE '%purple%' OR Name LIKE '%pink%' OR Name LIKE '%orange%' OR Name LIKE '%brown%' OR Name LIKE '%grey%' OR Name LIKE '%gray%';

Assistant: Here are the artists that have a color in their name:

1. Black Label Society
2. Black Sabbath
3. Pedro Luís & A Parede
4. Banda Black Rio
5. Green Day
6. Deep Purple
7. James Brown
8. Pink Floyd
9. Red Hot Chili Peppers
10. Simply Red
11. The Black Crowes
12. Whitesnake
13. Jackson Browne
14. Black Eyed Peas
15. Charlie Brown Jr.
16. Pedro Luís E A Parede

[36]
User: Who are our top Customers according to Invoices?

Tool: Getting SQL schema of table Customer

Tool: Getting SQL schema of table Invoice

Tool: Running code SELECT Customer.CustomerId, Customer.FirstName, Customer.LastName, SUM(Invoice.Total) as TotalSpent FROM Customer JOIN Invoice ON Customer.CustomerId = Invoice.CustomerId GROUP BY Customer.CustomerId ORDER BY TotalSpent DESC LIMIT 10;

Assistant: Here are our top 10 customers according to invoices :

1. Helena Holý : 49.62
2. Richard Cunningham : 47.62
3. Luis Rojas : 46.62
4. Ladislav Kovács : 45.62
5. Hugh O'Reilly : 45.62
6. Julia Barnett : 43.62
7. Frank Ralston : 43.62
8. Fynn Zimmermann : 43.62
9. Astrid Gruber : 42.62
10. Victor Stevens : 42.62

Evaluating

Let's try to evaluate the agent in a more formal way.

We will build a test set based on the questions from this Medium article Chinook question/answers

We will evaluate answers via LLM as a judge through the framework DeepEval from which the image here below is taken.

image.png

[37]
[38]
Event loop is already running. Applying nest_asyncio patch to allow async execution...
Evaluating 7 test case(s) in parallel: |██████████|100% (7/7) [Time Taken: 01:11, 10.16s/test case]

======================================================================

Metrics Summary

  - ✅ Correctness (GEval) (score: 1.0, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output does not contradict any facts in the expected output and directly answers the question., error: None)

For test case:

  - input: Which Employee has the Highest Total Number of Customers?
  - actual output: The employee with the highest total number of customers is Jane Peacock, with 21 customers.
  - expected output: Peacock Jane has the most customers (she has 21 customers)
  - context: None
  - retrieval context: None

======================================================================

Metrics Summary

  - ✅ Correctness (GEval) (score: 0.8, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output correctly identifies the top customers and their spending amounts, aligning with the facts in the expected output. However, it does not directly answer the question by explicitly stating the top five customers, which is a minor deviation from the expected output., error: None)

For test case:

  - input: Who are our top Customers according to Invoices?
  - actual output: Here are our top customers according to invoices :

| CustomerId | FirstName | LastName   | TotalSpent |
|------------|-----------|------------|------------|
| 6          | Helena    | Holý       | 49.62      |
| 26         | Richard   | Cunningham | 47.62      |
| 57         | Luis      | Rojas      | 46.62      |
| 45         | Ladislav  | Kovács     | 45.62      |
| 46         | Hugh      | O'Reilly   | 45.62      |
| 28         | Julia     | Barnett    | 43.62      |
| 24         | Frank     | Ralston    | 43.62      |
| 37         | Fynn      | Zimmermann | 43.62      |
| 7          | Astrid    | Gruber     | 42.62      |
| 25         | Victor    | Stevens    | 42.62      |
  - expected output: Helena Holy, Richard Cunningham, Luis Rojas, Ladislav Kovacs, and Hugh O’Reilly are the top five customers who have spent the highest amount of money according to the invoice
  - context: None
  - retrieval context: None

======================================================================

Metrics Summary

  - ✅ Correctness (GEval) (score: 0.9, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output directly answers the question and does not contradict the expected output., error: None)

For test case:

  - input: How many Rock music listeners are there?
  - actual output: There are 59 Rock music listeners.
  - expected output: We found out that all 59 customers in the database have listened to Rock Music.
  - context: None
  - retrieval context: None

======================================================================

Metrics Summary

  - ✅ Correctness (GEval) (score: 0.8, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output does not contradict any facts in the expected output and directly answers the question. However, the format and some details differ from the expected output., error: None)

For test case:

  - input: What artists have written most rock music songs?
  - actual output: The artists who have written the most rock music songs are:

1. **Led Zeppelin** with 114 songs
2. **U2** with 112 songs
3. **Deep Purple** with 92 songs
4. **Iron Maiden** with 81 songs
5. **Pearl Jam** with 54 songs
6. **Van Halen** with 52 songs
7. **Queen** with 45 songs
8. **The Rolling Stones** with 41 songs
9. **Creedence Clearwater Revival** with 40 songs
10. **Kiss** with 35 songs
  - expected output: Led Zeppelin tops the list of Artists who have written the most Rock Music with 114 songs followed Closely by U2 with 112 music.
  - context: None
  - retrieval context: None

======================================================================

Metrics Summary

  - ✅ Correctness (GEval) (score: 0.8, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output correctly identifies Iron Maiden as the artist who earned the most, matching the expected output. However, it uses 'approximately' and 'units of currency' instead of the exact amount and currency symbol '$' as in the expected output., error: None)

For test case:

  - input: Which artist has earned the most according to the Invoice Lines? How much is it?
  - actual output: The artist who earned the most according to the invoice lines is Iron Maiden, with a total earning of approximately 138.60 units of currency.
  - expected output: The Artist who has earned the most according to the invoice lines is Iron Maiden with a total of $138.6.
  - context: None
  - retrieval context: None

======================================================================

Metrics Summary

  - ✅ Correctness (GEval) (score: 0.8, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output does not contradict any facts in the expected output and answers the question. However, it lacks additional details provided in the expected output, such as the total number of songs and the average music length., error: None)

For test case:

  - input: How many tracks have a song length greater than the average song length?
  - actual output: There are 494 tracks with a song length greater than the average song length.
  - expected output: Out of 3503 songs in the database, we found out that 494 of these songs have length more than the average music length of 393,599.21 milliseconds.
  - context: None
  - retrieval context: None

======================================================================

Metrics Summary

  - ✅ Correctness (GEval) (score: 0.5, threshold: 0.5, strict: False, evaluation model: Mistral-large-latest, reason: The actual output correctly identifies Rock as the most popular genre, matching the expected output. However, it provides additional details about other genres and their sales, which are not present in the expected output. It answers the question, but the additional information is not required., error: None)

For test case:

  - input: What is the most popular genre for Australia?
  - actual output: The most popular genre in Australia is Rock, with a total sales of approximately $21.78. This is followed by Metal with approximately $7.92 in sales. Other genres such as Heavy Metal, Reggae, Latin, and Blues have lower sales ranging from approximately $3.00 to $1.00.
  - expected output: Rock is the most popular song for Australia
  - context: None
  - retrieval context: None

======================================================================

Overall Metric Pass Rates

Correctness (GEval): 100.00% pass rate

======================================================================


EvaluationResult(test_results=[TestResult(name='test_case_0', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=1.0, reason='The actual output does not contradict any facts in the expected output and directly answers the question.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n    "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n    "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='Which Employee has the Highest Total Number of Customers?', actual_output='The employee with the highest total number of customers is Jane Peacock, with 21 customers.', expected_output='Peacock Jane has the most customers (she has 21 customers)', context=None, retrieval_context=None), TestResult(name='test_case_1', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.8, reason='The actual output correctly identifies the top customers and their spending amounts, aligning with the facts in the expected output. However, it does not directly answer the question by explicitly stating the top five customers, which is a minor deviation from the expected output.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n    "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n    "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='Who are our top Customers according to Invoices?', actual_output="Here are our top customers according to invoices :\n\n| CustomerId | FirstName | LastName   | TotalSpent |\n|------------|-----------|------------|------------|\n| 6          | Helena    | Holý       | 49.62      |\n| 26         | Richard   | Cunningham | 47.62      |\n| 57         | Luis      | Rojas      | 46.62      |\n| 45         | Ladislav  | Kovács     | 45.62      |\n| 46         | Hugh      | O'Reilly   | 45.62      |\n| 28         | Julia     | Barnett    | 43.62      |\n| 24         | Frank     | Ralston    | 43.62      |\n| 37         | Fynn      | Zimmermann | 43.62      |\n| 7          | Astrid    | Gruber     | 42.62      |\n| 25         | Victor    | Stevens    | 42.62      |", expected_output='Helena Holy, Richard Cunningham, Luis Rojas, Ladislav Kovacs, and Hugh O’Reilly are the top five customers who have spent the highest amount of money according to the invoice', context=None, retrieval_context=None), TestResult(name='test_case_2', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.9, reason='The actual output directly answers the question and does not contradict the expected output.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n    "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n    "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='How many Rock music listeners are there?', actual_output='There are 59 Rock music listeners.', expected_output='We found out that all 59 customers in the database have listened to Rock Music.', context=None, retrieval_context=None), TestResult(name='test_case_3', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.8, reason='The actual output does not contradict any facts in the expected output and directly answers the question. However, the format and some details differ from the expected output.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n    "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n    "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='What artists have written most rock music songs?', actual_output='The artists who have written the most rock music songs are:\n\n1. **Led Zeppelin** with 114 songs\n2. **U2** with 112 songs\n3. **Deep Purple** with 92 songs\n4. **Iron Maiden** with 81 songs\n5. **Pearl Jam** with 54 songs\n6. **Van Halen** with 52 songs\n7. **Queen** with 45 songs\n8. **The Rolling Stones** with 41 songs\n9. **Creedence Clearwater Revival** with 40 songs\n10. **Kiss** with 35 songs', expected_output='Led Zeppelin tops the list of Artists who have written the most Rock Music with 114 songs followed Closely by U2 with 112 music.', context=None, retrieval_context=None), TestResult(name='test_case_4', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.8, reason="The actual output correctly identifies Iron Maiden as the artist who earned the most, matching the expected output. However, it uses 'approximately' and 'units of currency' instead of the exact amount and currency symbol '$' as in the expected output.", strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n    "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n    "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='Which artist has earned the most according to the Invoice Lines? How much is it?', actual_output='The artist who earned the most according to the invoice lines is Iron Maiden, with a total earning of approximately 138.60 units of currency.', expected_output='The Artist who has earned the most according to the invoice lines is Iron Maiden with a total of $138.6.', context=None, retrieval_context=None), TestResult(name='test_case_5', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.8, reason='The actual output does not contradict any facts in the expected output and answers the question. However, it lacks additional details provided in the expected output, such as the total number of songs and the average music length.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n    "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n    "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='How many tracks have a song length greater than the average song length?', actual_output='There are 494 tracks with a song length greater than the average song length.', expected_output='Out of 3503 songs in the database, we found out that 494 of these songs have length more than the average music length of 393,599.21 milliseconds.', context=None, retrieval_context=None), TestResult(name='test_case_6', success=True, metrics_data=[MetricData(name='Correctness (GEval)', threshold=0.5, success=True, score=0.5, reason='The actual output correctly identifies Rock as the most popular genre, matching the expected output. However, it provides additional details about other genres and their sales, which are not present in the expected output. It answers the question, but the additional information is not required.', strict_mode=False, evaluation_model='Mistral-large-latest', error=None, evaluation_cost=None, verbose_logs='Criteria:\nDetermine whether the actual output is factually correct based on the expected output. \n \nEvaluation Steps:\n[\n    "Check whether the facts in \'actual output\' contradicts any facts in \'expected output",\n    "You should also heavily penalize situations where the actual output does not answers the question"\n]')], conversational=False, multimodal=False, input='What is the most popular genre for Australia?', actual_output='The most popular genre in Australia is Rock, with a total sales of approximately $21.78. This is followed by Metal with approximately $7.92 in sales. Other genres such as Heavy Metal, Reggae, Latin, and Blues have lower sales ranging from approximately $3.00 to $1.00.', expected_output='Rock is the most popular song for Australia', context=None, retrieval_context=None)], confident_link=None)
[ ]