Observability for Databricks Models with Langfuse

Databricks provides a powerful platform for hosting and serving large language models. By combining Databricks' serving endpoints with Langfuse, you can trace, monitor, and analyze your AI workloads in development and production.

This notebook demonstrates three different ways to use Databricks models with Langfuse:

1. OpenAI SDK: Use Databricks model endpoints via the OpenAI SDK.
2. LangChain: Integrate with the Databricks LLM interface in a LangChain pipeline.
3. LlamaIndex: Use Databricks endpoints within LlamaIndex.

What is Databricks Model Serving?
Databricks Model Serving allows you to serve large-scale models in a production environment, with automatic scaling and a robust infrastructure. It also enables you to fine-tune LLMs on your private data, ensuring your models can leverage proprietary information while maintaining data privacy.

What is Langfuse?
Langfuse is an open source platform for LLM observability and monitoring. It helps you trace and monitor your AI applications by capturing metadata, prompt details, token usage, latency, and more.

1. Install Dependencies

Before you begin, install the necessary packages in your Python environment:

• openai: Needed to call Databricks endpoints via the OpenAI SDK.
• databricks-langchain: Needed to call Databricks endpoints via an "OpenAI-like" interface.
• llama-index and llama-index-llms-databricks: For using Databricks endpoints within LlamaIndex.
• langfuse: Required for sending trace data to the Langfuse platform.

2. Set Up Environment Variables

Configure your Langfuse credentials and Databricks credentials as environment variables. Replace the dummy keys below with the real ones from your respective accounts.

• LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY: From your Langfuse Project Settings.
• LANGFUSE_BASE_URL: https://cloud.langfuse.com (EU region) or https://us.cloud.langfuse.com (US region).
• DATABRICKS_TOKEN: Your Databricks personal access token.
• DATABRICKS_HOST: Your Databricks workspace URL (e.g., https://dbc-xxxxxxx.cloud.databricks.com).

Approach 1: Using Databricks Models via the OpenAI SDK

Databricks endpoints can act as a drop-in replacement for the OpenAI API. This makes it easy to integrate with existing code that relies on the openai library. Under the hood, langfuse.openai.OpenAI automatically traces your requests to Langfuse.

Steps

1. Import the OpenAI client from langfuse.openai.
2. Create a client, setting api_key to your Databricks token and base_url to your Databricks workspace endpoints.
3. Use the client’s chat.completions.create() method to send a prompt.
4. See the trace in your Langfuse dashboard.

Note: For more examples on tracing OpenAI with Langfuse see the OpenAI integration docs.

Once the request completes, log in to your Langfuse dashboard and look for the new trace. You will see details like the prompt, response, latency, token usage, etc.

Link to public trace in Langfuse

Approach 2: Using LangChain

Databricks models can also be used via LangChain. The ChatDatabricks class wraps your Databricks Model Serving endpoint.

Steps

1. Set DATABRICKS_HOST as an environment variable.
2. Initialize a Langfuse CallbackHandler that automatically collects trace data.
3. Use ChatDatabricks with your endpoint name, temperature, or other parameters.
4. Invoke the model with messages and pass in the Langfuse callback handler.
5. See the trace in your Langfuse dashboard.

Note: For more examples on tracing LangChain with Langfuse see the LangChain integration docs.

After running the code, open your Langfuse dashboard to see the recorded conversation.

Link to public trace in Langfuse

Approach 3: Using LlamaIndex

If you use LlamaIndex for data ingestion, indexing, or retrieval-augmented generation, you can replace the default LLM with a Databricks endpoint.

Steps

1. Import Databricks from llama_index.llms.databricks.
2. Initialize a Databricks LLM with your endpoint name and Databricks credentials.
3. Use LlamaIndexInstrumentor from langfuse.llama_index to enable automatic tracing.
4. Invoke the LLM with a chat request.
5. See the trace in your Langfuse dashboard.

Note: For more examples on tracing LlamaIndex with Langfuse see the LlamaIndex integration docs.

You can now log into Langfuse to view your LlamaIndex calls, with details on prompts, token usage, completion data, and more.

Link to public trace in Langfuse

Next Steps

• See how to use Databricks models in the Langfuse Playground and for LLM-as-a-Judge evaluations here.
• Explore the Databricks documentation for advanced model serving configurations.
• Learn more about Langfuse tracing features to track your entire application flow.
• Try out Langfuse Prompt Management or set up LLM-as-a-Judge evaluations.