Example Data Migration
title: Migrating Data from One Langfuse Project to Another description: Script to migrate data (prompts, traces, datasets) from a source Langfuse project to a destination Langfuse project
Migrating Data from One Langfuse Project to Another
This guide provides Python code to migrate data (prompts, traces, datasets) from a source Langfuse project to a destination Langfuse project using the Langfuse Public API.
This is useful for scenarios such as:
- Moving data from Langfuse Cloud to a self-hosted instance (or vice-versa).
- Migrating between different Langfuse Cloud regions or compliance environments (e.g., non-HIPAA to HIPAA).
- Syncing production data to a development/staging environment for testing.
- Creating custom scripts to synchronize specific data types (like prompts) across instances.
Prerequisites
- Langfuse Projects: Access to both the source and destination Langfuse projects.
- Rate Limits: Be mindful of API rate limits on your Langfuse instances (Cloud or self-hosted). Large migrations might require implementing back-off logic or running the script in batches.
Section I – Migrating Prompts
This section covers migrating prompts, including their version history.
Section II – Migrating Traces, Observations & Scores
This section handles the core tracing data: traces, their nested observations (spans, generations, events), and associated scores.
Section III – Migrating Datasets
This section covers migrating datasets, dataset items, and dataset runs (used for evaluations and experiments).
Other Possible Migratable Objects
The Langfuse Public API allows migrating other configuration objects, although they are less commonly required than prompts, traces, and datasets. Implementing their migration follows a similar pattern (list from source, create in destination).
- Score Configs (
/api/public/score-configs): Define configurations for how scores are calculated or displayed (e.g., category mappings, numeric ranges). Useful if you rely heavily on predefined score structures. API Ref - Model Definitions (
/api/public/models): Define custom model pricing or metadata for models not built-in to Langfuse. API Ref - Annotation Queues: (https://api.reference.langfuse.com/#tag/annotationqueues)
UI / Manual Migration
Some Langfuse features are primarily managed through the UI and do not currently have direct Public API endpoints for creation or migration:
- Model-Based Evaluators: Configurations for running evaluations using LLMs (e.g., "LLM-as-a-Judge"). These need to be recreated manually in the destination project's UI.
- Dashboards: Custom analytics dashboards created in the UI. You can often export the dashboard configuration as JSON from the source UI and import it into the destination UI.
- RBAC / Team Members: User roles and permissions are managed at the project or organization level, typically through the UI or specific SSO/SCIM integrations (for self-hosted enterprise). They are not part of a standard project data migration via this script.