1. Executive Summary

1.1. Purpose and Audience

This notebook provides a hands-on guide for building temporally-aware knowledge graphs and performing multi-hop retrieval directly over those graphs.

It's designed for engineers, architects, and analysts working on temporally-aware knowledge graphs. Whether you’re prototyping, deploying at scale, or exploring new ways to use structured data, you’ll find practical workflows, best practices, and decision frameworks to accelerate your work.

This cookbook presents two hands-on workflows you can use, extend, and deploy right away:

1. Temporally-aware knowledge graph (KG) construction
   

   A key challenge in developing knowledge-driven AI systems is maintaining a database that stays current and relevant. While much attention is given to boosting retrieval accuracy with techniques like semantic similarity and re-ranking, this guide focuses on a fundamental—yet frequently overlooked—aspect: systematically updating and validating your knowledge base as new data arrives.

   No matter how advanced your retrieval algorithms are, their effectiveness is limited by the quality and freshness of your database. This cookbook demonstrates how to routinely validate and update knowledge graph entries as new data arrives, helping ensure that your knowledge base remains accurate and up to date.

2. Multi-hop retrieval using knowledge graphs
   

   Learn how to combine OpenAI models (such as o3, o4-mini, GPT-4.1, and GPT-4.1-mini) with structured graph queries via tool calls, enabling the model to traverse your graph in multiple steps across entities and relationships.

   This method lets your system answer complex, multi-faceted questions that require reasoning over several linked facts, going well beyond what single-hop retrieval can accomplish.

Inside, you'll discover:

• Practical decision frameworks for choosing models and prompting techniques at each stage
• Plug-and-play code examples for easy integration into your ML and data pipelines
• Links to in-depth resources on OpenAI tool use, fine-tuning, graph backend selection, and more
• A clear path from prototype to production, with actionable best practices for scaling and reliability

Note: All benchmarks and recommendations are based on the best available models and practices as of June 2025. As the ecosystem evolves, periodically revisit your approach to stay current with new capabilities and improvements.

1.2. Key takeaways

Creating a Temporally-Aware Knowledge Graph with a Temporal Agent

1. Why make your knowledge graph temporal?
   

   Traditional knowledge graphs treat facts as static, but real-world information evolves constantly. What was true last quarter may be outdated today, risking errors or misinformed decisions if the graph does not capture change over time. Temporal knowledge graphs allow you to precisely answer questions like “What was true on a given date?” or analyse how facts and relationships have shifted, ensuring decisions are always based on the most relevant context.

2. What is a Temporal Agent?
   

   A Temporal Agent is a pipeline component that ingests raw data and produces time-stamped triplets for your knowledge graph. This enables precise time-based querying, timeline construction, trend analysis, and more.

3. How does the pipeline work?
   

   The pipeline starts by semantically chunking your raw documents. These chunks are decomposed into statements ready for our Temporal Agent, which then creates time-aware triplets. An Invalidation Agent can then perform temporal validity checks, spotting and handling any statements that are invalidated by new statements that are incident on the graph.

Multi-Step Retrieval Over a Knowledge Graph

1. Why use multi-step retrieval?
   

   Direct, single-hop queries frequently miss salient facts distributed across a graph's topology. Multi-step (multi-hop) retrieval enables iterative traversal, following relationships and aggregating evidence across several hops. This methodology surfaces complex dependencies and latent connections that would remain hidden with one-shot lookups, providing more comprehensive and nuanced answers to sophisticated queries.

2. Planners
   

   Planners orchestrate the retrieval process. Task-orientated planners decompose queries into concrete, sequential subtasks. Hypothesis-orientated planners, by contrast, propose claims to confirm, refute, or evolve. Choosing the optimal strategy depends on where the problem lies on the spectrum from deterministic reporting (well-defined paths) to exploratory research (open-ended inference).

3. Tool Design Paradigms
   

   Tool design spans a continuum: Fixed tools provide consistent, predictable outputs for specific queries (e.g., a service that always returns today’s weather for San Francisco). At the other end, Free-form tools offer broad flexibility, such as code execution or open-ended data retrieval. Semi-structured tools fall between these extremes, restricting certain actions while allowing tailored flexibility—specialized sub-agents are a typical example. Selecting the appropriate paradigm is a trade-off between control, adaptability, and complexity.

4. Evaluating Retrieval Systems
   

   High-fidelity evaluation hinges on expert-curated "golden" answers, though these are costly and labor-intensive to produce. Automated judgments, such as those from LLMs or tool traces, can be quickly generated to supplement or pre-screen, but may lack the precision of human evaluation. As your system matures, transition towards leveraging real user feedback to measure and optimize retrieval quality in production.

   A proven workflow: Start with synthetic tests, benchmark on your curated human-annotated "golden" dataset, and iteratively refine using live user feedback and ratings.

Prototype to Production

1. Keep the graph lean
   

   Established archival policies and assign numeric relevance scores to each edge (e.g., recency x trust x query-frequency). Automate the archival or sparsification of low-value nodes and edges, ensuring only the most critical and frequently accessed facts remain for rapid retrieval.

2. Parallelize the ingestion pipeline
   

   Transition from a linear document → chunk → extraction → resolution pipeline to a staged, asynchronous architecture. Assign each processing phase its own queue and dedicated worker pool. Apply clustering or network-based batching for invalidation jobs to maximize efficiency. Batch external API requests (e.g., OpenAI) and database writes wherever possible. This design increases throughput, introduces backpressure for reliability, and allows you to scale each pipeline stage independently.

3. Integrate Robust Production Safeguards
   

   Enforce rigorous output validation: standardise temporal fields (e.g., ISO-8601 date formatting), constrain entity types to your controlled vocabulary, and apply lightweight model-based sanity checks for output consistency. Employ structured logging with traceable identifiers and monitor real-time quality and performance metrics in real lime to proactively detect data drift, regressions, or pipeline anomalised before they impact downstream applications.

2. How to Use This Cookbook

This cookbook is designed for flexible engagement:

1. Use it as a comprehensive technical guide—read from start to finish for a deep understanding of temporally-aware knowledge graph systems.
2. Skim for advanced concepts, methodologies, and implementation patterns if you prefer a high-level overview.
3. Jump into any of the three modular sections; each is self-contained and directly applicable to real-world scenarios.

Inside, you'll find:

1. Creating a Temporally-Aware Knowledge Graph with a Temporal Agent
   

   Build a pipeline that extracts entities and relations from unstructured text, resolves temporal conflicts, and keeps your graph up-to-date as new information arrives.

2. Multi-Step Retrieval Over a Knowledge Graph
   

   Use structured queries and language model reasoning to chain multiple hops across your graph and answer complex questions.

3. Prototype to Production
   

   Move from experimentation to deployment. This section covers architectural tips, integration patterns, and considerations for scaling reliably.

2.1. Pre-requisites

Before diving into building temporal agents and knowledge graphs, let's set up your environment. Install all required dependencies with pip, and set your OpenAI API key as an environment variable. Python 3.12 or later is required.

3. Creating a Temporally-Aware Knowledge Graph with a Temporal Agent

Accurate data is the foundation of any good business decision. OpenAI’s latest models like o3, o4-mini, and the GPT-4.1 family are enabling businesses to build state-of-the-art retrieval systems for their most important workflows. However, information evolves rapidly: facts ingested confidently yesterday may already be outdated today.

Without the ability to track when each fact was valid, retrieval systems risk returning answers that are outdated, non-compliant, or misleading. The consequences of missing temporal context can be severe in any industry, as illustrated by the following examples.

While we've called out some specific examples here, this theme is true across many industries including pharmaceuticals, law, consumer goods, and more.

Looking beyond standard retrieval

A temporally-aware knowledge graph allows you to go beyond static fact lookup. It enables richer retrieval workflows such as factual Q&A grounded in time, timeline generation, change tracking, counterfactual analysis, and more. We dive into these in more detail in our retrieval section later in the cookbook.

3.1. Introducing our Temporal Agent

A temporal agent is a specialized pipeline that converts raw, free-form statements into time-aware triplets ready for ingesting into a knowledge graph that can then be queried with the questions of the character “What was true at time T?”.

Triplets are the basic building blocks of knowledge graphs. It's a way to represent a single fact or piece of knowledge using three parts (hence, "triplet"):

• Subject - the entity you are talking about
• Predicate - the type of relationship or property
• Object - the value or other entity that the subject is connected to

You can thinking of this like a sentence with a structure [Subject] - [Predicate] - [Object]. As a more clear example:

"London" - "isCapitalOf" - "United Kingdom"

The Temporal Agent implemented in this cookbook draws inspiration from Zep and Graphiti, while introducing tighter control over fact invalidation and a more nuanced approach to episodic typing.

3.1.1. Key enhancements introduced in this cookbook

1. Temporal validity extraction
   

   Builds on Graphiti's prompt design to identify temporal spans and episodic context without requiring auxiliary reference statements.

2. Fact invalidation logic
   

   Introduces bidirectionality checks and constrains comparisons by episodic type. This retains Zep's non-lossy approach while reducing unnecessary evaluations.

3. Temporal & episodic typing
   

   Differentiates between Fact, Opinion, Prediction, as well as between temporal classes Static, Dynamic, Atemporal.

4. Multi‑event extraction
   

   Handles compound sentences and nested date references in a single pass.

This process allows us to update our sources of truth efficiently and reliably:

Note: While the implementation in this cookbook is focused on a graph-based implementation, this approach is generalizable to other knowledge base structures e.g., pgvector-based systems.

3.1.2. The Temporal Agent Pipeline

The Temporal Agent processes incoming statements through a three-stage pipeline:

1. Temporal Classification
   

   Labels each statement as Atemporal, Static, or Dynamic:

   • Atemporal statements never change (e.g., “The speed of light in a vaccuum is ≈3×10⁸ m s⁻¹”).
   • Static statements are valid from a point in time but do not change afterwards (e.g., "Person YY was CEO of Company XX on October 23rd 2014.").
   • Dynamic statements evolve (e.g., "Person YY is CEO of Company XX.").
2. Temporal Event Extraction
   

   Identifies relative or partial dates (e.g., “Tuesday”, “three months ago”) and resolves them to an absolute date using the document timestamp or fallback heuristics (e.g., default to the 1st or last of the month if only the month is known).

3. Temporal Validity Check
   

   Ensures every statement includes a t_created timestamp and, when applicable, a t_expired timestamp. The agent then compares the candidate triplet to existing knowledge graph entries to:

   • Detect contradictions and mark outdated entries with t_invalid
   • Link newer statements to those they invalidate with invalidated_by

3.1.3. Selecting the right model for a Temporal Agent

When building systems with LLMs, it is a good practice to start with larger models then later look to optimize and shrink.

The GPT-4.1 series is particularly well-suited for building Temporal Agents due to its strong instruction-following ability. On benchmarks like Scale’s MultiChallenge, GPT-4.1 outperforms GPT-4o by $10.5\%_{abs}$, demonstrating superior ability to maintain context, reason in-conversation, and adhere to instructions - key traits for extracting time-stamped triplets. These capabilities make it an excellent choice for prototyping agents that rely on time-aware data extraction.

Recommended development workflow

1. Prototype with GPT-4.1
   

   Maximize correctness and reduce prompt-debug time while you build out the core pipeline logic.

2. Swap to GPT-4.1-mini or GPT-4.1-nano
   

   Once prompts and logic are stable, switch to smaller variants for lower latency and cost-effective inference.

3. Distill onto GPT-4.1-mini or GPT-4.1-nano
   

   Use OpenAI's Model Distillation to train smaller models with high-quality outputs from a larger 'teacher' model such as GPT-4.1, preserving (or even improving) performance relative to GPT-4.1.

In practice, this looks like: prototype with GPT-4.1 → measure quality → step down the ladder until the trade-offs no longer meet your needs.

3.2. Building our Temporal Agent Pipeline

Before diving into the implementation details, it's useful to understand the ingestion pipeline at a high level:

1. Load transcripts
   
2. Creating a Semantic Chunker
   
3. Laying the Foundations for our Temporal Agent
   
4. Statement Extraction
   
5. Temporal Range Extraction
   
6. Creating our Triplets
   
7. Temporal Events
   
8. Defining our Temporal Agent
   
9. Entity Resolution
   
10. Invalidation Agent
    
11. Building our pipeline
    

Architecture diagram

3.2.1. Load transcripts

For the purposes of this cookbook, we have selected the "Earnings Calls Dataset" (jlh-ibm/earnings_call) which is made available under the Creative Commons Zero v1.0 license. This dataset contains a collection of 188 earnings call transcripts originating in the period 2016-2020 in relation to the NASDAQ stock market. We believe this dataset is a good choice for this cookbook as extracting information from - and subsequently querying information from - earnings call transcripts is a common problem in many financial institutions around the world.

Moreover, the often variable character of statements and topics from the same company across multiple earnings calls provides a useful vector through which to demonstrate the temporal knowledge graph concept.

Despite this dataset's focus on the financial world, we build up the Temporal Agent in a general structure, so it will be quick to adapt to similar problems in other industries such as pharmaceuticals, law, automotive, and more.

For the purposes of this cookbook we are limiting the processing to two companies - AMD and Nvidia - though in practice this pipeline can easily be scaled to any company.

Let’s start by loading the dataset from HuggingFace.

Database Set-up

Before we get to processing this data, let’s set up our database.

For convenience within a notebook format, we've chosen SQLite as our database for this implementation. In the "Prototype to Production" section, and in Appendix section A.1 "Storing and Retrieving High-Volume Graph Data" we go into more detail of considerations around different dataset choices in a production environment.

If you are running this cookbook locally, you may chose to set memory = False to save the database to storage, the default file path my_database.db will be used to store your database or you may pass your own db_path arg into make_connection.

We will set up several tables to store the following information:

• Transcripts
• Chunks
• Temporal Events
• Triplets
• Entities (including canonical mappings)

This code is abstracted behind a make_connection method which creates the new SQLite database. The details of this method can be found in the db_interface.py script in the GitHub repository for this cookbook.

3.2.2. Creating a Semantic Chunker

Before diving into buidling the Chunker class itself, we begin by defining our first data models. As is generally considered good practice when working with Python, Pydantic is used to ensure type safety and clarity in our model definitions. Pydantic provides a clean, declarative way to define data structures whilst automatically validating and parsing input data, making our data models both robust and easy to work with.

Chunk model

This is a core data model that we'll use to store individual segments of text extracted from transcripts, along with any associated metadata. As we process the transcripts by breaking them into semantically meaningful chunks, each piece will be saved as a separate Chunk.

Each Chunk contains:

• id: A unique identifier automatically generated for each chunk. This helps us identify and track chunks of text throughout
• text: A string field that contains the text content of the chunk
• metadata: A dictionary to allow for flexible metadata storage

Transcript model

As the name suggests, we will use the Transcript model to represent the full content of an earnings call transcript. It captures several key pieces of information:

• id: Analogous to Chunk, this gives us a unique identifier
• text: The full text of the transcript
• company: The name of the company that the earnings call was about
• date: The date of the earnings call
• quarter: The fiscal quarter that the earnings call was in
• chunks: A list of Chunk objects, each representing a meaningful segment of the full transcript

To ensure the date field is handled correctly, the to_datetime validator is used to convert the value to datetime format.

Chunker class

Now, we define the Chunker class to split each transcript into semantically meaningful chunks. Instead of relying on arbitrary rules like character count or line break, we apply semantic chunking to preserve more of the contextual integrity of the original transcript. This ensures that each chunk is a self-contained unit that keeps contextually linked ideas together. This is particularly helpful for downstream tasks like statement extraction, where context heavily influences accuracy.

The chunker class contains two methods:

• find_quarter

  This method attempts to extract the fiscal quarter (e.g., "Q1 2023") directly from the transcript text using a simple regular expression. In this case, this is straightforward as the data format of quarters in the transcripts is consistent and well defined.

  However, in real world scenarios, detecting the quarter reliably may require more work. Across multiple sources or document types the detailing of the quarter is likely to be different. LLMs are great tools to help alleviate this issue. Try using GPT-4.1-mini with a prompt specifically to extract the quarter given wider context from the document.

• generate_transcripts_and_chunks

  This is the core method that takes in a dataset (as an iterable of dictionaries) and returns a list of Transcript objects each populated with semantically derived Chunks. It performs the following steps:

  1. Transcript creation: Initializes Transcript objects using the provided text, company, and date fields
  2. Filtering: Uses the SemanticChunker from chonkie along with OpenAI's text-embedding-3-small model to split the transcript into logical segments
  3. Chunk assignment: Wraps each semantic segment into a Chunk model, attaching relevant metadata like start and end indices

The chunker falls in to this part of our pipeline:

Alternatively, we can load just the AMD and NVDA pre-chunked transcripts from pre-processed files in transcripts/

Now we can inspect a couple of chunks:

With this, we have successfully split our transcripts into semantically sectioned chunks. We can now move onto the next steps in our pipeline.

3.2.3. Laying the Foundations for our Temporal Agent

Before we move onto defining the TemporalAgent class, we will first define the prompts and data models that are needed for it to function.

Formalizing our label definitions

For our temporal agent to be able to accurately extract the statement and temporal types we need to provide it with sufficiently detailed and specific context. For convenience, we define these within a structured format below.

Each label contains three crucial pieces of information that we will later pass to our LLMs in prompts.

• definition
  

  Provides a concise description of what the label represents. It establishes the conceptual boundaries of the statement or temporal type and ensures consistency in interpretation across examples.

• date_handling_guidance
  

  Explains how to interpret the temporal validity of a statement associated with the label. It describes how the valid_at and invalid_at dates should be derived when processing instances of that label.

• date_handling_examples
  

  Includes illustrative examples of how real-world statements would be labelled and temporally annotated under this label. These will be used as few-shot examples to the LLMs downstream.

3.2.4. Statement Extraction

"Statement Extraction" refers to the process of splitting our semantic chunks into the smallest possible "atomic" facts. Within our Temporal Agent, this is achieved by:

1. Finding every standalone, declarative claim
   

   Extract statements that can stand on their own as complete subject-predicate-object expressions without relying on surrounding context.

2. Ensuring atomicity
   

   Break down complex or compound sentences into minimal, indivisible factual units, each expressing a single relationship.

3. Resolving references
   

   Replace pronouns or abstract references (e.g., "he" or "The Company") with specific entities (e.g., "John Smith", "AMD") using the main subject for disambiguation.

4. Preserving temporal and quantitative precision
   

   Retain explicit dates, durations, and quantities to anchor each fact precisely in time and scale.

5. Labelling each extracted statement
   

   Every statement is annotated with a StatementType and a TemporalType.

Temporal Types

The TemporalType enum provides a standardized set of temporal categories that make it easier to classify and work with statements extracted from earnings call transcripts.

Each category captures a different kind of temporal reference:

• Atemporal: Statements that are universally true and invariant over time (e.g., “The speed of light in a vacuum is ≈3×10⁸ m s⁻¹.”).
• Static: Statements that became true at a specific point in time and remain unchanged thereafter (e.g., “Person YY was CEO of Company XX on October 23rd, 2014.”).
• Dynamic: Statements that may change over time and require temporal context to interpret accurately (e.g., “Person YY is CEO of Company XX.”).

Statement Types

Similarly, the StatementType enum classifies the nature of each extracted statement, capturing its epistemic characteristics.

• Fact: A statement that asserts a verifiable claim considered true at the time it was made. However, it may later be superseded or contradicted by other facts (e.g., updated information or corrections).
• Opinion: A subjective statement reflecting a speaker’s belief, sentiment, or judgment. By nature, opinions are considered temporally true at the moment they are expressed.
• Prediction: A forward-looking or hypothetical statement about a potential future event or outcome. Temporally, a prediction is assumed to hold true from the time of utterance until the conclusion of the inferred prediction window.

Raw Statement

The RawStatement model represents an individual statement extracted by an LLM, annotated with both its semantic type (StatementType) and temporal classification (TemporalType). These raw statements serve as intermediate representations and are intended to be transformed into TemporalEvent objects in later processing stages.

Core fields:

• statement: The textual content of the extracted statement
• statement_type: The type of statement (Fact, Opinion, Prediction), based on the StatementType enum
• temporal_type: The temporal classification of the statement (Static, Dynamic, Atemporal), drawn from the TemporalType enum

The model includes field-level validators to ensure that all type annotations conform to their respective enums, providing a layer of robustness against invalid input.

The companion model RawStatementList contains the output of the statement extraction step: a list of RawStatement instances.

Statement Extraction Prompt

This is the core prompt that powers our Temporal Agent's ability to extract and label atomic statements. It is written in Jinja allowing us to modularly compose dynamic inputs without rewriting the core logic.

Anatomy of the prompt

1. Set up the extraction task
   

   We instruct the assistant to behave like a domain expert in finance and clearly define the two subtasks: (i) extracting atomic, declarative statements, and (ii) labelling each with a statement_type and a temporal_type.

2. Enforces strict extraction guidelines
   

   The rules for extraction help to enforce consistency and clarity. Statements must:

   • Be structured as clean subject-predicate-object triplets
   • Be self-contained and context-independent
   • Resolve co-references (e.g., "he" → "John Smith")
   • Include temporal/quantitative qualifiers where present
   • Be split when multiple events or temporalities are described
3. Supports plug-and-play definitions
   

   The {% if definitions %} block makes it easy to inject structured definitions such as statement categories, temporal types, and domain-specific terms.

4. Includes few-shot examples
   

   We provide an annotated example chunk and the corresponding JSON output to demonstrate to the model how it should behave.

3.2.5. Temporal Range Extraction

Raw temporal range

The RawTemporalRange model holds the raw extraction of valid_at and invalid_at date strings for a statement. These both use the date-time supported string property.

• valid_at represents the start of the validity period for a statement
• invalid_at represents the end of the validity period for a statement

Temporal validity range

While the RawTemporalRange model preserves the originally extracted date strings, the TemporalValidityRange model transforms these into standardized datetime objects for downstream processing.

It parses the raw valid_at and invalid_at values, converting them from strings into timezone-aware datetime instances. This is handled through a field-level validator.

Date extraction prompt

Let's now create the prompt that guides our Temporal Agent in accurately determining the temporal validity of statements.

Anatomy of the prompt

This prompt helps the Temporal Agent precisely understand and extract temporal validity ranges.

1. Clearly Defines the Extraction Task
   

   The prompt instructs our model to determine when a statement became true (valid_at) and optionally when it stopped being true (invalid_at).

2. Uses Contextual Guidance
   

   By dynamically incorporating {{ inputs.temporal_type }} and {{ inputs.statement_type }}, the prompt guides the model in interpreting temporal nuances based on the nature of each statement (like distinguishing facts from predictions or static from dynamic contexts).

3. Ensures Consistency with Clear Formatting Rules
   

   To maintain clarity and consistency, the prompt requires all dates to be converted into standardized ISO 8601 date-time formats, normalized to UTC. It explicitly anchors relative expressions (like "last quarter") to known publication dates, making temporal information precise and reliable.

4. Aligns with Business Reporting Cycles
   

   Recognizing the practical need for quarter-based reasoning common in business and financial contexts, the prompt can interpret and calculate temporal ranges based on business quarters, minimizing ambiguity.

5. Adapts to Statement Types for Semantic Accuracy
   

   Specific rules ensure the semantic integrity of statements—for example, opinions might only have a start date (valid_at) reflecting the moment they were expressed, while predictions will clearly define their forecast window using an end date (invalid_at).

3.2.6. Creating our Triplets

We will now build up the definitions and prompts to create the our triplets. As discussed above, these are a combination of:

• Subject - the entity you are talking about
• Predicate - the type of relationship or property
• Object - the value or other entity that the subject is connected to

Let's start with our predicate.

Predicate

The Predicate enum provides a standard set of predicates that clearly describe relationships extracted from text.

We've defined the set of predicates below to be appropriate for earnings call transcripts. Here are some examples for how each of these predicates could fit into a triplet in our knowledge graph: Here are more anonymized, generalized examples following your template:

• IS_A: [Company ABC]-[IS_A]-[Software Provider]
• HAS_A: [Corporation XYZ]-[HAS_A]-[Innovation Division]
• LOCATED_IN: [Factory 123]-[LOCATED_IN]-[Germany]
• HOLDS_ROLE: [Jane Doe]-[HOLDS_ROLE]-[CEO at Company LMN]
• PRODUCES: [Company DEF]-[PRODUCES]-[Smartphone Model X]
• SELLS: [Retailer 789]-[SELLS]-[Furniture]
• LAUNCHED: [Company UVW]-[LAUNCHED]-[New Subscription Service]
• DEVELOPED: [Startup GHI]-[DEVELOPED]-[Cloud-Based Tool]
• ADOPTED_BY: [New Technology]-[ADOPTED_BY]-[Industry ABC]
• INVESTS_IN: [Investment Firm JKL]-[INVESTS_IN]-[Clean Energy Startups]
• COLLABORATES_WITH: [Company PQR]-[COLLABORATES_WITH]-[University XYZ]
• SUPPLIES: [Manufacturer STU]-[SUPPLIES]-[Auto Components to Company VWX]
• HAS_REVENUE: [Corporation LMN]-[HAS_REVENUE]-[€500 Million]
• INCREASED: [Company YZA]-[INCREASED]-[Market Share]
• DECREASED: [Firm BCD]-[DECREASED]-[Operating Expenses]
• RESULTED_IN: [Cost Reduction Initiative]-[RESULTED_IN]-[Improved Profit Margins]
• TARGETS: [Product Launch Campaign]-[TARGETS]-[Millennial Consumers]
• PART_OF: [Subsidiary EFG]-[PART_OF]-[Parent Corporation HIJ]
• DISCONTINUED: [Company KLM]-[DISCONTINUED]-[Legacy Product Line]
• SECURED: [Startup NOP]-[SECURED]-[Series B Funding]

We also assign a definition to each predicate, which we will then pass to the extraction prompt downstream.

Defining your own predicates

When working with different data sources, you'll want to define your own predicates that are specific to your use case.

To define your own predicates:

1. First, run your pipeline with PREDICATE_DEFINITIONS = {} on a representative sample of your documents. This initial run will derive a noisy graph with many non-standardized and overlapping predicates
2. Next, drop some of your intial results into ChatGPT or manually review them to merge similar predicate classes. This process helps to eliminate duplicates such as IS_CEO and IS_CEO_OF
3. Finally, carefully review and refine this list of predicates to ensure clarity and precision. These finalized predicate definitions will then guide your extraction process and ensure a consistent extraction pipeline

Raw triplet

With predicates now well-defined, we can begin building up the data models for our triplets.

The RawTriplet model represents a basic subject-predicate-object relationship that is extracted directly from textual data. This serves as a precursor for the more detailed triplet representation in Triplet which we introduce later.

Core fields:

• subject_name: The textual representation of the subject entity
• subject_id: Numeric identifier for the subject entity
• predicate: The relationship type, specified by the Predicate enum
• object_name: The textual representation of the object entity
• object_id: Numeric identifier for the object entity
• value: Numeric value associated to relationship, may be None e.g. Company -> HAS_A -> Revenue with value='$100 mill'

Triplet

The Triplet model extends the RawTriplet by incorporating unique identifiers and optionally linking each triplet to a specific event. These identifiers help with integration into structured knowledge bases like our temporal knowledge graph.

RawEntity

The RawEntity model represents an Entity as extracted from the Statement. This serves as a precursor for the more detailed triplet representation in Entity which we introduce next.

Core fields:

• entity_idx: An integer to differentiate extracted entites from the statement (links to RawTriplet)
• name: The name of the entity extracted e.g. AMD
• type: The type of entity extracted e.g. Company
• description: The textual description of the entity e.g. Technology company know for manufacturing semiconductors

Entity

The Entity model extends the RawEntity by incorporating unique identifiers and optionally linking each entity to a specific event. Additionally, it contains resolved_id which will be populated during entity resolution with the canonical entity's id to remove duplicate naming of entities in the database. These updated identifiers help with integration and linking of entities to events and triplets .

Raw extraction

Both RawTriplet and RawEntity are extracted at the same time per Statement to reduce LLM calls and to allow easy referencing of Entities through Triplets.

Triplet Extraction Prompt

The prompt below guides our Temporal Agent to effectively extract triplets and entities from provided statements.

Anatomy of the prompt

• Avoids temporal details
  

  The agent is specifically instructed to ignore temporal relationships, as these are captured separately within the TemporalValidityRange. Defined Predicates are deliberately designed to be time-neutral—for instance, HAS_A covers both present (HAS_A) and past (HAD_A) contexts.

• Maintains structured outputs
  

  The prompt yields structured RawExtraction outputs, supported by detailed examples that clearly illustrate:

  • How to extract information from a given Statement
  • How to link Entities with corresponding Triplets
  • How to handle extracted values
  • How to manage multiple Triplets involving the same Entity

3.2.7. Temporal Event

The TemporalEvent model brings together the Statement and all related information into one handy class. It's a primary output of the TemporalAgent and plays an important role within the InvalidationAgent.

Main fields include:

• id: A unique identifier for the event
• chunk_id: Points to the specific Chunk associated with the event
• statement: The specific RawStatement extracted from the Chunk detailing a relationship or event
• embedding: A representation of the statement used by the InvalidationAgent to gauge event similarity
• triplets: Unique identifiers for the individual Triplets extracted from the Statement
• valid_at: Timestamp indicating when the event becomes valid
• invalid_at: Timestamp indicating when the event becomes invalid
• temporal_type: Describes temporal characteristics from the RawStatement
• statement_type: Categorizes the statement according to the original RawStatement
• created_at: Date the event was first created.
• expired_at: Date the event was marked invalid (set to created_at if invalid_at is already set when building the TemporalEvent)
• invalidated_by: ID of the TemporalEvent responsible for invalidating this event, if applicable

3.2.8. Defining our Temporal Agent

Now we arrive at a central point in our pipeline: The TemporalAgent class. This brings together the steps we've built up above - chunking, data models, and prompts. Let's take a closer look at how this works.

The core function, extract_transcript_events, handles all key processes:

1. It extracts a RawStatement from each Chunk.
2. From each RawStatement, it identifies the TemporalValidityRange along with lists of related Triplet and Entity objects.
3. Finally, it bundles all this information neatly into a TemporalEvent for each RawStatement.

Here's what you'll get:

• transcript: The transcript currently being analyzed.
• all_events: A comprehensive list of all generated TemporalEvent objects.
• all_triplets: A complete collection of Triplet objects extracted across all events.
• all_entities: A detailed list of all Entity objects pulled from the events, which will be further refined in subsequent steps.

The diagram below visualizes this portion of our pipeline:

3.2.9. Entity Resolution

Before diving into Temporal Invalidation, we need to first tackle entity resolution. This process is crucial to ensure that each real-world entity has a single, authoritative representation, eliminating duplicates and maintaining data consistency. For instance, AMD and Advanced Micro Devices clearly refer to the same entity, so they should be represented under a unified canonical entity.

Here's our approach to entity resolution:

• We use the EntityResolution class to batch entities by type (Entity.type), which helps us make context-specific comparisons—like distinguishing companies from individuals.

• To address noisy data effectively, we leverage RapidFuzz to cluster entities based on name similarity. This method involves a simple, case-insensitive, punctuation-free comparison using a partial match ratio, allowing tolerance for minor typos and substring matches.

• Within each fuzzy-matched cluster, we select the medoid—the entity most representative of the cluster based on overall similarity. This prevents bias toward the most frequently occurring or earliest listed entity. The medoid then serves as the initial canonical entity, providing a semantically meaningful representation of the group.

• Before adding a new canonical entity, we cross-check the medoid against existing canonicals, considering both fuzzy matching and acronyms. For example, Advanced Micro Devices Inc. may yield AMDI, closely matching the acronym AMD. This step helps prevent unnecessary creation of duplicate canonical entities.

• If a global match isn't found, the medoid becomes a new canonical entity, with all entities in the cluster linked to it via a resolved ID.

• Finally, we perform an additional safeguard check to resolve potential acronym duplication across all canonical entities, ensuring thorough cleanup.

To further enhance entity resolution, you could consider advanced techniques such as:

• Using embedding-based similarity on Entity.description alongside Entity.name, improving disambiguation beyond simple text similarity.
• Employing a large language model (LLM) to intelligently group entities under their canonical forms, enhancing accuracy through semantic understanding.

3.2.10. Invalidation agent

Understanding the Invalidation Process

To effectively invalidate temporal events, the agent performs checks in both directions:

1. Incoming vs. Existing: Are incoming events invalidated by events already present?
2. Existing vs. Incoming: Are current events invalidated by the new incoming events?

This bi-directional assessment results in a clear True/False decision.

Event Invalidation Prompt

The prompt has three key components:

1. Task Setup
   Defines two roles—primary and secondary—for event comparison. The assessment checks if the primary event is invalidated by the secondary event.
2. Guidelines
   Provides clear criteria on interpreting temporal metadata. Importantly, invalidation must rely solely on the relationships explicitly stated between events. External information cannot influence the decision.
3. Event Information
   Both events (primary and secondary) include timestamp details (valid_at and invalid_at) along with semantic context through either Statement, Triplet, or both. This context ensures accurate and relevant comparisons.

Requirements to be compared for Invalidation

We can only invalidate dynamic facts that haven't been marked invalid yet. These facts serve as our primary events, while potential candidates for invalidation are our secondary events. To streamline the invalidation process, consider these guidelines when evaluating secondary events:

1. Must be a FACT type and not Atemporal
2. Share at least one canonical entity at the triplet level
3. Belong to the same semantic predicate group at the triplet level (defined below)
4. Temporally overlap and be currently ongoing
5. Have a statement cosine similarity above the threshold (currently set to 0.5)
6. The similarity threshold (0.5) helps us filter noise effectively by selecting only the top_k most relevant results. Low-level semantic similarities are acceptable since our goal is refining the data sent to the LLM for further assessment

When invalidation occurs, we annotate the affected events with expired_at and invalidated_by to clearly indicate cause-and-effect relationships.

When we put all of this together, the workflow for our InvalidationAgent looks like this:

1. Temporal Range Detection
   

   We start by identifying when events happen with get_incoming_temporal_bounds(). This function checks the event's valid_at and, if it's dynamic, its invalid_at. Atemporal events aren't included here.

2. Temporal Event Selection
   

   We use select_events_temporally() to filter events by:

   • Checking if they're static or dynamic.
   • Determining if their time ranges overlap with our incoming event.
   • Handling dynamic events carefully, especially "ongoing" ones without an invalid_at, or events with various overlaps.
3. Embedding Similarity Filtering
   

   Then, filter_by_embedding_similarity() compares events based on semantic similarity:

   • It calculates cosine similarity between embeddings.
   • Events below a similarity threshold (_similarity_threshold = 0.5) are filtered out.
   • We keep only the top-K most similar events (_top_k = 10).
4. Combining Temporal and Semantic Filters
   

   With select_temporally_relevant_events_for_invalidation(), we:

   • Apply temporal filters first.
   • Then apply embedding similarity filters.
   • This gives us a refined list of events most likely interacting or conflicting with the incoming one.
5. Event Invalidation Decision (LLM-based)
   

   The LLM-based invalidation_step() (powered by GPT-4.1-mini) determines whether the incoming event invalidates another event:

   • If it does, we update:
     • invalid_at to match the secondary event's valid_at.
     • expired_at with the current timestamp.
     • invalidated_by with the ID of the secondary event.
6. Bidirectional Event Check
   

   We use bi_directional_event_invalidation() to check:

   • If the incoming event invalidates existing events.
   • If existing, later events invalidate the incoming event, especially if the incoming one is dynamic and currently valid.
7. Deduplication Logic
   

   Lastly, resolve_duplicate_invalidations() ensures clean invalidation:

   • It allows only one invalidation per event.
   • Picks the earliest invalidation time to avoid conflicts.
   • This helps manage batch processing effectively.

The invalidation below represents this part of our pipeline:

We can create a batch processing function for invalidation for a set of Temporal Events. This is where we filter our Statements to type FACT before passing into the invalidation agent to process.

3.2.11. Putting it all together

Now that we have built out each individual component of the Temporal Knowledge Graph workflow, we can integrate them into a cohesive workflow.

Given a chunked transcript, the Temporal Agent sequentially processes each chunk, initially extracting relevant statements. These statements are then classified and enriched through subsequent extraction phases, resulting in Temporal Events, structured Triplets, and identified Entities.

The extracted Entities are cross-referenced with existing records in the database, ensuring accurate resolution and avoiding redundancy. Following entity resolution, the Dynamic Facts undergo validation via the Invalidation Agent to verify temporal consistency and validity.

After successful processing and validation, the refined data is systematically stored into their respective tables within the SQLite database, maintaining an organized and temporally accurate knowledge graph.

To help visually ground the code presented below, we can look again at the pipeline diagram:

We can then ingest the rest of the Transcripts. Note that this code has not been optimised to be production ready and on average takes 2-5 mins per Transcript. This bulk ingestion using the data in /transcripts (~30 files) will take up to 2 hours to run. Optimizing this is a critical step in scaling to production. We outline some methods you can use to approach this in the Appendix in A.3 "Implementing Concurrency in the Ingestion Pipeline", including batch chunking, entity clustering, and more.

Note: Running the below cell for all transcripts in this dataset can take approximately 1 hour

We recommend loading the pre-processed AMD and NVDA data from file by creating a new SQLite connection using the code below. This will create the database needed for building the graph and retriever.

You can find this data on HuggingFace.

3.3. Knowledge Graphs

3.3.1 Building our Knowledge Graph with NetworkX

When constructing the knowledge graph, canonical entity identifiers derived from triplets ensure accurate mapping of entity names, allowing storage of detailed temporal metadata directly on edges. Specifically, the implementation utilizes attributes:

• valid_at, invalid_at, and temporal_type for Temporal Validity, representing real-world accuracy at specific historical moments—critical for analysis of historical facts.
• Optionally, attributes created_at and expired_at may also be used for Transactional Validity, enabling audit trails and source attribution by tracking when information was recorded, updated, or corrected.

Transactional validity is particularly beneficial in scenarios such as:

• Finance: Determining the accepted financial facts about Company X’s balance sheet on a specific historical date, based on contemporaneously accepted knowledge.
• Law: Identifying applicable legal frameworks as understood at a contract signing date, or compliance obligations recognized at past dates.
• Journalism: Assessing if previously reported information has become outdated, ensuring press releases and reporting remain accurate and credible over time.

3.3.2 NetworkX versus Neo4j in Production

To effectively implement and utilize the knowledge graph we utilise NetworkX for the purposes of this cookbook for several reasons.

1. Python integration: NetworkX seamlessly integrates with Python, facilitating rapid prototyping and iterative development
2. Ease of setup: It requires minimal initial setup, not requiring a client-server setup featured in alternatives. This makes it ideal for users who wish to run this cookbook themselves
3. Compatibility with In-Memory Databases: NetworkX can efficiently manage graphs with fewer than c.100,000 nodes, which is appropriate for this cookbook's data scale

However, it should be noted that NetworkX lacks built-in data persistence and is therefore not typically recommended for production builds.

For production builds, Neo4j emerges as a more optimal choice due to a wider set of production-centric features, including:

• Native Graph Storage and Processing: Optimized for graph data with high-performance and efficient handling
• Optimized Query Engine: Leverages the Cypher query language, explicitly designed for efficient graph traversal
• Scalability and Persistence: Effectively manages extensive graph datasets, ensuring data persistence, reliability, and durability
• Production Tooling: Offers integrated tooling such as Neo4j Bloom for vislualization and Neo4j Browser for exploration, enhancing user interaction and analysis
• Advanced Access Control: Provides granular security options to control data access

3.4. Evaluation and Suggested Feature Additions

The approach presented above offers a foundational implementation of a Temporal Agent for knowledge graph construction. However, it does not fully address complexities or all possible edge cases encountered in real-world applications. Below, we outline several possible enhancements that could be used to further improve the robustness and applicability of this implementation. In the later "Prototype to Production" section, we expand on these enhancements by suggesting additional considerations essential for deploying such agents effectively in production environments. Further details on scaling to production are included in the Appendix.

3.4.1. Temporal Agent

Statement Extraction and Temporal Events

Duplicate Temporal Events

In this cookbook, the Temporal Agent does not identify or merge duplicate Temporal Events arising from statements referring to the same event, especially when originating from different sources. These events are saved separately rather than unified into a single, consolidated event.

Static and Dynamic Representation

There's an opportunity to enrich the dataset by consistently capturing both Static and Dynamic representations of events, even when explicit statements aren't available.

For Dynamic events without corresponding Static statements, creating explicit Static entries marking the start (valid_at) and end (invalid_at) can enhance temporal clarity, particularly for the purposes of retrieval tasks.

Conversely, Static events lacking Dynamic counterparts can have Dynamic relationships inferred, though this would require careful checks for potential invalidation within statement cohorts.

Date Extraction

The implementation in this cookbook does not explictly record assumptions made during date disambiguation.

In the absence of an explicit publication date, the present date is used implicitly as a reference. For some workflows, this assumption may have to be changed to meet the needs of the end users.

Abstract dates (e.g., "until next year") are resolved into explicit dates, however the vagueness is not represented in the stored data structure. The inclusion of more granular metadata can capture more abstract date ranges:

temporal_event = {
  "summary": "The event ran from April to September",
  "label": "dynamic",
  "valid_at": {
      "date": "2025-04-01",
      "literal": False,
      "abstract_date": "2025-04"
  },
  "invalid_at": {
      "date": "2025-09-30",
      "literal": False,
      "abstract_date": "2025-09"
  }
}

This structure permits the explicit representation of both literal and abstract date interpretations.

Triplet Extraction

There are several possible avenues for improving the Triplet Extraction presented in this cookbook. These include:

• Utilising a larger model and optimizing the extraction prompts further
• Running the extraction process multiple times and consolidating results via e.g., a modal pooling mechanism to improve the accuracy and confidence in a prediction
• Incorporating entity extraction tools (e.g., Spacy and leveraging predefined ontologies tailored to specific use cases for improved consistency and reliability

3.4.2. Invalidation Agent

The presented Invalidation Agent does not refine temporal validity ranges, but one could extend its functionality to perform said refinement as well as intra-cohort invalidation checks to identify temporal conflicts among incoming statements.

There are also several opportunities for efficiency enhancements.

• Transitioning from individual (1:1) comparisons to omni-directional (1:many) invalidation checks would reduce the number of LLM calls required
• Applying network analysis techniques to cluster related statements could enable batching of invalidation checks. Clusters can be derived from several properties including semantic similarity, temporal proximity, or more advanced techniques. This would significantly reduce bottlenecks arising from sequential processing, which is particularly important when ingesting large volumes of data

4. Multi-Step Retrieval Over a Knowledge Graph

Simple retrieval systems can often handle straightforward "look-up" queries with a single search against a vector store or document index. In practice, though, agents deployed in real-world settings frequently need more. User questions often require LLMs to synthesise information from multiple parts of a knowledge base or across several endpoints.

The temporal knowledge graphs introduced earlier provide a natural foundation for this, explicitly encoding entities (nodes), relationships (edges), and their evolution over time.

Multi-step retrieval allows us to fully harness the capabilities of these graphs. It involves iteratively traversing the graph through a series of targeted queries, enabling the agent to gather all necessary context before forming a response.

We can see the power of multi-step retrieval below:

In this case, the initial query to the knowledge graph returned no information on some competitors’ R&D activities. Rather than failing silently, the system pivoted to an alternative source—the strategy content—and successfully located the missing information. This multi-step approach allowed it to navigate sparse data and deliver a complete response to the user.

4.1. Building our Retrieval Agent

At a high level, we will build out the following structure:

1. User question → Planner → Orchestrator
   

   A planner utilising GPT 4.1 will decompose the user's question into a small sequence of proposed graph operations. This is then passed to the orchestrator to execute

2. Tool calls to retrieve information from the Temporal Knowledge Graph
   

   Considering the user query and the plan, the Orchestrator (o4-mini) makes a series of initial tool calls to retrieve information from the knowledge graph

3. Loop until done → Generate answer
   

   The responses to the tool calls are fed back to the Orchestrator which can then decide to either make more queries to the graph or answer the user's question

4.1.1. Imports

4.1.2. (Re-)Initialise OpenAI Client

4.1.3. (Re-)Load our Temporal Knowledge Graph

4.1.4. Planner

Planning steps are incorporated in many modern LLM applications.

The explicit inclusion of a planning step improves overall performance by having the system consider the full scope of the problem before acting.

In this implementation, the plan remains static. In longer-horizon agentic pipelines, however, it's common to include mechanisms for replanning or updating the plan as the system progresses.

Broadly, planners take two forms:

1. Task-orientated (used in this cookbook)
   

   The planner outlines the concrete subtasks the downstream agentic blocks should execute. The tasks are phrased in an action-orientated sense such as "1. Extract information on R&D activities of Company IJK between 2018–2020." These planners are typically preferred when the goal is mostly deterministic and the primary risk is skipping or duplicating work.

   Example tasks where this approach is useful:

   • Law: "Extract and tabulate termination-notice periods from every master service agreement executed in FY24"
   • Finance: "Fetch every 10-K filed by S&P 500 banks for FY24, extract tier-1 capital and liquidity coverage ratios, and output a ranked table of institutions by capital adequacy"
   • Automotive: "Compile warranty-claim counts by component for Model XYZ vehicles sold in Europe since the new emissions regulation came into force"
   • Manufacturing: "Analyse downtime logs from each CNC machine for Q1 2025, classify the root-cause codes, and generate a Pareto chart of the top five failure drivers"
2. Hypothesis-orientated
   

   The plan is framed as a set of hypotheses the system can confirm, reject, or refine in response to the user's question. Each step represents a testable claim, optionally paired with suggested actions. This approach excels in open-ended research tasks where new information can significantly reshape the solution space.

   Example tasks where this approach is useful:

   • Law: "Does the supplied evidence satisfy all four prongs of the fair-use doctrine? Evaluate each prong against relevant case law"
   • Pharmaceuticals: "What emerging mRNA delivery methods could be used to target the IRS1 gene to treat obesity?"
   • Finance: "Is Bank Alpha facing a liquidity risk? Compare its LCR trend, interbank borrowing costs, and deposit-outflow and anything else you find that is interesting"

Prompting our planner

We will define two prompts (one system and one user) for the initial planner.

The most notable characteristic of our system prompt below is the use of 'persona-based' prompting. We prompt the LLM giving it a persona of an internal company expert. This helps to frame the tone of the model's response to the behaviour that we want - a direct, action-orientated task list that is fit for the financial industry.

This is then extended in the user prompt, where we prepend the user_question with information on this specific situation and how the planner should handle it.

In production settings you can super-charge this template by dynamically enriching the prompt before each call. You can inject information on the user's profile —sector, role, preferred writing style, prior conversation context—so the planner tailors its actions to their environment. You can also perform a quick “question-building” loop: have the assistant propose clarifying questions, gather the answers, and merge them back into the prompt so the planner starts with a well-scoped, information-rich request rather than a vague one.

Another flow that can work well is to allow users to view the plan and optionally edit it before it is executed. This is particularly effective when your AI system is acting in more of an assistant role. Giving domain experts such as lawyers or pharmaceutical researchers the flexibility to steer and incorporate their ideas and research directions deeper into the system often has the dual benefit of improving both system performance and end user satisfaction.

4.1.5. Function calling

OpenAI function calling (otherwise known as tools) enable models to perform specific external actions by calling predefined functions. Some of the tools provided on the OpenAI platform include:

• Code interpreter: Executes code for data analysis, math, plotting, and file manipulation
• Web search: Include data from the internet in model response generation
• File search: Search the contents of uploaded files for context
• Image generation: Generate or edit images using GPT image
• Remote MCP servers: Give the model access to new capabilities via Model Context Protocol (MCP) servers

Other cookbooks cover how to build tools for use with LLMs. In this example, we’ll develop several tools designed to efficiently explore the temporal knowledge graph and help answer the user’s question.

There are several schools of thought on tool design, and the best choice depends on the application at hand.

Fixed Tools

In this context, 'fixed' tools refer to those with a rigid, well-defined functionality. Typically, these tools accept a limited number of specific arguments and perform clearly outlined tasks. For instance, a fixed tool might execute a simple query such as "Get today's weather for the user's location." Due to their structured nature, these tools excel at performing consistent lookups or monitoring values within structured environments like ERP systems, regulatory frameworks, or dashboards. However, their rigidity limits flexibility, prompting users to often replace them with more dynamic, traditional data pipelines, particularly for continuous data streaming.

Examples of fixed tools in various industries include:

• Finance: "What's the current exchange rate from USD to EUR?"
• Pharmaceuticals: "Retrieve the known adverse effects for Drug ABC."
• Manufacturing: "What was the defect rate for batch #42?"

Free-form

Free-form tools represent the most flexible end of the tool spectrum. These tools are capable of executing complex, open-ended tasks with minimal constraints on input structure. A common example is a code interpreter, capable of handling diverse analytical tasks. Although their flexibility offers substantial advantages, they can also introduce unpredictability and can be more challenging to optimize for consistent reliability.

In industry applications, free-form tools can look like:

• Finance: "Backtest this momentum trading strategy using ETF price data over the past 10 years, and plot the Sharpe ratio distribution."
• Automotive: "Given this raw telemetry log, identify patterns that indicate early brake failure and simulate outcomes under various terrain conditions."
• Pharmaceuticals: "Create a pipeline that filters for statistically significant gene upregulation from this dataset, then run gene set enrichment analysis and generate a publication-ready figure."

Semi-structured Tools (used in this cookbook)

Modern agentic workflows frequently require tools that effectively balance structure and flexibility. Semi-structured tools are designed specifically to manage this middle ground. They accept inputs in moderately complex formats—such as text fragments, JSON-like arguments, or small code snippets—and often embed basic reasoning, retrieval, or decision-making capabilities. These tools are ideal when tasks are well-defined but not entirely uniform, such as when the required dataset or service is known, but the query or expected output varies.

Two common paradigms of semi-structured tools are:

• Extended Capabilities: Tools that function as specialized agents themselves, incorporating internal logic and analysis routines
• Flexible Argument Interfaces: Tools permitting the LLM to pass expressive yet structured arguments, such as detailed queries, filters, or embedded functions

Semi-structured tools are particularly valuable when:

• Delegating specific yet non-trivial tasks (like searches, transformations, or summarizations) to specialized tools
• The source data or APIs are known, but the results returned can be unpredictable

In production environments, these tools are often preferable to free-form tools, like code interpreters, due to their enhanced reliability and performance. For instance, executing complex, multi-step queries against large Neo4j knowledge graphs is more reliable and efficient using optimized Cypher queries templated within semi-structured tools rather than generating each query from scratch.

Industry applications of semi-structured tools include:

• Finance: "Extract all forward-looking risk factors from company filings for Q2 2023."
• Automotive: "Identify recurring electrical faults from maintenance logs across EV models launched after 2020."
• Pharmaceuticals: "Locate omics data supporting the hypothesis that a specific mRNA treatment effectively upregulates the IRS1 gene."

Creating tools for our retriever to use

Factual Q&A

The factual_qa tool provides an efficient way for our agent to retrieve information from our temporal knowledge graph pertaining to a particular company, topic, and date range. This will help the agent answer questions about the data such as "What were AMD's earnings in Q3 2017?"

This tool sits somewhere in the middle of the fixed and semi-structured tools we introduced earlier. This is generally quite a rigid tool in that it restricts the agent to a small number of parameters. However, the degrees of freedom in the input are large and the tool is still flexible in what information it can retrieve from the knowledge graph. This helps avoid the need for the core agent to write new queries for networkx from scratch on each query, improving accuracy and latency.

The tool has the following arguments:

• entity: This is the entity (or object with respect to triplet ontology) that the tool should retrieve information for
• start_date_range: This is the lower bound of the date range that the tool should retrieve over
• end_date_range: This is the upper bound of the date range that the tool should retrieve over
• predicate: This is the name of the predicate that the tool will connect the entity to perform a retrieval

We begin by loading the predicate definitions. We will use these to improve error tolerance in the tool, using a GPT-4.1-nano to normalize the predicate passed in the argument to a valid predicate name.

We define several helper functions for the factual QA tool.

First is _as_datetime. This tool is used to coerce the arguments that define the date range to the correct datetime format.

Next, we introduce two new data models: PredicateMatching and PredicateMatchValidation. PredicateMatching defines the output format for the GPT-4.1-nano call that matches the predicate in the function arguments to valid predicate names. PredicateMatchValidation then performs a secondary validation step to assert that this output from GPT-4.1-nano is a valid predicate name, leveraging a Pydantic field validator. This process helps to ensure that the tool runs smoothly and helps to eliminate some of the rare edge cases which would lead to an unsuccessful graph query.

Our factual QA tool can be decomposed into four steps.

1. Predicate coercion
   

   If the provided predicate is not found in the PREDICATE_DEFINITIONS dictionary, this step uses GPT-4.1-nano to coerce it into a valid predicate

2. Entity location
   

   Performs fuzzy matching to identify the corresponding entity nodes within the networkx graph

3. Edge collection
   

   Retrieves both inbound and outbound edges associated with the identified entity nodes

4. Response formatting
   

   Structures the collected information into a well-formatted response that is easy for the orchestrator to consume

Trend analysis

The trend_analysis tool is designed to compare how specific metrics or signals evolve over time—often across multiple companies and/or topics. It exposes a structured interface that lets the agent specify the time window, subject set, and target metric, then delegates the comparison logic to a specialised agent for handling this analysis. In this case we utilised o4-mini with high reasoning effort as this is a 'harder' anaysis task.

This allows us to build a highly focused and optimised pipeline for dealing with comparison-style tasks. Whilst this could be built into the core orchestrator itself, it's often more manageable to split this into specialised tools so they can be more easily swapped out or updated later without much concern for impact on the wider system.

4.1.6. Retriever

We design a simple retriever containing only a run method which encompasses the planning step and a while loop to execute each tool call that the orchestrator makes before returning a final answer.

We can now run our MultiStepRetriever.

We observe that the answer returned is detailed, and includes a detailed walkthrough of how AMD's research priorities evolved from 2016 to 2020, with references to the underlying quotes that were used to derive these answers.

We can also inspect the tools used by our MultiStepRetriever to answer this query.

Appendix section A.5. "Scaling and Productionizing our Retrieval Agent" outlines some guidelines for how one could take the Retrieval Agent we've built up to production.

4.1.7. Selecting the right model for Multi-Step Knowledge-Graph Retrieval

Multi-step retrieval agents need strong reasoning to hop through entities and relations, verify answers, and decide what to do next. Latency still matters to users, but usually less than raw accuracy. Hence, this is one of the domains where OpenAI's o3 and o4-mini reasoning models shine.

Once again, for development we recommend a “start big, then specialise” ladder:

1. Start with o3 – ensure your retrieval logic (chaining, re-ranking, fallback prompts) is sound. o3 may also be the best choice for production if your retrieval system is working with particularly complex data such as pharmaceutical or legal data. You can test this by looking at the severity of performance degradation with smaller models. If the drop off in performance is large, consider sticking with o3
2. Move to o4-mini
   • Prompt enhancement - optimise your prompts to push the performance of the o4-mini system as close to that of the full o3 model
   • Reinforcement fine-tuning (RFT) - OpenAI's Reinforcement Fine-Tuning offering enables you to fine-tune OpenAI's o-series models to improve their performance on hard reasoning tasks. With as little as ~50 golden answers you can leverage the power of reinforcement learning to fine-tune o4-mini which can help it come close or even exceed the base o3's performance on the same task
3. Fallback to GPT 4.1 when latency dominates – for cases when latency is particularly important or you've tuned your prompts well enough that performance drop-off is minimal, consider moving to the GPT 4.1 series

Why is Reinforcement Fine-Tuning powerful for long horizon, multi-step retrieval tasks?

RFT has a number of benefits over Supervised Fine-Tuning or Direct Preference Optimization for this use case.

Firstly, reinforcement fine-tuning can be performed with a far small number of examples, sometimes requiring as little as 50 training examples.

Additionally, RFT eliminates the necessity of providing labeled step-by-step trajectories. By supplying only the final correct answer, the system learns implicitly how to navigate the knowledge graph effectively. This feature is particularly valuable in real-world contexts where end users typically face time constraints and may struggle to curate the extensive sets of labeled examples (often numbering in the hundreds or thousands) required by traditional SFT methods.

4.2 Evaluating your Retrieval System

1. Human-annotated “Golden Answers”
   

   The traditional baseline for retrieval evaluation: a curated set of query → gold answer pairs, vetted by domain experts. Metrics such as precision@k or recall@k are computed by matching retrieved passages against these gold spans.

   Pros: Highest reliability, clear pass/fail thresholds, excellent for regression testing
   Cons: Expensive to create, slow to update, narrow coverage (quickly becomes stale when the knowledge base evolves)

2. Synthetically generated answers
   

   Use an LLM to generate reference answers or judgments, enabling rapid, low-cost expansion of the evaluation set. Three common pathways:

   • LLM-as-judge: Feed the query, retrieved passages, and candidate answer to a judge model that outputs a graded score or e.g., “yes / partial / no”
   • Tool-use pathway: For different question types you can either manually or synthetically generate the 'correct' tool-use pathways and score responses against this

   Pros: Fast, infinitely scalable, easier to keep pace with a dynamic application specification
   Cons: Judgement quality is typically of lower quality than expert human-annotated solutions

3. Human feedback
   

   Collect ratings directly from end-users or domain reviewers (thumbs-up/down, five-star scores, pairwise comparisons). Can be in-the-loop (model trains continuously on live feedback) or offline (periodic eval rounds).

   Pros: Captures real-world utility, surfaces edge-cases synthetic tests miss
   Cons: Noisy and subjective; requires thoughtful aggregation (e.g., ELO scoring), risk of user biases becoming incorporated in the model

Which is the best evaluation method?

There is no single best method. However, a workflow that we have found that works well on projects is:

1. Start building and iterate synthetic evaluations
2. Test with your golden human set of evaluations before deployment
3. Make it easy for end-users to annotate good and bad answers, and use this feedback to continue to develop your application over time

5. Prototype to Production

Transitioning your knowledge graph system from a proof-of-concept to a robust, production-grade pipeline requires you to address several key points:

• Storing and retrieving high-volume graph data
• Mananging and pruning datasets
• Implementing concurrency in the ingestion pipeline
• Minimizing token cost
• Scaling retrieval agents
• Safeguards

This section serves as a walkthrough of key considerations and best practices to ensure your temporally-aware knowledge graph can operate reliably in a real-world environment. A more detailed Prototype to Production Appendix section can be found towards the end of this cookbook.

1. Storing and Retrieving High-Volume Graph Data
   

   Appendix section A.1. "Storing and Retrieving High-Volume Graph Data"

   Manage scalability through thoughtful schema design, sharding, and partitioning. Clearly define entities, relationships, and ensure schema flexibility for future evolution. Use high-cardinality fields like timestamps for efficient data partitioning.

2. Temporal Validity & Versioning
   

   Appendix section A.1.2. "Temporal Validity & Versioning"

   Include temporal markers (valid_from, valid_to) for each statement. Maintain historical records non-destructively by marking outdated facts as inactive and indexing temporal fields for efficient queries.

3. Indexing & Semantic Search
   

   Appendix section A.1.3. "Indexing & Semantic Search"

   Utilize B-tree indexes for efficient temporal querying. Leverage PostgreSQL’s pgvector extension for semantic search with approximate nearest-neighbor algorithms like ivfflat, ivfpq, and hnsw to optimize query speed and memory usage.

4. Managing and Pruning Datasets
   

   Appendix section A.2. "Managing and Pruning Datasets"

   Establish TTL and archival policies for data retention based on source reliability and relevance. Implement automated archival tasks and intelligent pruning with relevance scoring to optimize graph size.

5. Concurrent Ingestion Pipeline
   

   Appendix section A.3. "Implementing Concurrency in the Ingestion Pipeline"

   Implement batch processing with separate, scalable pipeline stages for chunking, extraction, invalidation, and entity resolution. Optimize throughput and parallelism to manage ingestion bottlenecks.

6. Minimizing Token Costs
   

   Appendix section A.4. "Minimizing Token Cost"

   Use caching strategies to avoid redundant API calls. Adopt service tiers like OpenAI's flex option to reduce costs and replace expensive model queries with efficient embedding and nearest-neighbor search.

7. Scaling Retrieval Agents
   

   Appendix section A.5. "Scaling and Productionizing our Retrieval Agent"

   Use a controller and traversal workers architecture to handle multi-hop queries. Implement parallel subgraph extraction, dynamic traversal with chained reasoning, caching, and autoscaling for high performance.

8. Safeguards & Verification
   

   Appendix section A.6. "Safeguards"

   Deploy multi-layered output verification, structured logging, and monitoring to ensure data integrity and operational reliability. Track critical metrics and perform regular audits.

9. Prompt Optimization
   

   Appendix section A.7. "Prompt Optimization"

   Optimize LLM interactions with personas, few-shot prompts, chain-of-thought methods, dynamic context management, and automated A/B testing of prompt variations for continuous performance improvement.

Closing thoughts

This cookbook equips you with foundational techniques and concrete workflows to effectively build and deploy temporally-aware knowledge graphs coupled with powerful multi-hop retrieval capabilities.

Whether you're starting from a prototype or refining a production system, leveraging structured graph data with OpenAI models can unlock richer, more nuanced interactions with your data. As these technologies evolve rapidly, look out for updates in OpenAI's model lineup and keep experimenting with indexing methods and retrieval strategies to continuously enhance your knowledge-centric AI solutions.

You can easily adapt the frameworks presented in this cookbook to your respective domain by customizing the provided ontologies and refining the extraction prompts. Swapping in Neo4j as the graph database takes you well on the way to an MVP level application, providing data persistence out of the box. It also opens the door to levelling up your retriever's tools with Cypher queries.

Iterively develop your solution by making use of synthetic evals, and then test your solution against "golden" expert-human annotated solutions. Once in production, you can quickly iterate from human feedback to push your application to new heights.

Contributors

This cookbook serves as a joint collaboration between OpenAI and Tomoro.

• Alex Heald
• Douglas Adams
• Rishabh Sagar
• Danny Wigg
• Shikhar Kwatra

Appendix

Within this appendix, you'll find a more in-depth Prototype to Production section.

A. Prototype to Production

A.1. Storing and Retrieving High-Volume Graph Data

A.1.1. Data Volume & Schema Complexity

As your dataset scales to millions or even billions of nodes and edges, managing performance and maintainability becomes critical. This requires thoughtful approaches to both schema design and data partitioning:

1. Schema design for growth and change
   

   Clearly define core entity types (e.g., Person, Organization, Event) and relationships. Design the schema with versioning and flexibility in mind, enabling future schema evolution with minimal downtime.

2. Sharding & partitioning
   

   Use high-cardinality fields (such as timestamps or unique entity IDs) for partitioning to preserve query performance as data volume grows. This is particularly important for temporally-aware data. For example:

   CREATE TABLE statements (
     statement_id UUID PRIMARY KEY,
     entity_id UUID NOT NULL,
     text TEXT NOT NULL,
     valid_from TIMESTAMP NOT NULL,
     valid_to TIMESTAMP,
     status VARCHAR(16) DEFAULT 'active',
     embedding VECTOR(1536),
     ...
   ) PARTITION BY RANGE (valid_from);
   

A.1.2. Temporal Validity & Versioning

In our temporal knowledge graph, each statement includes temporal markers (e.g., valid_from, valid_to).

1. Preserve history non-destructively
   

   Avoid deleting or overwriting records. Instead mark outdated facts as inactive by setting a status (e.g., inactive).

2. Optimize for temporal access
   

   Index temporal fields (valid_from, valid_to) to support efficient querying of both current and historical states.

Example: Non-Destructive Updates

Rather than removing or overwriting a record, update its status and close its validity window:

UPDATE statements
SET status = 'inactive', valid_to = '2025-03-15T00:00:00Z'
WHERE statement_id = '...' AND entity_id = '...';

A.1.3. Indexing & Semantic Search

Temporal Indexes

To support efficient temporal queries create B-tree indexes on valid_from and valid_to. A 'B-tree' index is a tree data structure that keeps data sorted to facilitate fast lookups, range queries, and ordered scans in logarithmic time. It's the default index type in many relational databases.

CREATE INDEX ON statements (valid_from);
CREATE INDEX ON statements (valid_to);

Semantic search with pgvector

Storing vector embeddings in PostgreSQL (via the pgvector extension) enables similarity-based retrieval via semantic search. This follows a two-step process:

1. Store high-dimensional vectors that represent the semantic meaning of the text. These can be created with embedding models such as OpenAI's text-embedding-3-small and text-embedding-3-large
2. Use Approximate Nearest-Neighbour (ANN) for efficient similarity matching at scale

There are several different indexing options available in pgvector, each with different purposes. These indexing options are described in more detail, along with in-depth implementation steps in the README on the Github repository for pgvector.

Tuning parameters for vector indexing

ivfflat

• lists: Number of partitions (e.g., 100)
• probes: Number of partitions to scan at query time (e.g., 10-20), controls recall vs. latency

ivfpq

• subvectors: Number of blocks to quantize (e.g., 16)
• bits: Number of bits per block (e.g., 8)
• probes: Same as in ivfflat

hnsw

• M: Max connections per node (e.g., 16)
• ef_construction: Build-time dynamic candidate list size (e.g., 200)
• ef_search: Queyr-time candidate pool (e.g., 64-128)

Best practices

• flat for debugging or small datasets
• ivfflat when you want tunable accuracy with good speed
• ivfpq when memory efficieny is critical
• hnsw when optimizing for lowest latency on massive collections

Other vector database options in the ecosystem