Milvus Langgraph Graphrag Agent Local

Langgraph Graphrag Agent Local

image-searchvector-databasesemantic-searchmilvusembeddingsunstructured-dataquestion-answeringLLMmilvus-bootcampdeep-learningimage-recognitionimage-classificationaudio-searchPythonbootcampragadvanced_ragNLP

alph-notebooks/milvus-bootcamp / langgraph-graphrag-agent-local.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

[ ]

LangGraph GraphRAG agent with Llama 3.x and GPT4o

Let's build an Advanced RAG with a GraphRAG agent that will run a combination of Llama 3.1 and GPT4o, for Llama 3.1 we will use Ollama. The idea is that we use GPT4o for advanced tasks, like generating the Neo4j query and Llama3.1 for the rest.

Ideas

We'll combine ideas from three RAG papers into a RAG agent:

Routing: Adaptive RAG (paper). Route questions to different retrieval approaches
Fallback: Corrective RAG (paper). Fallback to web search if docs are not relevant to query
Self-correction: Self-RAG (paper). Fix answers w/ hallucinations or don’t address question

Note that this will incorperate a few general ideas for agents:

Reflection: The self-correction mechanism is a form of reflection, where the LangGraph agent reflects on its retrieval and generations
Planning: The control flow laid out in the graph is a form of planning
Tool use: Specific nodes in the control flow (e.g., web search) will use tools

Local models

LLM

Use Ollama and llama3:

ollama pull llama3.1

Env Variables

Variables needed in an .env file or loaded as variables at start:

Required:

OPENAI_API_KEY=sk-...
TAVILY_API_KEY=tvly-...
NEO4J_URI=...
NEO4J_USERNAME=...
NEO4J_PASSWORD=...

Search

Uses Tavily

[55]

True

[56]

[57]

[4]

[37]

Number of papers: 20
Number of chunks: 20

/var/folders/kv/3dw9p_ts4b114chqt9m027pc0000gn/T/ipykernel_10082/2843254102.py:28: LangChainDeprecationWarning: Default values for HuggingFaceEmbeddings.model_name were deprecated in LangChain 0.2.16 and will be removed in 0.4.0. Explicitly pass a model_name to the HuggingFaceEmbeddings constructor instead.
  embedding=HuggingFaceEmbeddings(),
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

[38]

[9]

Graph documents: 20
Nodes from 1st graph doc:[Node(id='Prompt Design And Engineering', type='Topic'), Node(id='Large Language Models', type='Topic'), Node(id='Chain-Of-Thought', type='Topic'), Node(id='Reflection', type='Topic'), Node(id='Llm-Based Agents', type='Topic'), Node(id='Tools For Prompt Engineers', type='Topic'), Node(id='This Paper', type='Paper', properties={'summary': 'This paper introduces core concepts, advanced techniques like Chain-of-Thought and Reflection, and the principles behind building LLM-based agents. It also provides a survey of tools for prompt engineers.'})]
Relationships from 1st graph doc:[Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Prompt Design And Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Large Language Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Chain-Of-Thought', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Reflection', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Llm-Based Agents', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Tools For Prompt Engineers', type='Topic'), type='DISCUSSES')]

[39]

Document 0:
  Nodes: [Node(id='Prompt Design And Engineering', type='Topic'), Node(id='Large Language Models', type='Topic'), Node(id='Chain-Of-Thought', type='Topic'), Node(id='Reflection', type='Topic'), Node(id='Llm-Based Agents', type='Topic'), Node(id='Tools For Prompt Engineers', type='Topic'), Node(id='This Paper', type='Paper', properties={'summary': 'This paper introduces core concepts, advanced techniques like Chain-of-Thought and Reflection, and the principles behind building LLM-based agents. It also provides a survey of tools for prompt engineers.'})]
  Relationships: [Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Prompt Design And Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Large Language Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Chain-Of-Thought', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Reflection', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Llm-Based Agents', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='This Paper', type='Paper'), target=Node(id='Tools For Prompt Engineers', type='Topic'), type='DISCUSSES')]
---
Document 1:
  Nodes: [Node(id='Unlocking Reasoning Capability Of Large Language Models', type='Paper', properties={'title': 'Unlocking Reasoning Capability of Large Language Models', 'summary': 'This paper discusses two methods to enhance reasoning in large language models: prompt engineering and multi-agent discussion. It theoretically justifies multi-agent discussion from the symmetry of agents and reports empirical results on the interplay of prompts and discussion mechanisms. It also proposes a scalable discussion mechanism based on conquer and merge.'}), Node(id='Prompt Engineering', type='Topic'), Node(id='Multi-Agent Discussion', type='Topic')]
  Relationships: [Relationship(source=Node(id='Unlocking Reasoning Capability Of Large Language Models', type='Paper'), target=Node(id='Prompt Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Unlocking Reasoning Capability Of Large Language Models', type='Paper'), target=Node(id='Multi-Agent Discussion', type='Topic'), type='DISCUSSES')]
---
Document 2:
  Nodes: [Node(id='Simulation Of Human Interactions Using Llms', type='Paper', properties={'title': 'Simulation of Human Interactions using LLMs', 'summary': 'The research investigates simulations of human interactions using large language models (LLMs). It presents two simulations of believable proxies of human behavior: a two-agent negotiation and a six-agent murder mystery game.'}), Node(id='Agent-Based Modeling', type='Topic'), Node(id='Large Language Models', type='Topic'), Node(id='Chatgpt', type='Topic'), Node(id='Prompt Engineering', type='Topic'), Node(id='Park Et Al. (2023)', type='Paper')]
  Relationships: [Relationship(source=Node(id='Simulation Of Human Interactions Using Llms', type='Paper'), target=Node(id='Agent-Based Modeling', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Simulation Of Human Interactions Using Llms', type='Paper'), target=Node(id='Large Language Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Simulation Of Human Interactions Using Llms', type='Paper'), target=Node(id='Chatgpt', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Simulation Of Human Interactions Using Llms', type='Paper'), target=Node(id='Prompt Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Simulation Of Human Interactions Using Llms', type='Paper'), target=Node(id='Park Et Al. (2023)', type='Paper'), type='RELATED_TO')]
---
Document 3:
  Nodes: [Node(id='Ai Community', type='Topic'), Node(id='Artificial General Intelligence', type='Topic'), Node(id='Language Agents', type='Topic'), Node(id='Large Language Models', type='Topic'), Node(id='Agent Symbolic Learning', type='Topic'), Node(id='Paper On Agent Symbolic Learning', type='Paper', properties={'summary': 'Introduces agent symbolic learning, a framework enabling language agents to optimize themselves in a data-centric way using symbolic optimizers.'}), Node(id='Self-Evolving Agents', type='Topic')]
  Relationships: [Relationship(source=Node(id='Ai Community', type='Topic'), target=Node(id='Artificial General Intelligence', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Language Agents', type='Topic'), target=Node(id='Artificial General Intelligence', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Language Agents', type='Topic'), target=Node(id='Large Language Models', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Agent Symbolic Learning', type='Topic'), target=Node(id='Language Agents', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Paper On Agent Symbolic Learning', type='Paper'), target=Node(id='Agent Symbolic Learning', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Agent Symbolic Learning', type='Topic'), target=Node(id='Self-Evolving Agents', type='Topic'), type='RELATED_TO')]
---
Document 4:
  Nodes: [Node(id='Reprompt', type='Paper', properties={'title': 'RePrompt', 'summary': "RePrompt is a novel method that uses 'gradient descent' to optimize step-by-step instructions in the prompt of LLM agents based on chat history from interactions with LLM agents. It aims to improve performance in specific domains by optimizing the prompt."}), Node(id='Large Language Models', type='Topic'), Node(id='Automatic Prompt Engineering', type='Topic'), Node(id='Pddl Generation', type='Topic'), Node(id='Travel Planning', type='Topic')]
  Relationships: [Relationship(source=Node(id='Reprompt', type='Paper'), target=Node(id='Large Language Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Reprompt', type='Paper'), target=Node(id='Automatic Prompt Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Reprompt', type='Paper'), target=Node(id='Pddl Generation', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Reprompt', type='Paper'), target=Node(id='Travel Planning', type='Topic'), type='DISCUSSES')]
---
Document 5:
  Nodes: [Node(id='Traditional Base Station Siting Methods', type='Topic'), Node(id='Large Language Models', type='Topic'), Node(id='Prompt Engineering', type='Topic'), Node(id='Agent Engineering', type='Topic'), Node(id='Network Optimization', type='Topic'), Node(id='Artificial Intelligence As A Service', type='Topic'), Node(id='Llm-Empowered Bss Optimization Framework', type='Topic'), Node(id='Prompt-Optimized Llm', type='Topic'), Node(id='Human-In-The-Loop Llm', type='Topic'), Node(id='Llm-Empowered Autonomous Bss Agent', type='Topic'), Node(id='Cooperative Multiple Llm-Based Autonomous Bss Agents', type='Topic')]
  Relationships: [Relationship(source=Node(id='Traditional Base Station Siting Methods', type='Topic'), target=Node(id='Network Optimization', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Large Language Models', type='Topic'), target=Node(id='Network Optimization', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Prompt Engineering', type='Topic'), target=Node(id='Network Optimization', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Agent Engineering', type='Topic'), target=Node(id='Network Optimization', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Llm-Empowered Bss Optimization Framework', type='Topic'), target=Node(id='Network Optimization', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Artificial Intelligence As A Service', type='Topic'), target=Node(id='Llm-Empowered Bss Optimization Framework', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Prompt-Optimized Llm', type='Topic'), target=Node(id='Llm-Empowered Bss Optimization Framework', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Human-In-The-Loop Llm', type='Topic'), target=Node(id='Llm-Empowered Bss Optimization Framework', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Llm-Empowered Autonomous Bss Agent', type='Topic'), target=Node(id='Llm-Empowered Bss Optimization Framework', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Cooperative Multiple Llm-Based Autonomous Bss Agents', type='Topic'), target=Node(id='Llm-Empowered Bss Optimization Framework', type='Topic'), type='RELATED_TO')]
---
Document 6:
  Nodes: [Node(id='Large Language Models', type='Topic'), Node(id='Prompt Engineering', type='Topic'), Node(id='Hallucinations', type='Topic'), Node(id='Tool-Calling Agents', type='Topic'), Node(id='Paper On Prompting Strategies', type='Paper', properties={'summary': 'This paper provides a comprehensive empirical evaluation of different prompting strategies and frameworks aimed at reducing hallucinations in LLMs. Various prompting techniques are applied to a broad set of benchmark datasets to assess the accuracy and hallucination rate of each method. Additionally, the paper investigates the influence of tool-calling agents on hallucination rates in the same benchmarks.'})]
  Relationships: [Relationship(source=Node(id='Paper On Prompting Strategies', type='Paper'), target=Node(id='Large Language Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Paper On Prompting Strategies', type='Paper'), target=Node(id='Prompt Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Paper On Prompting Strategies', type='Paper'), target=Node(id='Hallucinations', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Paper On Prompting Strategies', type='Paper'), target=Node(id='Tool-Calling Agents', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Large Language Models', type='Topic'), target=Node(id='Hallucinations', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Prompt Engineering', type='Topic'), target=Node(id='Hallucinations', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Tool-Calling Agents', type='Topic'), target=Node(id='Hallucinations', type='Topic'), type='RELATED_TO')]
---
Document 7:
  Nodes: [Node(id='Instruction-Following Agents', type='Topic'), Node(id='Language Grounding', type='Topic'), Node(id='Pretrained Vision-Language Models', type='Topic'), Node(id='Embodied Agents', type='Topic'), Node(id='Model Distillation', type='Topic'), Node(id='Hindsight Experience Replay', type='Topic'), Node(id='3D Rendered Environment', type='Topic'), Node(id='Fewshot Prompting', type='Topic'), Node(id='Abstract Category Membership', type='Topic')]
  Relationships: [Relationship(source=Node(id='Instruction-Following Agents', type='Topic'), target=Node(id='Language Grounding', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Language Grounding', type='Topic'), target=Node(id='Pretrained Vision-Language Models', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Pretrained Vision-Language Models', type='Topic'), target=Node(id='Embodied Agents', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Model Distillation', type='Topic'), target=Node(id='Pretrained Vision-Language Models', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Hindsight Experience Replay', type='Topic'), target=Node(id='Pretrained Vision-Language Models', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='3D Rendered Environment', type='Topic'), target=Node(id='Embodied Agents', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Fewshot Prompting', type='Topic'), target=Node(id='Abstract Category Membership', type='Topic'), type='RELATED_TO')]
---
Document 8:
  Nodes: [Node(id='Pre-Trained And Frozen Large Language Models', type='Topic'), Node(id='Scene Rearrangement Instructions', type='Topic'), Node(id="Robot'S Visuomotor Functions", type='Topic'), Node(id='Open-Domain Natural Language', type='Topic'), Node(id="User'S Idiosyncratic Procedures", type='Topic'), Node(id='Prompt Engineering', type='Topic'), Node(id='Helper', type='Paper', properties={'title': 'HELPER: An Embodied Agent with External Memory for Human-Robot Dialogue Parsing', 'summary': 'HELPER is an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting.', 'url': 'https://helper-agent-llm.github.io'}), Node(id='Teach Benchmark', type='Topic'), Node(id='Execution From Dialog History (Edh)', type='Topic'), Node(id='Trajectory From Dialogue (Tfd)', type='Topic')]
  Relationships: [Relationship(source=Node(id='Pre-Trained And Frozen Large Language Models', type='Topic'), target=Node(id='Scene Rearrangement Instructions', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Pre-Trained And Frozen Large Language Models', type='Topic'), target=Node(id="Robot'S Visuomotor Functions", type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Open-Domain Natural Language', type='Topic'), target=Node(id="User'S Idiosyncratic Procedures", type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Helper', type='Paper'), target=Node(id='Teach Benchmark', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Helper', type='Paper'), target=Node(id='Execution From Dialog History (Edh)', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Helper', type='Paper'), target=Node(id='Trajectory From Dialogue (Tfd)', type='Topic'), type='DISCUSSES')]
---
Document 9:
  Nodes: [Node(id='Promptagent', type='Paper', properties={'title': 'PromptAgent: Autonomous Expert-Level Prompt Optimization', 'summary': 'PromptAgent is an optimization method that autonomously crafts prompts equivalent in quality to those handcrafted by experts. It views prompt optimization as a strategic planning problem and employs a principled planning algorithm, rooted in Monte Carlo tree search, to navigate the expert-level prompt space. It applies to 12 tasks spanning three practical domains: BIG-Bench Hard (BBH), domain-specific, and general NLP tasks, significantly outperforming strong baselines.'}), Node(id='Big-Bench Hard (Bbh)', type='Topic'), Node(id='Domain-Specific Nlp Tasks', type='Topic'), Node(id='General Nlp Tasks', type='Topic')]
  Relationships: [Relationship(source=Node(id='Promptagent', type='Paper'), target=Node(id='Big-Bench Hard (Bbh)', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Promptagent', type='Paper'), target=Node(id='Domain-Specific Nlp Tasks', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Promptagent', type='Paper'), target=Node(id='General Nlp Tasks', type='Topic'), type='DISCUSSES')]
---
Document 10:
  Nodes: [Node(id='Recent Advancements In Large Language Models (Llms) And Prompt Engineering', type='Topic'), Node(id='Chatbot Customization', type='Topic'), Node(id='Prompt Evaluation', type='Topic'), Node(id='Awesum', type='Paper', properties={'title': 'Awesum: A Visual Analytics System for Prompt Evaluation in Text Summarization'}), Node(id='Feature-Oriented Workflow', type='Topic'), Node(id='Text Summarization', type='Topic'), Node(id='Summary Characteristics', type='Topic'), Node(id='Prompt Comparator', type='Topic'), Node(id='Dimensionality Reduction Techniques', type='Topic'), Node(id='Feature-Oriented Evaluation Of Llm Prompts', type='Topic'), Node(id='Human-Agent Interaction', type='Topic')]
  Relationships: [Relationship(source=Node(id='Recent Advancements In Large Language Models (Llms) And Prompt Engineering', type='Topic'), target=Node(id='Chatbot Customization', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Recent Advancements In Large Language Models (Llms) And Prompt Engineering', type='Topic'), target=Node(id='Prompt Evaluation', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Awesum', type='Paper'), target=Node(id='Prompt Evaluation', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Awesum', type='Paper'), target=Node(id='Feature-Oriented Workflow', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Feature-Oriented Workflow', type='Topic'), target=Node(id='Text Summarization', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Feature-Oriented Workflow', type='Topic'), target=Node(id='Summary Characteristics', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Awesum', type='Paper'), target=Node(id='Prompt Comparator', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt Comparator', type='Topic'), target=Node(id='Dimensionality Reduction Techniques', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Feature-Oriented Evaluation Of Llm Prompts', type='Topic'), target=Node(id='Human-Agent Interaction', type='Topic'), type='RELATED_TO')]
---
Document 11:
  Nodes: [Node(id='Leveraging_Multiple_Llm_Agents', type='Topic'), Node(id='Adaptive_Team-Building_Paradigm', type='Topic'), Node(id='Captain_Agent', type='Topic'), Node(id='Paper_1', type='Paper', properties={'title': 'Leveraging multiple large language model (LLM) agents', 'summary': 'Leveraging multiple large language model (LLM) agents has shown to be a promising approach for tackling complex tasks, while the effective design of multiple agents for a particular application remains an art. It is thus intriguing to answer a critical question: Given a task, how can we build a team of LLM agents to solve it effectively? Our new adaptive team-building paradigm offers a flexible solution, realized through a novel agent design named Captain Agent. It dynamically forms and manages teams for each step of a task-solving process, utilizing nested group conversations and reflection to ensure diverse expertise and prevent stereotypical outputs, allowing for a flexible yet structured approach to problem-solving. A comprehensive evaluation across six real-world scenarios demonstrates that Captain Agent significantly outperforms existing multi-agent methods with 21.94% improvement in average accuracy, providing outstanding performance without requiring task-specific prompt engineering. Our exploration of different backbone LLM and cost analysis further shows that Captain Agent can improve the conversation quality of weak LLM and achieve competitive performance with extremely low cost, which illuminates the application of multi-agent systems.'})]
  Relationships: [Relationship(source=Node(id='Paper_1', type='Paper'), target=Node(id='Leveraging_Multiple_Llm_Agents', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Paper_1', type='Paper'), target=Node(id='Adaptive_Team-Building_Paradigm', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Paper_1', type='Paper'), target=Node(id='Captain_Agent', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Adaptive_Team-Building_Paradigm', type='Topic'), target=Node(id='Captain_Agent', type='Topic'), type='RELATED_TO')]
---
Document 12:
  Nodes: [Node(id='Drama Engine', type='Paper', properties={'title': 'Drama Engine', 'summary': 'A novel framework for agentic interaction with large language models designed for narrative purposes, adapting multi-agent system principles to create dynamic, context-aware companions.'}), Node(id='Multi-Agent Workflows', type='Topic'), Node(id='Dynamic Prompt Assembly', type='Topic'), Node(id='Model-Agnostic Design', type='Topic'), Node(id='Companion Development', type='Topic'), Node(id='Mood Systems', type='Topic'), Node(id='Automatic Context Summarising', type='Topic'), Node(id='Multi-Agent Chats', type='Topic'), Node(id='Virtual Co-Workers For Creative Writing', type='Topic'), Node(id="System'S Architecture", type='Topic'), Node(id='Prompt Assembly Process', type='Topic'), Node(id='Delegation Mechanisms', type='Topic'), Node(id='Moderation Techniques', type='Topic'), Node(id='Ethical Considerations', type='Topic'), Node(id='Future Extensions', type='Topic')]
  Relationships: [Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Multi-Agent Workflows', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Dynamic Prompt Assembly', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Model-Agnostic Design', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Companion Development', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Mood Systems', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Automatic Context Summarising', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Multi-Agent Chats', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Virtual Co-Workers For Creative Writing', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id="System'S Architecture", type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Prompt Assembly Process', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Delegation Mechanisms', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Moderation Techniques', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Ethical Considerations', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Drama Engine', type='Paper'), target=Node(id='Future Extensions', type='Topic'), type='DISCUSSES')]
---
Document 13:
  Nodes: [Node(id='Prompt Engineering', type='Topic'), Node(id='Vision-Language Models', type='Topic'), Node(id='Multimodal-To-Text Generation Models', type='Topic'), Node(id='Image-Text Matching Models', type='Topic'), Node(id='Text-To-Image Generation Models', type='Topic'), Node(id='Natural Language Processing', type='Topic'), Node(id='Vision-Language Modeling', type='Topic'), Node(id='Flamingo', type='Topic'), Node(id='Clip', type='Topic'), Node(id='Stable Diffusion', type='Topic'), Node(id='Prompt Engineering On Vision-Language Models', type='Paper', properties={'title': 'Prompt Engineering on Vision-Language Models'})]
  Relationships: [Relationship(source=Node(id='Prompt Engineering', type='Topic'), target=Node(id='Vision-Language Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt Engineering', type='Topic'), target=Node(id='Natural Language Processing', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt Engineering', type='Topic'), target=Node(id='Vision-Language Modeling', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt Engineering On Vision-Language Models', type='Paper'), target=Node(id='Prompt Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt Engineering On Vision-Language Models', type='Paper'), target=Node(id='Multimodal-To-Text Generation Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt Engineering On Vision-Language Models', type='Paper'), target=Node(id='Image-Text Matching Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt Engineering On Vision-Language Models', type='Paper'), target=Node(id='Text-To-Image Generation Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Multimodal-To-Text Generation Models', type='Topic'), target=Node(id='Flamingo', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Image-Text Matching Models', type='Topic'), target=Node(id='Clip', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Text-To-Image Generation Models', type='Topic'), target=Node(id='Stable Diffusion', type='Topic'), type='RELATED_TO')]
---
Document 14:
  Nodes: [Node(id='Recent Trends In Llms', type='Topic'), Node(id='Large Language Models As Autonomous Agents', type='Topic'), Node(id='Guidance, Navigation, And Control In Space', type='Topic'), Node(id='Kerbal Space Program Differential Games', type='Topic'), Node(id='Llm-Based Solution For Kspdg', type='Paper', properties={'title': 'LLM-based solution for KSPDG', 'summary': 'A pure LLM-based solution for the Kerbal Space Program Differential Games challenge, leveraging prompt engineering, few-shot prompting, and fine-tuning techniques to create an effective LLM-based agent that ranked 2nd in the competition.', 'url': 'https://github.com/ARCLab-MIT/kspdg'})]
  Relationships: [Relationship(source=Node(id='Recent Trends In Llms', type='Topic'), target=Node(id='Large Language Models As Autonomous Agents', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Large Language Models As Autonomous Agents', type='Topic'), target=Node(id='Guidance, Navigation, And Control In Space', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Llm-Based Solution For Kspdg', type='Paper'), target=Node(id='Kerbal Space Program Differential Games', type='Topic'), type='DISCUSSES')]
---
Document 15:
  Nodes: [Node(id='Prompt Stealing Attacks', type='Paper', properties={'title': 'Prompt Stealing Attacks', 'summary': 'The paper proposes a novel attack against large language models (LLMs) named prompt stealing attacks, which aim to steal well-designed prompts based on generated answers. The attack consists of two primary modules: the parameter extractor and the prompt reconstruction. The parameter extractor identifies the type of prompts and predicts roles or contexts used, while the prompt reconstructor generates reversed prompts similar to the original ones. The study highlights the security issues in LLMs and adds a new dimension to prompt engineering.'})]
  Relationships: [Relationship(source=Node(id='Prompt Stealing Attacks', type='Paper'), target=Node(id='Prompt Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt Stealing Attacks', type='Paper'), target=Node(id='Security Issues On Llms', type='Topic'), type='DISCUSSES')]
---
Document 16:
  Nodes: [Node(id='Promptset', type='Paper', properties={'title': 'PromptSet', 'summary': 'A novel dataset with more than 61,000 unique developer prompts used in open source Python programs, introducing the notion of a static linter for prompts.'}), Node(id='Large Language Models', type='Topic'), Node(id='Prompting', type='Topic'), Node(id='Static Linter', type='Topic')]
  Relationships: [Relationship(source=Node(id='Promptset', type='Paper'), target=Node(id='Large Language Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Promptset', type='Paper'), target=Node(id='Prompting', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Promptset', type='Paper'), target=Node(id='Static Linter', type='Topic'), type='DISCUSSES')]
---
Document 17:
  Nodes: [Node(id='Interaction With Large Language Models', type='Topic'), Node(id='Prompt', type='Topic'), Node(id='Prompt Design', type='Topic'), Node(id='Prompt Engineering', type='Topic'), Node(id='Prompt Editing Behavior', type='Topic'), Node(id='Prompt Engineering Practices', type='Topic')]
  Relationships: [Relationship(source=Node(id='Interaction With Large Language Models', type='Topic'), target=Node(id='Prompt', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Prompt', type='Topic'), target=Node(id='Prompt Design', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Prompt Design', type='Topic'), target=Node(id='Prompt Engineering', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Prompt Engineering', type='Topic'), target=Node(id='Prompt Editing Behavior', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Prompt Engineering', type='Topic'), target=Node(id='Prompt Engineering Practices', type='Topic'), type='RELATED_TO')]
---
Document 18:
  Nodes: [Node(id='Large Language Models', type='Topic'), Node(id='Security Tasks Automation', type='Topic'), Node(id='Security Operation Centers', type='Topic'), Node(id='Software Pentesting', type='Topic'), Node(id='Software Security Vulnerabilities', type='Topic'), Node(id='Owasp Benchmark Project 1.2', type='Topic'), Node(id='Sonarqube', type='Topic'), Node(id="Google'S Gemini-Pro", type='Topic'), Node(id="Openai'S Gpt-3.5-Turbo", type='Topic'), Node(id="Openai'S Gpt-4-Turbo", type='Topic')]
  Relationships: [Relationship(source=Node(id='Large Language Models', type='Topic'), target=Node(id='Security Tasks Automation', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Security Tasks Automation', type='Topic'), target=Node(id='Security Operation Centers', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Large Language Models', type='Topic'), target=Node(id='Software Pentesting', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Software Pentesting', type='Topic'), target=Node(id='Software Security Vulnerabilities', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Owasp Benchmark Project 1.2', type='Topic'), target=Node(id='Software Security Vulnerabilities', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id='Sonarqube', type='Topic'), target=Node(id='Software Security Vulnerabilities', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id="Google'S Gemini-Pro", type='Topic'), target=Node(id='Large Language Models', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id="Openai'S Gpt-3.5-Turbo", type='Topic'), target=Node(id='Large Language Models', type='Topic'), type='RELATED_TO'), Relationship(source=Node(id="Openai'S Gpt-4-Turbo", type='Topic'), target=Node(id='Large Language Models', type='Topic'), type='RELATED_TO')]
---
Document 19:
  Nodes: [Node(id='Visual Navigation Using Large Language Models', type='Paper', properties={'summary': 'Recent efforts to enable visual navigation using large language models have mainly focused on developing complex prompt systems. These systems incorporate instructions, observations, and history into massive text prompts, which are then combined with pre-trained large language models to facilitate visual navigation. In contrast, our approach aims to fine-tune large language models for visual navigation without extensive prompt engineering. Our design involves a simple text prompt, current observations, and a history collector model that gathers information from previous observations as input. For output, our design provides a probability distribution of possible actions that the agent can take during navigation. We train our model using human demonstrations and collision signals from the Habitat-Matterport 3D Dataset (HM3D). Experimental results demonstrate that our method outperforms state-of-the-art behavior cloning methods and effectively reduces collision rates.'}), Node(id='Large Language Models', type='Topic'), Node(id='Visual Navigation', type='Topic'), Node(id='Prompt Engineering', type='Topic'), Node(id='Habitat-Matterport 3D Dataset (Hm3D)', type='Topic')]
  Relationships: [Relationship(source=Node(id='Visual Navigation Using Large Language Models', type='Paper'), target=Node(id='Large Language Models', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Visual Navigation Using Large Language Models', type='Paper'), target=Node(id='Visual Navigation', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Visual Navigation Using Large Language Models', type='Paper'), target=Node(id='Prompt Engineering', type='Topic'), type='DISCUSSES'), Relationship(source=Node(id='Visual Navigation Using Large Language Models', type='Paper'), target=Node(id='Habitat-Matterport 3D Dataset (Hm3D)', type='Topic'), type='DISCUSSES')]
---

[40]

Is our answer relevant to the question asked: {'score': 'yes'}

[41]

Yes, we have articles that talk about Prompt Engineering. The context includes papers discussing core concepts, advanced techniques, and principles behind building Large Language Model (LLM)-based agents, as well as a survey of tools for prompt engineers. Additionally, there are studies on prompt engineering technologies to improve the quality of model outputs and proposed attacks against LLMs that aim to steal well-designed prompts.

[42]



> Entering new GraphCypherQAChain chain...

Failed to write data to connection ResolvedIPv4Address(('35.241.237.34', 7687)) (ResolvedIPv4Address(('35.241.237.34', 7687)))
Failed to write data to connection IPv4Address(('a154a864.databases.neo4j.io', 7687)) (ResolvedIPv4Address(('35.241.237.34', 7687)))

Generated Cypher:
cypher
MATCH (p:Paper)
WHERE toLower(p.title) CONTAINS toLower("Multi-Agent")
RETURN p.title, p.id


> Finished chain.
{'query': 'What paper talks about Multi-Agent?', 'result': [{'p.title': 'Multi-Agent Assistant Code Generation (AgentCoder)', 'p.id': 'Multi-Agent Assistant Code Generation (Agentcoder)'}, {'p.title': 'Framework for Automatically Generating Process Models with Multi-Agent Orchestration (MAO)', 'p.id': 'This Article'}, {'p.title': 'Collaborative Multi-Agent, Multi-Reasoning-Path (CoMM) Prompting Framework', 'p.id': 'This Work'}], 'intermediate_steps': [{'query': 'cypher\nMATCH (p:Paper)\nWHERE toLower(p.title) CONTAINS toLower("Multi-Agent")\nRETURN p.title, p.id\n'}]}

[43]

[44]

[Document(metadata={'pk': 453568971862704139, 'summary': 'Leveraging multiple large language model (LLM) agents has shown to be a\npromising approach for tackling complex tasks, while the effective design of\nmultiple agents for a particular application remains an art. It is thus\nintriguing to answer a critical question: Given a task, how can we build a team\nof LLM agents to solve it effectively? Our new adaptive team-building paradigm\noffers a flexible solution, realized through a novel agent design named Captain\nAgent. It dynamically forms and manages teams for each step of a task-solving\nprocess, utilizing nested group conversations and reflection to ensure diverse\nexpertise and prevent stereotypical outputs, allowing for a flexible yet\nstructured approach to problem-solving. A comprehensive evaluation across six\nreal-world scenarios demonstrates that Captain Agent significantly outperforms\nexisting multi-agent methods with 21.94% improvement in average accuracy,\nproviding outstanding performance without requiring task-specific prompt\nengineering. Our exploration of different backbone LLM and cost analysis\nfurther shows that Captain Agent can improve the conversation quality of weak\nLLM and achieve competitive performance with extremely low cost, which\nilluminates the application of multi-agent systems.', 'title': 'Adaptive In-conversation Team Building for Language Model Agents', 'url': 'http://arxiv.org/abs/2405.19425v2'}, page_content='Leveraging multiple large language model (LLM) agents has shown to be a\npromising approach for tackling complex tasks, while the effective design of\nmultiple agents for a particular application remains an art. It is thus\nintriguing to answer a critical question: Given a task, how can we build a team\nof LLM agents to solve it effectively? Our new adaptive team-building paradigm\noffers a flexible solution, realized through a novel agent design named Captain\nAgent. It dynamically forms and manages teams for each step of a task-solving\nprocess, utilizing nested group conversations and reflection to ensure diverse\nexpertise and prevent stereotypical outputs, allowing for a flexible yet\nstructured approach to problem-solving. A comprehensive evaluation across six\nreal-world scenarios demonstrates that Captain Agent significantly outperforms\nexisting multi-agent methods with 21.94% improvement in average accuracy,\nproviding outstanding performance without requiring task-specific prompt\nengineering. Our exploration of different backbone LLM and cost analysis\nfurther shows that Captain Agent can improve the conversation quality of weak\nLLM and achieve competitive performance with extremely low cost, which\nilluminates the application of multi-agent systems.'), Document(metadata={'pk': 453568971862704140, 'summary': "This technical report presents the Drama Engine, a novel framework for\nagentic interaction with large language models designed for narrative purposes.\nThe framework adapts multi-agent system principles to create dynamic,\ncontext-aware companions that can develop over time and interact with users and\neach other. Key features include multi-agent workflows with delegation, dynamic\nprompt assembly, and model-agnostic design. The Drama Engine introduces unique\nelements such as companion development, mood systems, and automatic context\nsummarising. It is implemented in TypeScript. The framework's applications\ninclude multi-agent chats and virtual co-workers for creative writing. The\npaper discusses the system's architecture, prompt assembly process, delegation\nmechanisms, and moderation techniques, as well as potential ethical\nconsiderations and future extensions.", 'title': 'Drama Engine: A Framework for Narrative Agents', 'url': 'http://arxiv.org/abs/2408.11574v1'}, page_content="This technical report presents the Drama Engine, a novel framework for\nagentic interaction with large language models designed for narrative purposes.\nThe framework adapts multi-agent system principles to create dynamic,\ncontext-aware companions that can develop over time and interact with users and\neach other. Key features include multi-agent workflows with delegation, dynamic\nprompt assembly, and model-agnostic design. The Drama Engine introduces unique\nelements such as companion development, mood systems, and automatic context\nsummarising. It is implemented in TypeScript. The framework's applications\ninclude multi-agent chats and virtual co-workers for creative writing. The\npaper discusses the system's architecture, prompt assembly process, delegation\nmechanisms, and moderation techniques, as well as potential ethical\nconsiderations and future extensions."), Document(metadata={'pk': 453568971862704130, 'summary': 'The final frontier for simulation is the accurate representation of complex,\nreal-world social systems. While agent-based modeling (ABM) seeks to study the\nbehavior and interactions of agents within a larger system, it is unable to\nfaithfully capture the full complexity of human-driven behavior. Large language\nmodels (LLMs), like ChatGPT, have emerged as a potential solution to this\nbottleneck by enabling researchers to explore human-driven interactions in\npreviously unimaginable ways. Our research investigates simulations of human\ninteractions using LLMs. Through prompt engineering, inspired by Park et al.\n(2023), we present two simulations of believable proxies of human behavior: a\ntwo-agent negotiation and a six-agent murder mystery game.', 'title': 'Exploring the Intersection of Large Language Models and Agent-Based Modeling via Prompt Engineering', 'url': 'http://arxiv.org/abs/2308.07411v1'}, page_content='The final frontier for simulation is the accurate representation of complex,\nreal-world social systems. While agent-based modeling (ABM) seeks to study the\nbehavior and interactions of agents within a larger system, it is unable to\nfaithfully capture the full complexity of human-driven behavior. Large language\nmodels (LLMs), like ChatGPT, have emerged as a potential solution to this\nbottleneck by enabling researchers to explore human-driven interactions in\npreviously unimaginable ways. Our research investigates simulations of human\ninteractions using LLMs. Through prompt engineering, inspired by Park et al.\n(2023), we present two simulations of believable proxies of human behavior: a\ntwo-agent negotiation and a six-agent murder mystery game.'), Document(metadata={'pk': 453568971862704129, 'summary': 'Two ways has been discussed to unlock the reasoning capability of a large\nlanguage model. The first one is prompt engineering and the second one is to\ncombine the multiple inferences of large language models, or the multi-agent\ndiscussion. Theoretically, this paper justifies the multi-agent discussion\nmechanisms from the symmetry of agents. Empirically, this paper reports the\nempirical results of the interplay of prompts and discussion mechanisms,\nrevealing the empirical state-of-the-art performance of complex multi-agent\nmechanisms can be approached by carefully developed prompt engineering. This\npaper also proposes a scalable discussion mechanism based on conquer and merge,\nproviding a simple multi-agent discussion solution with simple prompts but\nstate-of-the-art performance.', 'title': 'On the Discussion of Large Language Models: Symmetry of Agents and Interplay with Prompts', 'url': 'http://arxiv.org/abs/2311.07076v1'}, page_content='Two ways has been discussed to unlock the reasoning capability of a large\nlanguage model. The first one is prompt engineering and the second one is to\ncombine the multiple inferences of large language models, or the multi-agent\ndiscussion. Theoretically, this paper justifies the multi-agent discussion\nmechanisms from the symmetry of agents. Empirically, this paper reports the\nempirical results of the interplay of prompts and discussion mechanisms,\nrevealing the empirical state-of-the-art performance of complex multi-agent\nmechanisms can be approached by carefully developed prompt engineering. This\npaper also proposes a scalable discussion mechanism based on conquer and merge,\nproviding a simple multi-agent discussion solution with simple prompts but\nstate-of-the-art performance.')]

[45]

The papers that talk about Multi-Agent are:

* "Adaptive In-conversation Team Building for Language Model Agents"
* "Drama Engine: A Framework for Narrative Agents"
* "Exploring the Intersection of Large Language Models and Agent-Based Modeling via Prompt Engineering"
* "On the Discussion of Large Language Models: Symmetry of Agents and Interplay with Prompts"

These papers discuss various aspects of multi-agent systems, including team building, narrative agents, simulations of human interactions, and discussion mechanisms.

[46]



> Entering new GraphCypherQAChain chain...
Generated Cypher:
cypher
MATCH (p:Paper)
WHERE toLower(p.title) CONTAINS toLower("multi-agent")
RETURN p.title, p.id


> Finished chain.
{'query': 'What paper talk about Multi-Agent?', 'result': [{'p.title': 'Multi-Agent Assistant Code Generation (AgentCoder)', 'p.id': 'Multi-Agent Assistant Code Generation (Agentcoder)'}, {'p.title': 'Framework for Automatically Generating Process Models with Multi-Agent Orchestration (MAO)', 'p.id': 'This Article'}, {'p.title': 'Collaborative Multi-Agent, Multi-Reasoning-Path (CoMM) Prompting Framework', 'p.id': 'This Work'}], 'intermediate_steps': [{'query': 'cypher\nMATCH (p:Paper)\nWHERE toLower(p.title) CONTAINS toLower("multi-agent")\nRETURN p.title, p.id\n'}]}

[47]

The papers that talk about Multi-Agent are:

* "Adaptive In-conversation Team Building for Language Model Agents"
* "Drama Engine: A Framework for Narrative Agents"
* "Exploring the Intersection of Large Language Models and Agent-Based Modeling via Prompt Engineering"
* "On the Discussion of Large Language Models: Symmetry of Agents and Interplay with Prompts"

These papers discuss various aspects of multi-agent systems, including team building, narrative agents, simulations of human interactions, and discussion mechanisms.

[48]

{'score': 'yes'}

[49]

{'score': 'yes'}

[50]

{'datasource': 'vectorstore'}

[51]

We'll implement these as a control flow in LangGraph.

[52]

Graph Build

[53]

[54]

---ROUTE QUESTION---
What are the types of Prompt Engineering?
{'datasource': 'vectorstore'}
vectorstore
---ROUTE QUESTION TO VECTORSTORE RAG---
---RETRIEVE---
'Finished running: retrieve:'
---CHECK DOCUMENT RELEVANCE TO QUESTION---
---GRADE: DOCUMENT RELEVANT---
---GRADE: DOCUMENT RELEVANT---
---GRADE: DOCUMENT RELEVANT---
---GRADE: DOCUMENT RELEVANT---
---ASSESS GRADED DOCUMENTS---
---DECISION: GENERATE---
'Finished running: grade_documents:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---
'Finished running: generate:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[54], line 8
      5 from pprint import pprint
      7 inputs = {"question": "What are the types of Prompt Engineering?"}
----> 8 for output in app.stream(inputs):
      9     for key, value in output.items():
     10         pprint(f"Finished running: {key}:")

File ~/Library/Caches/pypoetry/virtualenvs/milvus-bootcamp-rag-MiiP0ihC-py3.11/lib/python3.11/site-packages/langgraph/pregel/__init__.py:1221, in Pregel.stream(self, input, config, stream_mode, output_keys, interrupt_before, interrupt_after, debug, subgraphs)
   1210 # Similarly to Bulk Synchronous Parallel / Pregel model
   1211 # computation proceeds in steps, while there are channel updates
   1212 # channel updates from step N are only visible in step N+1
   1213 # channels are guaranteed to be immutable for the duration of the step,
   1214 # with channel updates applied only at the transition between steps
   1215 while loop.tick(
   1216     input_keys=self.input_channels,
   1217     interrupt_before=interrupt_before,
   1218     interrupt_after=interrupt_after,
   1219     manager=run_manager,
   1220 ):
-> 1221     for _ in runner.tick(
   1222         loop.tasks.values(),
   1223         timeout=self.step_timeout,
   1224         retry_policy=self.retry_policy,
   1225     ):
   1226         # emit output
   1227         for o in output():
   1228             yield o

File ~/Library/Caches/pypoetry/virtualenvs/milvus-bootcamp-rag-MiiP0ihC-py3.11/lib/python3.11/site-packages/langgraph/pregel/runner.py:58, in PregelRunner.tick(self, tasks, timeout, retry_policy)
     56 end_time = timeout + time.monotonic() if timeout else None
     57 while futures:
---> 58     done, _ = concurrent.futures.wait(
     59         futures,
     60         return_when=concurrent.futures.FIRST_COMPLETED,
     61         timeout=(max(0, end_time - time.monotonic()) if end_time else None),
     62     )
     63     if not done:
     64         break  # timed out

File ~/.pyenv/versions/3.11.8/lib/python3.11/concurrent/futures/_base.py:305, in wait(fs, timeout, return_when)
    301         return DoneAndNotDoneFutures(done, not_done)
    303     waiter = _create_and_install_waiters(fs, return_when)
--> 305 waiter.event.wait(timeout)
    306 for f in fs:
    307     with f._condition:

File ~/.pyenv/versions/3.11.8/lib/python3.11/threading.py:629, in Event.wait(self, timeout)
    627 signaled = self._flag
    628 if not signaled:
--> 629     signaled = self._cond.wait(timeout)
    630 return signaled

File ~/.pyenv/versions/3.11.8/lib/python3.11/threading.py:327, in Condition.wait(self, timeout)
    325 try:    # restore state no matter what (e.g., KeyboardInterrupt)
    326     if timeout is None:
--> 327         waiter.acquire()
    328         gotit = True
    329     else:

KeyboardInterrupt:

[58]

---ROUTE QUESTION---
Did Emmanuel Macron visit Germany recently?
{'datasource': 'web_search'}
web_search
---ROUTE QUESTION TO WEB SEARCH---
---WEB SEARCH---
'Finished running: websearch:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---
---GRADE GENERATION vs QUESTION---
---DECISION: GENERATION ADDRESSES QUESTION---
'Finished running: generate:'
('Yes, Emmanuel Macron visited Germany recently. He arrived on May 26 for the '
 'first state visit by a French president in 24 years. The trip was meant to '
 'ease recent tensions and show unity between France and Germany.')

[60]

---ROUTE QUESTION---
Which paper talk about Collaborative Multi-Agent?
{'datasource': 'graphrag'}
graphrag
---TRYING GRAPH SEARCH---
---GRAPH SEARCH---


> Entering new GraphCypherQAChain chain...
Generated Cypher:
cypher
MATCH (p:Paper)
WHERE toLower(p.title) CONTAINS toLower("Collaborative Multi-Agent")
RETURN p.title, p.id


> Finished chain.
---GRAPH SEARCH---


> Entering new GraphCypherQAChain chain...
Generated Cypher:
cypher
MATCH (p:Paper)
WHERE toLower(p.title) CONTAINS toLower("Collaborative Multi-Agent")
RETURN p.title, p.id


> Finished chain.
'Finished running: graphrag:'
---GENERATE---
---CHECK HALLUCINATIONS---
---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---
---GRADE GENERATION vs QUESTION---
---DECISION: GENERATION ADDRESSES QUESTION---
'Finished running: generate:'
('The paper "Collaborative Multi-Agent, Multi-Reasoning-Path (CoMM) Prompting '
 'Framework" discusses Collaborative Multi-Agent.')

Give us a star on Github ⭐️