Building AI Agents with Haystack and Gaia Node: A Practical Guide

In this technical blog post, I’ll walk through how to build an AI agent that combines the flexibility of Haystack’s agent framework with the privacy benefits of self-hosted LLMs through Gaia Node.

This solution gives you the best of both worlds: sophisticated tool-using capabilities with complete control over your infrastructure.

What We’re Building

Our example creates an AI agent that can answer questions about current information by:

1. Using a self-hosted LLM via Gaia Node for processing
2. Performing real-time web searches when needed
3. Providing summarized, accurate responses based on the latest information

The agent specifically handles queries like:

• “How is the weather in Berlin?”
• “What are the latest news headlines?”
• “Tell me about recent events in technology”

Technical Architecture

Components Overview

Our solution leverages three main components:

1. Haystack: An open-source framework for building production-ready LLM applications
2. Gaia Node: A self-hosted inference server for open-source LLMs
3. SerperDev: A search API for obtaining real-time web information

Why This Stack?

• Data Privacy: All LLM processing happens on your infrastructure
• Cost Control: No per-token pricing, just hardware costs
• Flexibility: Choose any supported open-source model
• Real-time Capabilities: Access current information via web search

Step-by-Step Implementation

1. Environment Setup

First, we install the necessary dependencies:

pip install haystack-ai python-dotenv requests

2. Configuration

We use environment variables for secure configuration:

# .env file
GAIA_API_KEY=your_gaia_node_api_key
GAIA_NODE_URL=http://localhost:8080/v1
GAIA_MODEL_NAME=llama3b
SERPERDEV_API_KEY=your_serperdev_api_key

3. Core Code Implementation

Here’s the complete implementation:

import os
from haystack.components.agents import Agent
from haystack.tools.component_tool import ComponentTool
from haystack.components.websearch import SerperDevWebSearch
from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.utils import Secret
from dotenv import load_dotenv

load_dotenv()

# Create the web search component
web_search = SerperDevWebSearch(
    api_key=Secret.from_env_var("SERPERDEV_API_KEY"), 
    top_k=3
)

# Create the ComponentTool
web_tool = ComponentTool(
    component=web_search,
    name="web_search",
    description="Use this tool to search the web for current information like weather, news, or facts. Always use this tool when asked about current information."
)

# Configure Gaia Node connection
chat_generator = OpenAIChatGenerator(
    api_key=Secret.from_env_var("GAIA_API_KEY"),
    api_base_url=os.getenv("GAIA_NODE_URL"),
    model=os.getenv("GAIA_MODEL_NAME", "llama3b")
)

# Create the agent with the web tool
tool_calling_agent = Agent(
    chat_generator=chat_generator,
    system_prompt="""You are a helpful agent with web search capabilities. 
                     When asked about current information (weather, news, facts, real-time data),
                     ALWAYS use the web_search tool first.
                     After getting results, extract relevant information and present it clearly.
                     If search results are incomplete, mention this but still provide the best available information.""",
    tools=[web_tool]
)

# Run the agent
try:
    user_message = ChatMessage.from_user("How is the weather in Berlin today?")
    result = tool_calling_agent.run(messages=[user_message])

    response = result["messages"][-1].text
    print(f"Agent Response: {response}")

except Exception as e:
    print(f"Error: {e}")

Key Technical Concepts

1. Haystack Agents Framework

Haystack’s agent system provides a structured way to build LLM-powered applications that can use tools. The agent:

• Receives a user query
• Decides whether to use tools
• Processes the tool results
• Generates a final response

2. Gaia Node Integration

Gaia Node provides an OpenAI-compatible API endpoint for self-hosted models. This compatibility makes it easy to integrate with existing frameworks like Haystack that support OpenAI models.

3. Tool Calling Pattern

The tool calling pattern is fundamental to building capable AI agents:

1. The LLM decides when a tool is needed
2. The tool executes and returns results
3. The LLM incorporates these results into its response

Advanced Features and Customization

Adding Memory

To create conversational agents, you can add memory:

from haystack.components.agents.memory import ConversationMemory

memory = ConversationMemory()
agent_with_memory = Agent(
    chat_generator=chat_generator,
    system_prompt=system_prompt,
    tools=[web_tool],
    memory=memory
)

Multiple Tools

Agents can use multiple tools for different purposes:

# Example: Adding a calculator tool
from haystack.tools.predefined import CalculatorTool

calculator_tool = CalculatorTool()
agent = Agent(
    chat_generator=chat_generator,
    tools=[web_tool, calculator_tool],
    system_prompt="You can both search the web and perform calculations."
)

Custom Tools

You can create custom tools for specific needs:

from haystack.tools import tool

@tool
def database_lookup(query: str):
    """Look up information in our internal database."""
    # Implementation here
    return results

Deployment Considerations

Infrastructure Requirements

Running this stack requires:

• A server with sufficient RAM/CPU for your chosen model
• Network access for web search APIs
• Secure storage for API keys and credentials

Performance Optimization

• Use model quantization for faster inference
• Implement caching for frequent queries
• Consider batch processing for high-volume applications

Real-World Applications

This pattern can be adapted for various use cases:

1. Customer Support: Answer questions based on internal documentation + web search
2. Research Assistance: Gather and summarize information from multiple sources
3. Internal Knowledge Management: Combine internal data with external context

Combining Haystack with Gaia Node creates a powerful foundation for building AI applications that respect data privacy while maintaining sophisticated capabilities. The agent pattern demonstrated here provides a flexible framework that can be extended with additional tools, custom logic, and domain-specific knowledge.

This approach is particularly valuable for organizations that:

• Handle sensitive data
• Require cost-effective AI solutions
• Need custom model behavior
• Want to avoid vendor lock-in

Get Started

Ready to build your own AI agent with Haystack and Gaia Node? Check out the complete example and documentation in the Gaia Cookbook:

🔗 Gaia Cookbook – Haystack Integration