Skip to main content

Webpage Search

Enable your agents to browse the web by extracting textual content from specific URLs.

This guide will help you configure the Webpage Search tool to allow your agents to read articles, documentation, or any public web content for analysis and summarization.

💡 Core Concepts​

To use this tool effectively, it is helpful to understand how it processes data.

1. How does Webpage Search work?​

The tool acts as a bridge between your agent and the internet. When an agent provides a URL, the tool fetch the content of the page.

⚙️ Configuration Steps​

Follow these steps to equip your agent with browsing capabilities.

Assign the Tool​

To enable this feature, you simply need to assign the webpage_search tool to your agent in the YAML configuration. No API keys are required for public pages.

📚 Practical Recipes (Examples)​

Recipe 1: Research & Summarizer Agent​

Use Case: An agent that takes a URL from a user, reads the content, and provides a summary.

create_vertical_agent_network:
  agent-1:
    agent_name: Researcher_Agent
    LLM_config:
      params:
        model: gpt-4o
    tools:
      tool_assigned:
        - name: webpage_search
    agent_function:
      - You are a research assistant.
      - When a user provides a link, use the webpage_search tool to read it.
      - Summarize the key points of the content found in the URL.
    incoming_edge:
      - Start
    outgoing_edge: []

💡 Note: Handling Large Pages​

If a webpage is extremely large, downstream tools or LLM context windows might truncate the text.

🚑 Troubleshooting​

  • "Missing required field: 'urls'"

    • The agent failed to provide the target link. Ensure your prompt explicitly instructs the agent to "pass the 'urls' argument."

  • Empty or Garbled Output

    • The website might have anti-scraping blocks (403 Forbidden).

  • Error in scraping webpage

    • Check if the URL is valid and publicly accessible.