Step-by-step: build a research agent that takes a topic, searches the web, and writes a structured summary. Real code. Runs locally.
A research agent. Give it a topic like "quantum computing breakthroughs 2024" and it will search the web for recent articles, extract key information, synthesize findings across multiple sources, and write a structured markdown report — all autonomously.
By the end, you'll have a working agent you can customize, extend, and use as the foundation for any tool-using agent project.
Before writing any agent code, you need Python and a few dependencies. We'll set up an isolated virtual environment so your agent's packages don't conflict with anything else on your system.
# Create your project directory mkdir research-agent && cd research-agent # Create isolated Python environment python -m venv venv # Activate it (Mac/Linux) source venv/bin/activate # Activate it (Windows) # venv\Scripts\activate # Install all dependencies pip install anthropic requests beautifulsoup4 python-dotenv
Next, create a .env file in your project root to hold your API key. Never hardcode API keys directly in source files.
# .env — DO NOT commit this file to git
ANTHROPIC_API_KEY=your-api-key-hereGet your API key at console.anthropic.com — click your profile, then "API Keys". New accounts get free credits to start.
Run echo ".env" >> .gitignore to make sure you never accidentally commit your API key to GitHub. This is a good habit for every project.
Checkpoint: You should see (venv) at the start of your terminal prompt. If not, re-run the activate command above.
Tools are what make an agent genuinely useful. Without tools, you just have a language model generating text. With tools, the agent can take actions in the real world — in our case, searching the web and reading articles.
We'll give our agent two tools: search_web (uses DuckDuckGo, no API key needed) and read_url (fetches and extracts text from any URL). Create a new file called tools.py.
import requests from bs4 import BeautifulSoup import json def search_web(query: str, num_results: int = 5) -> str: """Search using DuckDuckGo (no API key needed)""" try: url = f"https://html.duckduckgo.com/html/?q={query}" headers = {"User-Agent": "Mozilla/5.0 (research-agent/1.0)"} response = requests.get(url, headers=headers, timeout=10) soup = BeautifulSoup(response.text, 'html.parser') results = [] for result in soup.find_all('div', class_='result', limit=num_results): title_tag = result.find('a', class_='result__a') snippet_tag = result.find('a', class_='result__snippet') url_tag = result.find('a', class_='result__url') if title_tag: results.append({ "title": title_tag.get_text(strip=True), "url": url_tag.get_text(strip=True) if url_tag else "", "snippet": snippet_tag.get_text(strip=True) if snippet_tag else "" }) return json.dumps(results, indent=2) except Exception as e: return f"Search failed: {str(e)}" def read_url(url: str) -> str: """Fetch and extract readable text from a URL""" try: headers = {"User-Agent": "Mozilla/5.0 (research-agent/1.0)"} response = requests.get(url, headers=headers, timeout=15) soup = BeautifulSoup(response.text, 'html.parser') # Remove navigation, footer, and scripts — they're noise for tag in soup(['script', 'style', 'nav', 'footer', 'header']): tag.decompose() text = soup.get_text(separator='\n', strip=True) # Limit to 3000 chars — enough content, won't blow the context window return text[:3000] + "..." if len(text) > 3000 else text except Exception as e: return f"Failed to read URL: {str(e)}" # Tool definitions — these tell the Anthropic API what tools the agent has TOOL_DEFINITIONS = [ { "name": "search_web", "description": "Search the web for current information. Returns titles, URLs, and snippets.", "input_schema": { "type": "object", "properties": { "query": {"type": "string", "description": "Search query"}, "num_results": {"type": "integer", "description": "Number of results (1-10)", "default": 5} }, "required": ["query"] } }, { "name": "read_url", "description": "Read the content of a specific URL. Use after search_web to get full article content.", "input_schema": { "type": "object", "properties": { "url": {"type": "string", "description": "The URL to read"} }, "required": ["url"] } } ]
Checkpoint: Run python -c "from tools import search_web; print(search_web('test')[:200])" — you should see JSON search results, not an error.
This is the heart of the agent: the loop that sends messages to the model, handles tool calls, feeds results back, and keeps going until the model is done. Create agent.py.
import anthropic import os from dotenv import load_dotenv from tools import search_web, read_url, TOOL_DEFINITIONS load_dotenv() client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY")) SYSTEM_PROMPT = """You are a research agent. When given a topic, you: 1. Search for recent, authoritative sources using search_web 2. Read the most relevant ones in full using read_url 3. Synthesize the information from multiple sources 4. Write a comprehensive markdown report with: - Executive summary (3-5 sentences) - Key findings (bullet points) - Main trends and developments - Notable quotes or statistics with source attribution - Sources section with titles and URLs Be thorough but concise. Prioritize sources from the last 12 months when possible. Always read at least 2-3 full articles before writing your report.""" def execute_tool(name: str, inputs: dict) -> str: """Route a tool call request to the correct implementation""" if name == "search_web": return search_web(inputs["query"], inputs.get("num_results", 5)) elif name == "read_url": return read_url(inputs["url"]) return f"Unknown tool: {name}" def run_research_agent(topic: str, max_iterations: int = 15) -> str: """Run the research agent on a given topic and return a markdown report""" print(f"\n Researching: {topic}") print("-" * 50) # Start conversation with the user's research request messages = [{ "role": "user", "content": f"Research this topic and write a comprehensive report: {topic}" }] iteration = 0 while iteration < max_iterations: iteration += 1 print(f"\n[Iteration {iteration}]") # Send messages to the model response = client.messages.create( model="claude-opus-4-6", max_tokens=4096, system=SYSTEM_PROMPT, tools=TOOL_DEFINITIONS, messages=messages ) print(f" Stop reason: {response.stop_reason}") if response.stop_reason == "end_turn": # Model finished — extract the final text response for block in response.content: if hasattr(block, 'text'): return block.text return "Research complete (no text output generated)" if response.stop_reason == "tool_use": # Model wants to use tools — add its response to history messages.append({"role": "assistant", "content": response.content}) # Execute every tool the model requested tool_results = [] for block in response.content: if block.type == "tool_use": print(f" {block.name}: {str(block.input)[:80]}...") result = execute_tool(block.name, block.input) tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": result }) # Return all tool results to the model and loop again messages.append({"role": "user", "content": tool_results}) return "Max iterations reached. The partial research above may be incomplete."
Every time the model calls a tool, we append the assistant's response (including the tool call request) to messages, then append the tool results, then send the whole growing list back to the model. This is how the model "remembers" what it has already done in the conversation. Without this, each API call would be stateless — the model would forget it already searched and read articles.
Checkpoint: The file should import without errors. Run python -c "from agent import run_research_agent; print('OK')"
Create main.py — a simple entry point that takes a topic from the command line, runs the agent, and saves the report to a markdown file.
import sys from agent import run_research_agent def main(): # Accept topic from command line args or prompt interactively if len(sys.argv) < 2: topic = input("Enter research topic: ") else: topic = " ".join(sys.argv[1:]) # Run the agent report = run_research_agent(topic) # Save the report as a markdown file filename = f"report_{topic[:30].replace(' ', '_')}.md" with open(filename, 'w') as f: f.write(report) print(f"\n Report saved to: {filename}") print("\n" + "="*50) print(report[:500] + "...") if __name__ == "__main__": main()
Now run it with a research topic:
python main.py "quantum computing breakthroughs 2024"
Your project directory should now look like this:
Checkpoint: You should see iteration logs printed to the terminal, then "Report saved to: report_quantum..." — open the .md file to read the agent's research output.
Your agent just ran a multi-step research loop autonomously. Here's a real trace showing exactly what a typical run looks like — and what was happening at each step.
Walk through what the agent decided at each step:
end_turn.Why max_iterations matters: Without a limit, a confused or stuck agent can loop forever — calling the same tool repeatedly, getting stuck in a cycle, or just not knowing when to stop. Always set a safety ceiling. 15 iterations is generous for most research tasks. Anything over 20 is almost certainly a bug or a runaway agent.
Your agent works. Now make it yours. Here are three practical customizations with complete code.
The default claude-opus-4-6 is the most capable model but also the most expensive. For research tasks that don't need maximum depth, claude-haiku-4-5-20251001 is roughly 20x cheaper, completes in half the time, and still produces solid summaries. Edit agent.py:
# In agent.py, change this line: response = client.messages.create( model="claude-haiku-4-5-20251001", # ~20x cheaper, still capable max_tokens=4096, ... )
Use Haiku for: quick summaries, high-volume runs, drafts. Use Opus for: deep research, complex reasoning, final reports.
Give the agent a save_note tool so it can bookmark important URLs and facts mid-research. This creates a running "research notebook" you can read alongside the final report.
# Add to tools.py notes = [] # in-memory store for this run def save_note(content: str, source_url: str = "") -> str: """Save an important finding or URL to the research notebook""" note = {"content": content, "source": source_url} notes.append(note) return f"Note saved. Total notes: {len(notes)}" # Add to TOOL_DEFINITIONS list: { "name": "save_note", "description": "Save an important finding, quote, or URL for your research report.", "input_schema": { "type": "object", "properties": { "content": {"type": "string", "description": "The finding to save"}, "source_url": {"type": "string", "description": "URL this came from"} }, "required": ["content"] } } # In agent.py, add to execute_tool: elif name == "save_note": from tools import save_note return save_note(inputs["content"], inputs.get("source_url", ""))
The system prompt is the most powerful lever you have. Changing it completely changes the agent's behavior without touching any other code. Here are three variants:
# For academic/technical research: SYSTEM_PROMPT = """You are a technical research agent. Only use academic papers, official documentation, and peer-reviewed sources. For each claim, cite the specific paper or study. Prefer arXiv, Google Scholar, Nature, and IEEE sources. Ignore news articles, blogs, and opinion pieces.""" # For competitive intelligence: SYSTEM_PROMPT = """You are a competitive intelligence analyst. Focus on: product features, pricing pages, job postings (reveals roadmap), press releases, and customer reviews on G2/Trustpilot. Build a feature comparison table. Highlight gaps and opportunities for a startup to differentiate.""" # For investment research: SYSTEM_PROMPT = """You are a financial research agent. Focus on earnings reports, SEC filings, analyst coverage, and industry reports. Structure your output as: bull case, bear case, key metrics, and risk factors. Always include the source and date for every data point — financial claims without citations are useless."""
You've got a working research agent. Here are five concrete ways to extend it, each building on what you just built.
Beyond this kit, here are the most useful next reads:
It searches the web, reads articles, and writes structured research reports. This same pattern — an agent loop, tools, and a system prompt — is the foundation of every production agent in use today. What you build next is up to you.