Master Prompt: Fully Automated Local Setup & User Guide for “Browser-Use”

# MISSION: Autonomous Local Setup & Interactive Guide Generator for "Browser-Use"

You are an expert DevOps and AI Systems Engineer. Your objective is to fully automate the local installation, environment configuration, dependency resolution, and test execution of the open-source library `browser-use` on my machine, followed by generating a comprehensive, beginner-friendly `USER_GUIDE.md` in this directory.

---

### PHASE 1: System Inspection & Environment Setup
1. **Detect OS & Shell:** Identify the current Operating System (Windows, macOS, or Linux) and active shell environment (Bash, Zsh, PowerShell, or CMD).
2. **Verify Prerequisites:**
   - Check if Python (>= 3.11) is installed.
   - Check if `git` and package managers (`uv` or `pip`) are available.
   - If `uv` is not installed, install it or fall back cleanly to standard Python `venv` and `pip`.
3. **Isolate Virtual Environment:**
   - Create a dedicated virtual environment named `.venv` using Python 3.11+.
   - Activate the virtual environment based on the detected OS.

---

### PHASE 2: Dependency Installation & Browser Engine Setup
1. **Install Core Libraries:**
   - Install `browser-use` along with core LLM connectors:
     pip install browser-use langchain-openai langchain-anthropic langchain-google-genai langchain-ollama python-dotenv
2. **Install Playwright Browsers & OS Dependencies:**
   - Run the Playwright browser installer with all necessary system dependencies:
     playwright install chromium --with-deps
   - Handle any permission or platform-specific installation flags automatically.

---

### PHASE 3: Configuration & Secret Management
1. **Create `.env.example` and `.env`:**
   Generate a ready-to-use `.env` template containing placeholders and clear comments:
   # --- Cloud LLM API Keys (Provide at least one) ---
   OPENAI_API_KEY=your_openai_api_key_here
   ANTHROPIC_API_KEY=your_anthropic_api_key_here
   GEMINI_API_KEY=your_gemini_api_key_here

   # --- Local LLM (Ollama) Settings ---
   OLLAMA_BASE_URL=http://localhost:11434

   # --- Browser Configuration ---
   BROWSER_HEADLESS=false
   # Optional: Path to your local Chrome profile for session reuse
   # CHROME_USER_DATA_DIR=

2. Ask me which LLM provider (OpenAI, Anthropic, Gemini, or Local Ollama) I want to configure right now, and update the `.env` accordingly.

---

### PHASE 4: Boilerplate Code Generation
Create the following modular, well-commented Python scripts in the project root:

1. `quickstart.py` (Default Cloud LLM Script):
   - Initializes `Agent` from `browser_use`.
   - Loads `.env` securely.
   - Accepts a search or web automation task (e.g., "Go to Hacker News and return the top 3 trending AI articles").
   - Runs headfully so the user can visually watch the browser actions.

2. `local_ollama_agent.py` (100% Offline / Local LLM Script):
   - Configures `ChatOllama` (e.g., `qwen2.5` or `llama3.2`) with `browser-use`.
   - Includes fallback error handling for local context limits and vision requirements.

3. `custom_chrome_session.py` (Reuse Real Chrome Profile):
   - Implements persistent browser profile connection so users do not need to re-login to protected websites.

---

### PHASE 5: Test Execution & Self-Healing
1. Execute a dry-run test using `quickstart.py` to confirm:
   - Python virtual environment is active.
   - Chromium binary launches without headless/display errors.
   - LLM connection and agent task loop start successfully.
2. If any error occurs (e.g., missing DLLs on Windows, missing Linux shared libraries, missing Playwright drivers), automatically diagnose the error, apply the fix, and re-test.

---

### PHASE 6: Generate `USER_GUIDE.md`
Generate a clean, structured `USER_GUIDE.md` document in the workspace covering:
- **Architecture Overview**: How Browser-Use interacts with Playwright, CDP, and Vision-capable LLMs.
- **How to Run**: Step-by-step commands to activate the environment and trigger tasks.
- **Switching Models**: How to toggle between OpenAI (GPT-4o), Anthropic (Claude 3.5 Sonnet), Google Gemini 2.0/1.5, and Local Ollama.
- **Headful vs. Headless Mode**: When and how to toggle visible browser vs. background headless execution.
- **Security & Session Safety**: Best practices for cookies, sensitive logins, and `.env` protection.
- **Troubleshooting FAQ**: Solutions for common errors (Playwright binary missing, rate limits, CAPTCHA handling).

---

### EXECUTION INSTRUCTION:
Begin Phase 1 immediately. Inform me of your progress at each milestone, prompt me for any missing API keys or preferences, and complete the full setup automatically.

1. The Death of Brittle Selectors: Why Legacy Scraping Dies in Production

Every engineer who has maintained a fleet of Selenium, Puppeteer, or Playwright scripts in production knows the dreaded 2:00 AM PagerDuty alert. A critical automated pipeline just flatlined. You open the logs, trace the failure, and find the culprit: a routine frontend release changed an obscure class name from .css-a8f3x91 to .css-k2m9p4. Your selector threw a TimeoutError, and downstream ETL jobs halted.

The core problem has never been bad code. The flaw lies in the premise that a website’s DOM is a stable API contract. In modern single-page applications (SPAs), it simply is not.

Traditional browser automation suffers from four structural points of failure:

• Build-Time Hashed Class Names: CSS-in-JS libraries (like Tailwind JIT, Styled Components, and Emotion) generate arbitrary string hashes on every build. Relying on class-based XPath or CSS locators turns every software release into a ticking time bomb.
• Shadow DOM Encapsulation: Component-driven web systems (built with Web Components, Lit, or Stencil) hide elements behind shadow roots. Closed shadow roots are completely invisible to standard querySelector queries.
• Stateless Client-Side Routing: When modern frameworks swap page views via history.pushState, traditional load navigation events never fire. Guessing with waitForSelector or networkidle creates intermittent race conditions.
• Virtualized DOM Rendering: Data grids built with libraries like react-window unmount invisible rows from the DOM. If your scraper looks for row 85 of a 500-item table, that element literally does not exist in the DOM tree until you scroll down to it.

“Traditional scrapers fail because they treat websites like static documents. Autonomous browser agents succeed because they interact with web pages exactly like human operators: by looking at the rendered viewport, understanding layout context, and reasoning about intent.”

Instead of hardcoding brittle navigational scripts, modern developers are building autonomous browser agents that observe live UI states, parse interactive visual nodes, and autonomously execute complex workflows.

2. Architectural Breakdown: How Autonomous Browser Agents Perceive and Act

An autonomous browser agent replaces hardcoded locators with a continuous, closed perception-action loop. Rather than asking “Where is this specific XPath?”, the agent asks “What interactive elements are currently visible, and what action moves me closer to the goal?”

From Raw DOM to Pruned Accessibility Trees

At the foundation of high-performance AI agent browser automation is the Chrome DevTools Protocol (CDP). Instead of injecting heavyweight JavaScript snippets that can alter page behavior or trigger bot-detection heuristics, CDP provides direct, binary communication with the browser engine.

During every step of an agent’s execution cycle:

1. Snapshotting: The agent captures a complete DOM and Accessibility (a11y) tree snapshot via CDP commands like DOM.getDocument.
2. Semantic Pruning: Raw web pages easily contain 15,000+ noisy nodes (styles, structural divs, hidden SVG defs). The agent strips layout clutter and extracts only interactive nodes—such as buttons, inputs, links, and dropdowns.
3. Bounding Box Calculation: CDP calculates exact screen coordinates and dimensions for each visible, interactable element.
4. Indexed Numbering: Each interactive node is assigned a lightweight integer label (e.g., [12] <button> "Submit Order").

Set-of-Marks and Visual Grounding with Vision Models

Text trees alone can struggle with icon-only buttons, canvas-rendered charts, or custom WebGL components. To overcome this, modern frameworks like browser-use utilize Set-of-Marks (SoM) visual prompting.

The framework captures a live screenshot of the viewport and overlays colored bounding boxes with numeric IDs on top of every interactable element. Both the visual screenshot and the pruned element map are passed to a multimodal reasoning model.

The reasoning model inspects the layout and decides the next action (e.g., click(index=14) or type(index=5, text="SKU-9901")). The agent’s controller maps that index back to live bounding box coordinates and fires native input events using CDP’s Input.dispatchMouseEvent. Because the input originates at the browser engine level, it mimics human physical interaction far more cleanly than synthetic JavaScript event triggers.

3. Hands-on Tutorial: Building a Multi-Step Autonomous Pipeline with browser-use

Let us build a real-world, production-ready pipeline using Python and browser-use—one of the leading open-source frameworks for autonomous browsing (available on the Browser AI agent GitHub repository). This script demonstrates an end-to-end workflow: logging into a portal, scraping paginated product inventories, safely managing downloads, and escalating unexpected CAPTCHAs to a human operator.

Agent, Controller, and BrowserSession Configuration

When engineering with browser-use, three core abstractions manage the automation lifecycle:

• BrowserSession (or BrowserProfile): Configures headless status, persistent browser profiles, proxy rotation, viewports, and custom download locations.
• Controller (Tools Registry): The extensible tool interface where custom Python actions (like CAPTCHA alerts or database saves) are registered for the LLM to call.
• Agent: The orchestrator that coordinates system prompts, reasoning steps, execution budgets, and sensitive credential masking.

Complete Production Python Implementation

First, install the required packages and download the browser binaries:

pip install browser-use pydantic
playwright install chromium

Here is the complete, self-contained Python pipeline:

"""
production_browser_agent.py
Demonstrating resilient autonomous browser automation with browser-use and CDP.
"""

import asyncio
import logging
from pathlib import Path
from pydantic import BaseModel, Field
from browser_use import (
    Agent,
    ActionResult,
    BrowserProfile,
    BrowserSession,
    ChatAnthropic,
    Controller,
)

# Configure logging for production observability
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("autonomous_agent")

DOWNLOAD_DIR = Path("./agent_downloads")
DOWNLOAD_DIR.mkdir(exist_ok=True)

# ---------------------------------------------------------------------------
# 1. Pydantic Output Validation Schema
# ---------------------------------------------------------------------------
class InventoryItem(BaseModel):
    sku: str = Field(description="Unique product SKU")
    title: str = Field(description="Product title or name")
    unit_price: float = Field(description="Price in USD")
    stock_status: bool = Field(description="True if in stock, False otherwise")

class ExtractionResult(BaseModel):
    items: list[InventoryItem] = Field(default_factory=list)
    scraped_pages: int = Field(default=0)

# ---------------------------------------------------------------------------
# 2. Browser Layer with Persistent Profile & Fixed Viewport
# ---------------------------------------------------------------------------
profile = BrowserProfile(
    headless=True,
    user_data_dir="./.browser_profiles/production_session",
    viewport={"width": 1440, "height": 900},
    downloads_path=str(DOWNLOAD_DIR),
)

browser_session = BrowserSession(browser_profile=profile)

# ---------------------------------------------------------------------------
# 3. Custom Controller Actions (Tools)
# ---------------------------------------------------------------------------
controller = Controller(output_model=ExtractionResult)

@controller.action("Escalate verification challenge or CAPTCHA to a human operator")
async def handle_human_intervention(reason: str) -> ActionResult:
    logger.warning("Human intervention required: %s", reason)
    # In production, dispatch a Slack or PagerDuty webhook notification
    await asyncio.get_event_loop().run_in_executor(
        None, input, "Please solve the challenge in the active browser, then press Enter..."
    )
    return ActionResult(extracted_content="Challenge cleared by human operator.", include_in_memory=True)

@controller.action("Register a downloaded inventory CSV file for ETL pickup")
async def register_downloaded_report(filename: str) -> ActionResult:
    file_path = DOWNLOAD_DIR / filename
    if not file_path.exists():
        return ActionResult(error=f"File {filename} was not found on disk.")
    logger.info("Successfully captured file: %s (%d bytes)", file_path, file_path.stat().st_size)
    return ActionResult(extracted_content=f"Report {filename} logged for data pipeline ingestion.")

# ---------------------------------------------------------------------------
# 4. Agent Execution with Domain-Scoped Credential Injection
# ---------------------------------------------------------------------------
async def execute_inventory_scrape(target_url: str, credentials: dict) -> ExtractionResult:
    # Initialize the vision-capable multimodal model
    llm = ChatAnthropic(model="claude-sonnet-4-6")

    instruction_prompt = f"""
    1. Navigate to {target_url} and log in using the sensitive account credentials provided.
    2. Open the 'Warehouse Inventory' tab and locate the paginated data table.
    3. Extract sku, title, unit_price, and stock_status for all rows across up to 5 pages.
    4. If a CAPTCHA or bot verification prompt appears, call handle_human_intervention immediately.
    5. If an 'Export to CSV' button is available, click to download and invoke register_downloaded_report.
    6. Click 'Next' to paginate through results until complete, then return the validated output.
    """

    agent = Agent(
        task=instruction_prompt,
        llm=llm,
        browser_session=browser_session,
        controller=controller,
        sensitive_data={"*.company-portal.internal": credentials},
        max_actions_per_step=4,
        use_vision=True,
    )

    history = await agent.run(max_steps=80)
    return ExtractionResult.model_validate_json(history.final_result())

if __name__ == "__main__":
    secure_credentials = {"username": "ops_service_account", "password": "super_secret_password"}
    target_endpoint = "https://admin.company-portal.internal/login"

    extracted_data = asyncio.run(execute_inventory_scrape(target_endpoint, secure_credentials))
    print(f"Extraction complete. Scraped {len(extracted_data.items)} items across {extracted_data.scraped_pages} pages.")

4. Production Hardening: Sessions, CAPTCHAs, and Circuit Breakers

Running a browser demo on your local machine is simple. Running autonomous browser agents against thousands of heterogeneous web targets in a production cluster requires rigorous defense against edge cases.

Zero-Login Authentication & Session Persistence

Re-authenticating through login pages on every single job wastes LLM tokens and triggers security alerts. There are two primary strategies for maintaining persistent authentication:

1. Persistent User Data Directory (Full Profile Storage): By pointing user_data_dir to a mounted persistent volume, Chrome stores cookies, localStorage, IndexedDB, and active session tokens across process restarts. You can perform multi-factor authentication (MFA) once interactively, and all subsequent headless runs inherit the authenticated state automatically.
2. Exported Storage State Snapshots: For ephemeral, autoscaled serverless containers (e.g., Kubernetes pods or AWS ECS), serialize the authenticated cookies and storage keys into an encrypted JSON payload. Mount this state into the container on launch to bootstrap authenticated sessions without baking heavy Chrome profile directories into container images.

Security Best Practice: Never paste raw passwords directly into natural language prompts. Use domain-scoped configuration mappings (like sensitive_data={"*.target.com": creds} in browser-use). This ensures the LLM’s context window only holds a symbolic token reference, while the actual secret is injected directly into input elements over CDP.

Loop Guards and Deterministic Execution Boundaries

When an autonomous agent encounters a visual dead-end (e.g., a disabled button or an unexpected cookie banner), it can fall into a hallucination loop, attempting the same action repeatedly and burning expensive API tokens. To prevent runaway costs, implement a three-tiered circuit breaker pattern:

from collections import deque

class ActionLoopGuard:
    """Detects and breaks repetitive action loops in autonomous browser sessions."""
    def __init__(self, repeat_limit: int = 4):
        self.action_history = deque(maxlen=repeat_limit)

    def verify_step(self, chosen_action_signature: str, step_num: int):
        self.action_history.append(chosen_action_signature)
        if len(self.action_history) == self.action_history.maxlen and len(set(self.action_history)) == 1:
            raise RuntimeError(
                f"LoopGuard triggered at step {step_num}: Action '{chosen_action_signature}' "
                f"repeated {self.action_history.maxlen} times consecutively."
            )

Combine this loop guard with a hard max_steps limit and an asynchronous wall-clock timeout (asyncio.wait_for(agent.run(), timeout=600)) to guarantee deterministic system bounds.

5. Comprehensive Comparison: Traditional Scripts vs. Autonomous Browser Agents

Selecting the right browser automation paradigm depends on the determinism, scale, and volatility of your target websites. The table below details how modern perception-driven agents compare against traditional scripted tools and headless browser APIs.

6. Token Costs, Latency, and Engineering ROI

A common concern among engineering leads is the cost of LLM inference compared to traditional scrapers. While traditional scrapers have nearly zero marginal compute cost per run, they carry an invisible, expensive overhead: engineering maintenance time.

Consider the real-world economics of running automation at scale:

• The Cost of Script Maintenance: In an organization maintaining 50 scrapers across third-party vendor portals, developers spend an average of 10–15 hours per month repairing broken selectors. At standard engineering billing rates ($100–$150/hr), that equates to $1,000–$2,250 every month in recurring developer labor.
• Inference Economics: High-efficiency vision models (such as Gemini Flash or GPT mini tiers) execute browser steps for $0.001–$0.005 per step. A full 10-step autonomous workflow costs roughly $0.02. For the price of one hour of developer debugging, an engineering team can run 5,000 to 10,000 autonomous workflows that never fail due to CSS renames.
• Hybrid Routing Strategy: High-performing architectures use deterministic Playwright code for static login sequences and known navigation steps, handing control over to autonomous browser agents only when navigating complex, dynamic, or frequently redesigned interfaces.

7. Frequently Asked Questions (FAQ)

Q1: What is the primary difference between Playwright and autonomous browser agents?

Playwright is a deterministic code library where developers explicitly define every click, scroll, and selector. Autonomous browser agents use an AI model in a closed loop to observe the screen, determine what interactive elements are present, and dynamically choose the correct actions in real time.

Q2: Can autonomous browser agents run headlessly in Docker or CI/CD?

Yes. Frameworks like browser-use support standard headless Chromium profiles. In containerized environments (such as Kubernetes or AWS ECS), set the viewport dimensions explicitly and mount persistent volumes for session directories to maintain stable execution.

Q3: How do browser agents bypass or solve CAPTCHA challenges?

While some cloud browser infrastructures integrate automated CAPTCHA solvers, production systems typically register a human-in-the-loop callback action. When a bot challenge or two-factor prompt is detected, the agent pauses, sends a notification (via Slack or PagerDuty), and waits for an operator to resolve the prompt before resuming execution.

Q4: Are autonomous browser agents safe to use with sensitive company credentials?

Yes, provided you do not hardcode secrets into LLM prompts. By leveraging domain-restricted credential injection (like the sensitive_data configuration in browser-use), the raw password is transmitted directly to the input field over Chrome DevTools Protocol without ever exposing the plain text to the LLM’s context window.

Q5: What is the difference between browser-use and agent-browser extensions?

Frameworks like browser-use run as standalone backend services controlling dedicated browser instances via CDP. In contrast, AI agent browser extensions operate directly inside a user’s active desktop browser tab, allowing the agent to inherit existing logins and session cookies without requiring proxy or profile synchronization.

Q6: Which vision models work best for autonomous browser automation?

Frontier vision models with high pixel-grounding accuracy (such as Claude Sonnet-class, GPT-4o, and Gemini Flash) excel at Set-of-Marks and coordinate-based clicking. For high-volume, simple form filling, smaller distilled models offer lower latency and high efficiency.

8. Conclusion and Production Deployment Checklist

Autonomous browser agents represent a fundamental evolution in software automation. By decoupling web interactions from brittle DOM hierarchies and anchoring them in real-time visual perception, developers can build resilient, adaptive pipelines that withstand continuous frontend redesigns.

Before deploying your next browser agent to production, verify these five architectural safeguards:

• [ ] Config-Layer Secret Injection: Ensure all passwords and API tokens are injected via CDP rather than exposed inside LLM prompt templates.
• [ ] Deterministic Execution Limits: Implement a loop guard, a hard max_steps cap, and an asynchronous wall-clock timeout.
• [ ] Versioned Profile Persistence: Store cookies and storage states on managed volumes with automated expiration schedules to avoid stale session bugs.
• [ ] Human Escalation Callbacks: Implement a first-class tool for handling CAPTCHAs and ambiguous modal dialogs.
• [ ] Hybrid Cost Optimization: Use fast, deterministic scripts for fixed paths, and deploy autonomous LLM reasoning where UI volatility demands adaptability.

Ready to eliminate selector maintenance forever? Start by cloning the open-source browser-use pipeline tutorial above, configure your environment variables, and build self-healing web automation for your organization today.