Integrations/Browser Use

Browser Use runtime authorization

Wrap Browser Use browser actions with Veto. Each governed browser action is evaluated before dispatch: allow, review, or deny, with an exportable decision record per governed decision.

Sign upRead docsBook a call

What is Browser Use?

Browser Use is an open-source Python library that gives AI agents broad browser control via Playwright. Agents can navigate URLs, fill forms, click buttons, extract page content, and take screenshots autonomously. It uses a Controller pattern for custom actions and supports any LLM provider. With browser access, the agent inherits the privileges of the browser session it runs in.

The real risks of browser agents

Browser agents inherit the user's session: cookies, credentials, and access to logged-in services. A crafted page, URL, or instruction can turn that session into a tool surface unless navigation, extraction, form submission, and download actions are governed before execution.

Credential exfiltration

Agent fills a login form on a phishing site, or types credentials into a form field on a malicious page that is been injected via prompt injection.

Payment triggering

Agent clicks "Purchase" or "Confirm Order" buttons when it was only supposed to check prices. No human reviews the action before money leaves the account.

PII extraction

Agent navigates to HR portals, payroll systems, or customer databases and extracts personally identifiable information from page content.

Prompt injection via web pages

Malicious web content instructs the browser agent to perform actions the user did not request: hidden text on a page can redirect the agent's behavior.

First governed call

1. Install

sh
pip install veto-sdk browser-use

2. Add Veto to Browser Use actions

Browser Use's Controller pattern lets you define custom actions. Put Veto on each governed action with veto.guard() to enforce policies before the browser acts.

agent.py
import os
from browser_use import Agent, Controller
from browser_use.agent.views import ActionResult
from langchain_openai import ChatOpenAI
from veto_sdk import Veto

veto = Veto(api_key=os.environ["VETO_API_KEY"])
controller = Controller()

@controller.action("Navigate to URL")
async def go_to_url(url: str) -> ActionResult:
    decision = await veto.guard(
        tool="go_to_url",
        arguments={"url": url},
    )
    if decision.decision == 'deny':
        return ActionResult(error=f"Blocked: {decision.reason}")
    return ActionResult(extracted_content=f"Navigating to {url}")

@controller.action("Fill form field")
async def input_text(selector: str, value: str) -> ActionResult:
    decision = await veto.guard(
        tool="input_text",
        arguments={"selector": selector, "value": value},
    )
    if decision.decision == 'deny':
        return ActionResult(error=f"Blocked: {decision.reason}")
    return ActionResult(extracted_content=f"Filled {selector}")

@controller.action("Click element")
async def click_element(selector: str) -> ActionResult:
    decision = await veto.guard(
        tool="click_element",
        arguments={"selector": selector},
    )
    if decision.decision == 'deny':
        return ActionResult(error=f"Blocked: {decision.reason}")
    if decision.decision == 'require_approval':
        return ActionResult(
            error=f"Requires approval: {decision.approval_id}"
        )
    return ActionResult(extracted_content=f"Clicked {selector}")

agent = Agent(
    task="Check order status on the company portal",
    llm=ChatOpenAI(model=os.environ["OPENAI_MODEL"]),
    controller=controller,
    max_steps=25,
)

result = await agent.run()
print(result)

3. Define browser policies

veto/policies.yaml
version: "1.0"
name: Browser Use agent policies

rules:
  - id: whitelist-domains
    tools: [go_to_url]
    action: deny
    conditions:
      - field: arguments.url
        operator: not_matches
        value: "^https://(.*\\.approved\\.invalid|.*\\.google\\.invalid|.*\\.github\\.invalid)/.*"
    reason: "Navigation restricted to approved domains"

  - id: block-admin-paths
    tools: [go_to_url]
    action: deny
    conditions:
      - field: arguments.url
        operator: matches
        value: ".*/admin/.*|.*/internal/.*|.*/settings/.*"
    reason: "Admin and internal paths are off-limits"

  - id: block-password-fields
    tools: [input_text]
    action: deny
    conditions:
      - field: arguments.selector
        operator: matches
        value: ".*password.*|.*passwd.*|.*secret.*"
    reason: "Password fields cannot be filled by agents"

  - id: block-credit-card-fields
    tools: [input_text]
    action: deny
    conditions:
      - field: arguments.selector
        operator: matches
        value: ".*credit.*card.*|.*card.?number.*|.*cvv.*|.*cvc.*"
    reason: "Payment fields cannot be filled by agents"

  - id: approve-form-submissions
    tools: [click_element]
    action: require_approval
    conditions:
      - field: arguments.selector
        operator: matches
        value: ".*submit.*|.*purchase.*|.*checkout.*|.*pay.*|.*confirm.*"
    approval:
      timeout_minutes: 10
      notify: [ops@approved.example]

  - id: block-delete-buttons
    tools: [click_element]
    action: deny
    conditions:
      - field: arguments.selector
        operator: matches
        value: ".*delete.*|.*remove.*|.*cancel-account.*"
    reason: "Destructive actions blocked"

Before and after

Without Veto
before.py
import os
from browser_use import Agent, Controller
from langchain_openai import ChatOpenAI

controller = Controller()

agent = Agent(
    task="Book a flight from SFO to JFK for next Monday",
    llm=ChatOpenAI(model=os.environ["OPENAI_MODEL"]),
    controller=controller,
)

result = await agent.run()

The agent has unrestricted browser access. It can navigate anywhere, fill any form, click any button. No policy enforcement, no decision record.

With Veto
after.py
import os
from browser_use import Agent, Controller
from browser_use.agent.views import ActionResult
from langchain_openai import ChatOpenAI
from veto_sdk import Veto

veto = Veto(api_key=os.environ["VETO_API_KEY"])

controller = Controller()

@controller.action("Navigate to URL")
async def go_to_url(url: str) -> ActionResult:
    decision = await veto.guard(
        tool="go_to_url",
        arguments={"url": url},
    )

    if decision.decision == 'deny':
        return ActionResult(
            error=f"Navigation blocked: {decision.reason}"
        )

    if decision.decision == 'require_approval':
        return ActionResult(
            error=f"Navigation requires approval: {decision.approval_id}"
        )

    return ActionResult(extracted_content=f"Navigating to {url}")

@controller.action("Fill form field")
async def input_text(selector: str, value: str) -> ActionResult:
    decision = await veto.guard(
        tool="input_text",
        arguments={"selector": selector, "value": value},
    )

    if decision.decision == 'deny':
        return ActionResult(
            error=f"Input blocked: {decision.reason}"
        )

    return ActionResult(extracted_content=f"Filled {selector}")

@controller.action("Click element")
async def click_element(selector: str) -> ActionResult:
    decision = await veto.guard(
        tool="click_element",
        arguments={"selector": selector},
    )

    if decision.decision == 'deny':
        return ActionResult(
            error=f"Click blocked: {decision.reason}"
        )

    if decision.decision == 'require_approval':
        return ActionResult(
            error=f"Click requires approval: {decision.approval_id}"
        )

    return ActionResult(extracted_content=f"Clicked {selector}")

agent = Agent(
    task="Book a flight from SFO to JFK for next Monday",
    llm=ChatOpenAI(model=os.environ["OPENAI_MODEL"]),
    controller=controller,
)

result = await agent.run()

What Veto controls

Browser actionRiskVeto policy
Navigate to URLPhishing, internal admin accessDomain whitelisting, path blocking
Fill form fieldCredential leakage, PII exposureSelector-based blocking for password/card fields
Click elementPayment triggering, account deletionRequire approval for submit/purchase/delete
Extract contentData exfiltration, scraping PIIBlock extraction from sensitive pages
Take screenshotCapturing confidential screensBlock screenshots of financial/HR pages

Defense in depth for browser agents

1

URL allowlisting

Restrict navigation to a set of approved domains. Block admin panels, internal tools, and any domain not on the whitelist. Navigation requests that do not match your patterns are denied.

2

Credential protection

Block form fills on any field matching password, credit card, SSN, or API key selectors. Credential-entry requests are denied before they reach the browser.

3

Click approval

Route clicks on submit, purchase, delete, and confirm buttons to human approval. The browser pauses until a human reviews and approves the action. Payment buttons require explicit human consent before execution.

4

Audit governed actions

Each governed browser action is recorded with the URL, selector, input value, and policy decision. When something goes wrong, you have a decision record of what the agent did and why each action was allowed or blocked.

Frequently asked questions

How does Veto limit credential theft risk from browser agents?
Veto intercepts governed input_text actions before they reach the browser. Policies can block form fills on selectors matching password, credit card, SSN, or API key patterns. The agent receives a denial response while the governed call stays denied. If a malicious page uses a non-standard selector name, you can add custom patterns to your policy.
Can an agent still click a purchase button accidentally?
Define a require_approval policy for click actions on selectors matching purchase, checkout, pay, or confirm patterns. The browser agent pauses and routes the action to a human for approval. On the governed path, the click executes only after explicit human consent through the configured approval path or API.
How do I restrict which websites my agent can visit?
Create a URL allowlist policy that permits navigation to approved domains. Use glob patterns like https://*.approved.example/* to allow your properties while denying unapproved destinations on the governed path. Admin paths, internal tools, and unknown domains can be denied by default.
Does Veto protect against prompt injection via web pages?
Veto does not prevent prompt injection itself (that is an LLM-level concern), but it limits the damage. If a malicious page injects instructions that steer the agent toward harmful actions, Veto's policies still evaluate URL restrictions, credential protection, and click approvals on the governed path. Governed calls that violate your policies are denied regardless of what the LLM decides.
What is the performance impact on browser automation?
Runs in-process before dispatch. Browser actions: navigation, rendering, form fills: usually dominate the runtime budget; policy evaluation runs before dispatch and is measurable in your own environment.

Related integrations

Playwright

Playwright and Playwright MCP action policy

PydanticAI

Python callable authorization

MCP

Model Context Protocol tool authorization

Python preview

Native Python guardrails SDK

Secure your browser agents before they click the wrong button.

Sign upRead docsBook a call