Browser Automation Architecture

Screenshot, reason, act, verify. Every browser action passes through safety gates with human-in-the-loop confirmation before form submission.

Overview

The Pauhu Browser Automation system automates browser-based tasks on EU institutional portals - such as TED eProcurement, EUR-Lex, and national law databases - while maintaining strict safety controls. Unlike unguarded browser automation, every action passes through safety gates before execution. Destructive actions (form submissions, data entry) require explicit human approval.

The system operates in a continuous loop: it observes the current browser state via screenshot, reasons about the next action using an AI model, executes the action (subject to safety gates), and verifies the result. This loop continues until the task is complete or the user intervenes.

The automation loop

Step-by-step

1. Screenshot. The system captures a screenshot of the current browser viewport. It has no DOM access, only pixel-level observation.
2. Reason. An AI model analyses the screenshot together with the task description and action history. It determines the next action: what to click, what to type, where to scroll, or whether the task is complete.
3. Safety gate. Before execution, the proposed action is classified against four safety levels (see below). Prohibited actions are blocked. Required confirmations pause for human approval.
4. Act. The action is executed in the browser: a click at specific coordinates, text entry into a field, scrolling, navigation, or form submission.
5. Verify. A new screenshot is taken. The system compares the result against expected state. If the action failed or produced unexpected results, the loop can retry or escalate to the user.

Safety gates

Every action passes through safety gates before execution. The four levels control what the system may, must, must not, and need not do.

Human-in-the-loop flow

When an action triggers an Obligation gate (typically form submission), the system pauses and presents a confirmation modal. The user sees:

• All fields the system has filled, with their values
• The target form and portal
• Three options: Approve (proceed with submission), Edit (modify field values before submission), or Cancel (abort the action)

The system does not resume until the user makes an explicit choice. This ensures no data is submitted to external portals without human review.

Action types

The system supports 7 browser action types. Each maps to a specific safety gate level.

Gate classification can be elevated by context. For example, a click on a “Submit” button is reclassified from Permission to Obligation. A navigate to a domain not on the allowlist is reclassified from Permission to Prohibition (blocked).

Action lifecycle

Each action progresses through a defined state machine with 7 possible statuses.

Architecture components

ActionOverlay

SessionPanel

ConfirmationModal

ReplayViewer

Dashboard

SSE event stream

The automation executor streams real-time events to the frontend via Server-Sent Events (SSE). The connection reconnects automatically with exponential backoff (maximum 5 attempts).

Event types

Connection example

const source = new EventSource('/v1/cua/stream?session=' + sessionId);

source.addEventListener('action_pending', (e) => {
  const action = JSON.parse(e.data);
  // Show ActionOverlay at action.coordinates
});

source.addEventListener('form_confirmation', (e) => {
  const form = JSON.parse(e.data);
  // Show ConfirmationModal with form.fields
});

source.addEventListener('action_completed', (e) => {
  const result = JSON.parse(e.data);
  // Update SessionPanel with completed step
});

Session model

Each automation session targets a specific portal and task. Sessions maintain full state across the action lifecycle.

Session properties

Session example

{
  "id": "cua_sess_a1b2c3d4",
  "status": "active",
  "portal": "TED eProcurement",
  "taskDescription": "Search for procurement notices matching CPV 72000000 in Finland",
  "actions": [
    {
      "step": 0,
      "type": "navigate",
      "target": "https://ted.europa.eu/en/search",
      "status": "completed",
      "timestamp": "2026-03-10T14:00:01Z",
      "screenshot": "screenshots/step-0.png"
    },
    {
      "step": 1,
      "type": "type",
      "target": "input#search-query",
      "value": "CPV 72000000",
      "coordinates": { "x": 540, "y": 312 },
      "status": "completed",
      "timestamp": "2026-03-10T14:00:03Z",
      "screenshot": "screenshots/step-1.png"
    },
    {
      "step": 2,
      "type": "click",
      "target": "Country filter: Finland",
      "coordinates": { "x": 180, "y": 480 },
      "status": "pending",
      "timestamp": "2026-03-10T14:00:05Z"
    }
  ],
  "currentStep": 2,
  "createdAt": "2026-03-10T14:00:00Z",
  "updatedAt": "2026-03-10T14:00:05Z"
}

Pause and resume

Sessions can be paused at any time. When paused, the system stops proposing new actions but retains full state. Resuming continues from the current step. This is useful when the user needs to inspect intermediate results or perform manual steps.

Audit trail

Every automation action is logged with full provenance. The audit trail is immutable - entries cannot be modified or deleted after creation.

Audit record fields

Audit records are stored in EU jurisdiction and retained per GDPR data retention policies. Screenshots are stored alongside action metadata for complete session reconstruction.

Rollback capability

The system supports rollback to any previously completed step within a session. When a rollback is triggered:

1. All actions after the target step are marked as rolled_back
2. The browser navigates back to the state captured in the target step’s screenshot
3. The session’s currentStep is reset to the target step
4. The automation loop resumes from that point, proposing new actions based on the restored state

Rollback is non-destructive: rolled-back actions remain in the audit trail with their original timestamps and screenshots. The audit record shows that a rollback occurred, when it was triggered, and which step was the target.

Rollback limitations

• Submitted forms cannot be un-submitted. If a form was submitted (approved) and the user rolls back past that point, the system will flag that the portal state may be inconsistent.
• External state changes persist. Rollback restores the system’s internal session state but cannot undo changes already made on external portals.

Support

Technical: support@pauhu.eu

Sales: sales@pauhu.eu