Cross‑Engine Agentic Browser Pipeline: unifying CDP and WebDriver BiDi with UA and Client‑Hints harmony, What Is My Browser Agent validation, and a smart browser agent switcher

Auto‑agent browsers are crossing the threshold from demos to production. They scrape, transact, and orchestrate multi‑step flows at human speed or better. The problem: under the hood the web is a minefield of protocol divergence, feature flags, counter‑automation heuristics, and subtle identity leaks. If your agent says it is Chrome on macOS, your headers, JS surfaces, and TLS fingerprint must agree — and your control plane must adapt across Chrome, Firefox, and WebKit.

This article proposes a practical, opinionated architecture and set of techniques to build a single control plane that:

• Runs Chrome, Firefox, and WebKit using CDP and WebDriver BiDi under one API.
• Keeps User‑Agent and Client Hints in harmony across network and JS surfaces.
• Auto‑switches browser agent per site with a policy engine and telemetry feedback.
• Verifies reality with a what is my browser agent validation loop using both public and self‑hosted echo endpoints.
• Ships reproducible tests for auto‑agent AI browsers so you can catch regressions deterministically.

The target audience: engineers shipping agentic browsing or site automation at scale, who need cross‑engine reach without being detected for trivial inconsistencies.

TL;DR

• Unify transport with an adapter layer that abstracts CDP, WebDriver BiDi, and classic WebDriver. Prefer BiDi where stable, fall back to CDP for Chrome‑only features, and classic WebDriver for WebKit gaps.
• Define a BrowserAgent profile that binds UA string, UA‑CH request headers, JS surfaces for Navigator and NavigatorUAData, and platform constraints. Never set these piecemeal.
• Implement a harmonizer that sets correct headers in network requests and injects preload scripts to align JS surfaces. For Chrome, use CDP Network.setUserAgentOverride with userAgentMetadata; for others, rely on request interception and script injection.
• Build a validation harness that navigates to echo endpoints and a self‑hosted agent page, capturing headers and JS values; assert they match the active BrowserAgent profile.
• Ship a Smart Switcher that uses per‑site rules, heuristics, and feedback signals to pick the best engine and agent profile. Persist outcomes and promote them to rules.
• Make runs reproducible by pinning browser versions, freezing time, seeding random, recording network and console traces, and snapshotting agent profiles.

1. Why a cross‑engine agentic browser pipeline now

• The CDP vs WebDriver BiDi transition is real. Chrome is migrating functionality into BiDi; Firefox leads with BiDi; WebKit is adding BiDi while classic WebDriver remains a mainstay. The safest path is to support both protocols simultaneously.
• User Agent Client Hints (UA‑CH) complicate identity. Many sites compare the legacy UA string with UA‑CH. Mismatches lead to subtle feature downgrades or active blocking.
• Detection is more than UA. JS surfaces (navigator, userAgentData), request headers, and even connection behavior need to tell the same story.
• Headless is much better than it used to be, especially with Chrome’s new headless mode, but you still need deliberate consistency.

2. Primer: CDP vs WebDriver BiDi vs classic WebDriver

• Chrome DevTools Protocol (CDP)

  • Low‑level, Chrome‑first. Rich domains: Network, Page, Runtime, Emulation, Fetch, etc.
  • Lets you override UA and UA‑CH via Network.setUserAgentOverride.
  • Unofficial across engines; Chromium‑based variants vary a bit.

• WebDriver BiDi

  • W3C standard. Bidirectional, event‑rich, supports sandboxed script injection, network intercept, and logging.
  • Implemented in Firefox and Chromium; Safari is in progress with growing support.
  • Cleaner multi‑engine story but some features lag CDP.

• Classic W3C WebDriver

  • Request‑response, stable, supported everywhere. Missing certain modern debugging and network manipulation features.

Opinion: default to BiDi when available; use CDP for Chrome specifics and performance; keep classic WebDriver for WebKit gaps. Hide all of this behind an adapter API.

3. User Agent and Client Hints: consistency doctrine

A browser’s identity leaks across layers:

• Legacy UA string in the User‑Agent header and navigator.userAgent.
• UA‑CH request headers like Sec‑CH‑UA, Sec‑CH‑UA‑Full‑Version‑List, Sec‑CH‑UA‑Mobile, Sec‑CH‑UA‑Platform, and Accept‑CH negotiation.
• JS surfaces: navigator.userAgentData, navigator.platform, navigator.vendor, screen metrics, and reduced‑fingerprinting quirks.

Rules of thumb:

• Pick a real world profile per engine and version; do not invent unrealistic brands or versions.
• Make the UA string and UA‑CH consistent with each other and with the engine. For example, Firefox does not implement UA‑CH like Chromium does; do not send Chromium‑style UA‑CH from Firefox.
• JS and network must agree. If headers say Android Chrome but navigator.platform returns MacIntel, you will trip detectors.
• Consider privacy features. Some engines randomize brands or reduce JS entropy. Be explicit about your expectations when you create a profile.

4. Threat model and non‑goals

• We are not trying to defeat sophisticated anti‑bot ML models with full TLS and canvas spoofing. The goal is to avoid self‑inflicted inconsistencies that cause avoidable blocks.
• We do not advise forging engines. Do not pretend to be Safari from Chrome; instead, run real WebKit when you need Safari identity.

5. Architecture: a single control plane

Components:

• Transport adapters: cdpp (CDP), bidi (WebDriver BiDi), webdriver (classic). Each implements a common set of capabilities: navigation, script injection, network intercept, header override, page events, screenshot, storage control.
• Session manager: creates, tracks, and tears down browser contexts and pages across engines. Exposes engine capabilities via feature flags.
• BrowserAgent profiles: canonical records of UA, UA‑CH, JS properties, and feature toggles per engine channel.
• Harmonizer: sets headers and JS surfaces at session start; keeps them in sync on navigation, new tabs, service workers, and subresources.
• Validator: a harness that navigates to echo pages and asserts the pipeline produced consistent identity.
• Smart Switcher: a policy engine to choose engine and profile per site, using rules plus telemetry from previous runs.
• Recorder and replayer: for reproducible tests.

6. BrowserAgent profile model

Example fields:

• id, label, engine: chrome|firefox|webkit, channel: stable|beta|dev|technology-preview
• os: win|mac|linux|android|ios, device: desktop|mobile, mobileModel, arch
• userAgent: legacy UA string
• uaCh: brands[], fullVersionList[], platform, platformVersion, mobile, model, wow64 flag
• jsSurface: navigator overrides (userAgent, platform, vendor), userAgentData emulation payload, language prefs, time zone
• display: screen size, device scale, DPR
• transportHints: requires cdp override vs script override, needs request interception, allow insecure certs

Opinion: Store these profiles as data files under version control. Keep one source of truth per engine version.

7. Implementation sketch: TypeScript control plane

Below is illustrative code. It favors clarity over exhaustiveness. The actual libraries you plug in may differ; the patterns are the transferable part.

ts
// core/types.ts
export type Engine = 'chrome' | 'firefox' | 'webkit'
export type Transport = 'cdp' | 'bidi' | 'webdriver'

export interface BrowserAgent {
  id: string
  engine: Engine
  label: string
  os: 'win' | 'mac' | 'linux' | 'android' | 'ios'
  device: 'desktop' | 'mobile'
  userAgent: string
  uaCh?: {
    brands?: { brand: string; version: string }[]
    fullVersionList?: { brand: string; version: string }[]
    mobile?: boolean
    platform?: string
    platformVersion?: string
    model?: string
    wow64?: boolean
  }
  jsSurface?: {
    navigatorPlatform?: string
    navigatorVendor?: string
    language?: string
    languages?: string[]
    timezone?: string
    userAgentData?: {
      brands: { brand: string; version: string }[]
      mobile: boolean
      platform: string
      platformVersion?: string
      architecture?: string
      bitness?: string
      model?: string
      wow64?: boolean
      fullVersionList?: { brand: string; version: string }[]
    }
  }
  display?: { width: number; height: number; dpr: number }
  transportHints?: { prefer: Transport; allowFallback: boolean }
}

A Chrome stable desktop on Windows 11, Chromium 120‑ish, example profile:

ts
export const CHROME_WIN_STABLE: BrowserAgent = {
  id: 'chrome-win-stable-120',
  engine: 'chrome',
  label: 'Chrome 120 on Windows 11',
  os: 'win',
  device: 'desktop',
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
  uaCh: {
    brands: [
      { brand: 'Not A;Brand', version: '99' },
      { brand: 'Chromium', version: '120' },
      { brand: 'Google Chrome', version: '120' },
    ],
    fullVersionList: [
      { brand: 'Not A;Brand', version: '99.0.0.0' },
      { brand: 'Chromium', version: '120.0.6099.71' },
      { brand: 'Google Chrome', version: '120.0.6099.71' },
    ],
    mobile: false,
    platform: 'Windows',
    platformVersion: '15.0.0',
    wow64: false,
  },
  jsSurface: {
    navigatorPlatform: 'Win32',
    navigatorVendor: 'Google Inc.',
    language: 'en-US',
    languages: ['en-US', 'en'],
    timezone: 'America/Los_Angeles',
    userAgentData: {
      brands: [
        { brand: 'Not A;Brand', version: '99' },
        { brand: 'Chromium', version: '120' },
        { brand: 'Google Chrome', version: '120' },
      ],
      mobile: false,
      platform: 'Windows',
      platformVersion: '15.0.0',
      fullVersionList: [
        { brand: 'Not A;Brand', version: '99.0.0.0' },
        { brand: 'Chromium', version: '120.0.6099.71' },
        { brand: 'Google Chrome', version: '120.0.6099.71' },
      ],
    },
  },
  display: { width: 1368, height: 768, dpr: 1 },
  transportHints: { prefer: 'cdp', allowFallback: true },
}

Note: numbers are illustrative; keep them consistent with the actual browser binary you launch. You can query the real UA and UA‑CH from a virgin session and persist them in your profile library.

Transport adapters

The adapters expose a common interface. Examples below show enough detail to convey the approach.

ts
// core/adapter.ts
export interface Adapter {
  open(browserPath?: string): Promise<void>
  newContext(opts?: any): Promise<void>
  newPage(url?: string): Promise<Page>
  setUserAgent(agent: BrowserAgent): Promise<void>
  setClientHints(agent: BrowserAgent): Promise<void>
  injectPreload(script: string): Promise<void>
  setRequestInterception(enabled: boolean): Promise<void>
  onRequest(cb: (req: InterceptedRequest) => Promise<void>): void
  onEvent(event: string, cb: (...args: any[]) => void): void
  close(): Promise<void>
}

Chrome CDP highlights:

• Use Network.setUserAgentOverride with userAgentMetadata for UA and UA‑CH harmony.
• Use Emulation.setLocaleOverride, Emulation.setTimezoneOverride for language and timezone.
• Inject a preload script to override navigator.userAgentData only when needed; CDP metadata usually suffices for JS.

ts
// adapters/cdp.ts
import { CDP } from './vendor-cdp'

export class CdpAdapter implements Adapter {
  /* ... connection plumbing ... */
  async setUserAgent(agent: BrowserAgent) {
    await this.cdp.Network.setUserAgentOverride({
      userAgent: agent.userAgent,
      acceptLanguage: agent.jsSurface?.language ?? 'en-US',
      platform: agent.jsSurface?.navigatorPlatform ?? 'Win32',
      userAgentMetadata: agent.uaCh ? {
        brands: agent.uaCh.brands,
        fullVersionList: agent.uaCh.fullVersionList,
        platform: agent.uaCh.platform,
        platformVersion: agent.uaCh.platformVersion,
        architecture: 'x86',
        model: agent.uaCh.model ?? '',
        mobile: !!agent.uaCh.mobile,
        wow64: !!agent.uaCh.wow64,
      } : undefined,
    })
  }

  async setClientHints(_agent: BrowserAgent) {
    // CDP UA metadata covers JS UA‑CH; request headers will be emitted automatically on eligible requests.
  }

  async injectPreload(script: string) {
    await this.cdp.Page.addScriptToEvaluateOnNewDocument({ source: script })
  }
}

Firefox BiDi highlights:

• No built‑in UA metadata override like CDP. Use network interception to set headers: User‑Agent plus Sec‑CH‑UA family when appropriate.
• Use script.addPreloadScript to patch navigator.userAgent and navigator.userAgentData.

ts
// adapters/bidi.ts
import { BiDi } from './vendor-bidi'

export class BidiAdapter implements Adapter {
  async setUserAgent(agent: BrowserAgent) {
    // No direct BiDi UA override; rely on request header override.
    await this.setRequestInterception(true)
    this.onRequest(async req => {
      const headers = { ...req.headers }
      headers['user-agent'] = agent.userAgent
      if (agent.uaCh && agent.engine === 'chrome') {
        headers['sec-ch-ua'] = formatBrands(agent.uaCh.brands)
        headers['sec-ch-ua-full-version-list'] = formatBrands(agent.uaCh.fullVersionList)
        headers['sec-ch-ua-mobile'] = agent.uaCh.mobile ? '?1' : '?0'
        headers['sec-ch-ua-platform'] = quote(agent.uaCh.platform ?? 'Windows')
        if (agent.uaCh.platformVersion) headers['sec-ch-ua-platform-version'] = quote(agent.uaCh.platformVersion)
      }
      await req.continue({ headers })
    })
  }

  async injectPreload(script: string) {
    await this.bidi.script.addPreloadScript({ functionDeclaration: script })
  }
}

function formatBrands(list?: { brand: string; version: string }[]) {
  if (!list) return ''
  return list.map(b => `"${b.brand}";v="${b.version}"`).join(', ')
}
function quote(s: string) { return `"${s}"` }

WebKit classic WebDriver highlights:

• Classic WebDriver has limited built‑in header control. Use a proxy layer to inject headers, or a devtools bridge where available, and preload scripts for JS surfaces.
• Safari has ongoing WebDriver BiDi support; integrate it as it stabilizes to reduce reliance on proxies.

JS surface harmonization

Preload script to align navigator and userAgentData. Only apply when the engine does not natively present the chosen values.

js
// scripts/harmonize.js
(function installNavigatorOverrides() {
  const ua = window.__agent_userAgent
  const platform = window.__agent_platform
  const vendor = window.__agent_vendor
  const uaData = window.__agent_userAgentData

  const defineRO = (obj, prop, value) => {
    try { Object.defineProperty(obj, prop, { get: () => value, configurable: false }) } catch {}
  }

  if (ua) defineRO(Navigator.prototype, 'userAgent', ua)
  if (platform) defineRO(Navigator.prototype, 'platform', platform)
  if (vendor) defineRO(Navigator.prototype, 'vendor', vendor)

  if (uaData && 'userAgentData' in Navigator.prototype) {
    const data = uaData
    const brands = data.brands || []
    const fullVersionList = data.fullVersionList || brands
    const uaDataObj = {
      brands: brands.slice(),
      mobile: !!data.mobile,
      platform: data.platform,
      getHighEntropyValues: async (hints) => {
        const out = { brands, mobile: !!data.mobile, platform: data.platform }
        for (const h of hints || []) {
          if (h === 'architecture' && data.architecture) out.architecture = data.architecture
          if (h === 'bitness' && data.bitness) out.bitness = data.bitness
          if (h === 'model' && data.model) out.model = data.model
          if (h === 'platformVersion' && data.platformVersion) out.platformVersion = data.platformVersion
          if (h === 'uaFullVersion' && fullVersionList[1]) out.uaFullVersion = fullVersionList[1].version
          if (h === 'fullVersionList') out.fullVersionList = fullVersionList
        }
        return out
      }
    }
    defineRO(Navigator.prototype, 'userAgentData', uaDataObj)
  }
})()

At session start, inject a bootstrap that defines window._agent* variables to desired values before harmonize.js runs.

8. Harmonizer: from profile to reality

The harmonizer composes three steps:

1. Network layer

• CDP: set Network.setUserAgentOverride including UA‑CH metadata.
• BiDi: enable interception, add UA and CH headers. Respect Accept‑CH negotiation if you want to emulate gradual hints delivery; or simply include widely accepted hints for navigation and subresource requests.
• WebKit classic: use an upstream proxy to inject headers when needed.

2. JS layer

• Preload harmonize.js, with early injection on every new document, including iframes and worker contexts if supported.
• Set language and timezone via protocol APIs where available; otherwise, patch Intl in JS as a last resort.

3. Display and device metrics

• Set viewport and device scale via protocol APIs so CSS media queries align with agent identity.

Edge cases to handle:

• Service workers: ensure they see the intended headers at initial registration and update checks.
• Cross‑origin iframes: preload injection may not apply; rely on network layer to keep headers consistent.
• Redirect chains: interception must persist across redirects.

9. Validation with what is my browser agent

Relying solely on your own config is risky. Build a validation loop that confirms the actual world view.

Checklist of sources to query per run:

• Local self‑hosted echo page that reports:
  • request headers as seen by the server
  • navigator.userAgent, navigator.platform, navigator.vendor
  • navigator.userAgentData fields and getHighEntropyValues output
  • timezone, language, screen metrics
• Public mirrors (expect variability):
  • httpbin.org headers endpoint
  • client‑hints echo services
  • whatismybrowser.com or similar UA inspector

Capture the outputs and compare against the active BrowserAgent. Fail fast if mismatched.

Example harness flow:

ts
// validator.ts
export async function validateIdentity(adapter: Adapter, agent: BrowserAgent) {
  const page = await adapter.newPage('https://your-echo.local/agent')
  const result = await page.evaluate(() => {
    return {
      headers: window.__serverHeaders || {},
      js: {
        ua: navigator.userAgent,
        platform: navigator.platform,
        vendor: (navigator as any).vendor,
        uaData: (navigator as any).userAgentData ? {
          brands: (navigator as any).userAgentData.brands,
          platform: (navigator as any).userAgentData.platform,
          mobile: (navigator as any).userAgentData.mobile,
        } : null,
      },
    }
  })

  // Assertions
  assertEqual(result.headers['user-agent'], agent.userAgent)
  if (agent.uaCh && agent.engine === 'chrome') {
    assertContains(result.headers, 'sec-ch-ua')
    assertContains(result.headers, 'sec-ch-ua-platform')
  }
  assertEqual(result.js.ua, agent.userAgent)
  if (agent.jsSurface?.navigatorPlatform) assertEqual(result.js.platform, agent.jsSurface.navigatorPlatform)
  if (agent.jsSurface?.navigatorVendor) assertEqual(result.js.vendor, agent.jsSurface.navigatorVendor)
}

Keep the echo page simple and stable. You can build it as an Express app that also reflects the raw headers back to the client under window.__serverHeaders via a tiny inline script.

10. Smart Browser Agent Switcher

A policy engine chooses the best engine and BrowserAgent profile per site. Sources of truth:

• Static rules: domain patterns mapped to engines and profiles. Example: bank.example.com prefers WebKit due to stricter fingerprinting; news.example.com allows Chromium headless.
• Capabilities matrix: does the site require HLS DRM playback, WebAuthn, or a specific media pipeline? Choose an engine with native support.
• Telemetry: recent failures, CAPTCHAs served, blocked requests, or inflated TTFB. Downrank the agent that triggered issues.
• Performance budget: target time to interactive; choose the engine profile with fastest page load for that site from historical data.

Rules DSL example:

yaml
# rules.yaml
- match: '*.bank.example.com'
  prefer:
    engine: webkit
    agent: safari-mac-stable-17
  fallback:
    - engine: firefox
    - engine: chrome
  notes: Strong UA validation and script injection restrictions.

- match: '*.maps.example.com'
  prefer:
    engine: chrome
    agent: chrome-win-stable-120
  flags:
    allowWebGL: true
    mobileUA: false

- match: '*.mobi.example.com'
  prefer:
    engine: chrome
    agent: chrome-android-pixel7-120
  notes: Requires touch events and mobile UA‑CH.

Online selection algorithm (simplified):

ts
export async function selectAgentForSite(url: string, history: TelemetryDB): Promise<BrowserAgent> {
  const domain = extractDomain(url)
  const rule = findRule(domain)
  const candidates = enumerateCandidates(rule)
  const scored = await Promise.all(candidates.map(async c => ({
    agent: c,
    score:  weightRule(c, rule)
          + weightTelemetry(c, history[domain])
          + weightPerf(c, history.perf[domain])
  })))
  scored.sort((a, b) => b.score - a.score)
  return scored[0].agent
}

Persist telemetry per domain: number of blocks, error codes, CAPTCHA frequency, median TTI. Promote winners to static rules as stability improves.

11. Reproducible tests for auto‑agent browsers

The hardest bugs are flakey. Make a run reproducible:

• Pin browser versions. Use Chrome for Testing, Firefox ESR or pinned beta, and Safari Technology Preview when needed. Manage via container images.
• Freeze time. Set fixed timezone and override Date.now via protocol emulation or preload shim for JS only checks.
• Seed randomness. Ensure Math.random shims to a seeded PRNG in your preload script when determinism is needed for snapshots.
• Record network. Persist HAR, request headers, response codes, and redirect chains.
• Capture console and protocol logs. Store CDP or BiDi traces and console output.
• Snapshot BrowserAgent profiles and rule set used for the run.
• Capture DOM or visual snapshots at checkpoints.

A minimal harness can export a run bundle: profile.json, rules.yaml, har.json, trace.log, screenshots, and a replay script that takes the same inputs and re‑drives the flow with network replay if feasible.

12. Handling service workers, prerender, and CH delegation

• Service workers: register before you finalize UA and CH and you may leak the wrong identity. Always harmonize before the first navigation or worker registration. If using request interception, make sure worker requests are covered.
• CH delegation: sites may require Client Hints only after sending Accept‑CH. To emulate that faithfully, store Accept‑CH preferences per origin and inject UA‑CH only after you have seen the hint negotiation. In practice, many automation stacks send a stable subset up front for consistency; choose based on your fidelity requirements.
• Prerender and prefetch: Chrome may issue requests from prerendered contexts. Ensure interception filters include these; otherwise, inconsistent headers show up in logs.

13. Capabilities matrix and caveats across engines

• UA‑CH

  • Chromium: full UA‑CH support. CDP userAgentMetadata affects both network and JS surfaces.
  • Firefox: UA‑CH is limited; userAgentData is not widely exposed. Do not send Chromium‑style UA‑CH unless the site explicitly requests and Firefox supports it; safer to omit UA‑CH and rely on UA string.
  • WebKit: partial and evolving. Prefer fewer hints and consistent UA string; align JS surfaces via preload when necessary.

• Network interception

  • CDP Fetch and Network enable clean request modification.
  • BiDi network intercept is supported in Firefox and Chromium; check versions for specifics.
  • Classic WebDriver needs a proxy to adjust headers in flight.

• Preload injection

  • BiDi has script preload hooks; CDP has addScriptToEvaluateOnNewDocument. Security contexts can block overrides on some high‑security pages; rely on network harmony when JS patches are prevented.

• Headless behavior

  • Chrome new headless closely matches headed; still ensure fonts and GPU features are configured if sites rely on them.
  • Firefox headless differs slightly in window metrics; set viewport explicitly.

14. Observability: know when you drift

• Diff expected vs observed identity on every session: a short text report attached to logs.
• Tag runs with engine, profile id, and rule id. Include them in all metrics and traces.
• Add health checks: at the start of a session, visit the local echo; if harmony fails, remediate or switch agent before visiting target sites.

15. Security and compliance

• Be honest about the engine. If you need Safari identity, run WebKit. Misrepresenting engines risks compliance and legal exposure.
• Store only non‑sensitive telemetry. Avoid recording PII in HARs and console logs; redact with allowlists.
• Respect robots and site terms. Agentic browsing for QA on your own properties is different from third‑party data extraction.

16. Putting it together: end‑to‑end flow

1. Select an agent with the Smart Switcher for a target URL.
2. Start the chosen adapter (CDP, BiDi, or WebDriver) and browser binary version pinned in config.
3. Apply harmonizer: set UA and CH; inject preload; set locale, time zone, and viewport.
4. Validate on self‑hosted echo; if mismatched, attempt remediation or choose a fallback agent.
5. Navigate to the target site; execute the agentic plan.
6. Capture traces, snapshots, and results; update telemetry.
7. In CI, replay with the same profile and rules; compare outputs deterministically.

17. Example: minimal bootstrap

ts
import { CdpAdapter } from './adapters/cdp'
import { BidiAdapter } from './adapters/bidi'
import { CHROME_WIN_STABLE } from './profiles/chrome'
import { selectAgentForSite } from './switcher'
import { validateIdentity } from './validator'

async function run(url: string) {
  const agent = await selectAgentForSite(url, loadTelemetry())
  const adapter = agent.engine === 'chrome' && agent.transportHints?.prefer === 'cdp'
    ? new CdpAdapter()
    : new BidiAdapter()

  await adapter.open()
  await adapter.newContext()
  await adapter.setUserAgent(agent)

  // Provide variables for harmonize.js
  const bootstrap = `
    window.__agent_userAgent = '${agent.userAgent.replace(/'/g, "\\'")}'
    window.__agent_platform = '${agent.jsSurface?.navigatorPlatform || ''}'
    window.__agent_vendor = '${agent.jsSurface?.navigatorVendor || ''}'
    window.__agent_userAgentData = ${JSON.stringify(agent.jsSurface?.userAgentData || {})}
  `
  await adapter.injectPreload(bootstrap)
  await adapter.injectPreload(await fs.promises.readFile('scripts/harmonize.js', 'utf8'))

  await validateIdentity(adapter, agent)
  const page = await adapter.newPage(url)
  // ... run your agent plan ...
}

18. Data‑driven profile generation

Rather than hand‑craft every profile, build a gatherer that runs pristine sessions of each engine and records identity:

• Start engine with default profile and no overrides.
• Visit your echo page plus a public UA inspector.
• Record UA, JS surfaces, UA‑CH headers, and screen metrics.
• Persist as a candidate profile; hand‑review and annotate transportHints and display.
• Use these real world profiles as the base, then create lightweight variants for regional locales or mobile vs desktop.

19. Roadmap and future‑proofing

• WebDriver BiDi growth: as Safari and Chromium complete support for network intercept and script preload, reduce reliance on CDP and proxies.
• UA‑CH evolution: hints and privacy budgets are evolving; keep your harmonizer modular to adapt quickly.
• Trust tokens and WebAuthn: if your agents interact with these, align platform and authenticator capabilities with your chosen engine and OS surface.

20. Practical pitfalls and how to avoid them

• Forgetting subresources: you harmonized the navigation request but not images, XHR, or fetch. Ensure interception applies to all resource types.
• Mismatched locales: Accept‑Language header says en‑US but Intl.DateTimeFormat returns a different locale. Set both via protocol and preload shim.
• Version drift: your UA string claims Chrome 120 but the binary is 121. Pin versions and regenerate profiles when you update bins.
• WebKit surprises: classic WebDriver cannot set headers natively. Use an upstream proxy to enforce headers, or prefer the BiDi path as WebKit lands features.
• Over‑eager CH: sending UA‑CH headers that the engine would not normally advertise can raise flags. Match the engine’s real behavior or negotiate via Accept‑CH.

21. Final checklist

• Transport unification
  • Implement adapters for CDP, BiDi, and classic WebDriver. Feature detect per session.
• BrowserAgent profiles
  • One source of truth for UA, UA‑CH, JS surfaces, display, and locale.
• Harmonizer
  • Network overrides plus JS preload. Handle workers and iframes where possible.
• Validation
  • Self‑hosted echo page and public endpoints. Fail on mismatch.
• Smart Switcher
  • Rules plus telemetry and performance signals. Persist outcomes.
• Reproducibility
  • Pinned engines, frozen time, seeded randomness, HAR and traces, snapshots.

Build this once, and you will have a durable platform that keeps your AI browsers honest, reduces detection due to self‑inflicted inconsistencies, and gives you the knobs to adapt rapidly as engine capabilities evolve.

References and further reading

• WebDriver BiDi specification: w3c.github.io/webdriver-bidi/
• Chrome DevTools Protocol: chromedevtools.github.io/devtools-protocol/
• User Agent Client Hints: w3c.github.io/client-hints-infrastructure/
• Chrome for Testing: developer.chrome.com/blog/chrome-for-testing/
• Headless new mode overview: developer.chrome.com/articles/new-headless/
• Firefox WebDriver BiDi docs: firefox-source-docs.mozilla.org/testing/webdriver/bidi/
• Safari WebDriver and automation: developer.apple.com/documentation/webkit/testing_with_webdriver