Updated ETGet the DI Daily by Email

Technology news for e-commerce professionals, with our take on why it matters.

Google publishes guide for letting Google-Agent past firewalls and bot blockers

Google publishes guide for letting Google-Agent past firewalls and bot blockers
Image: Google for Developers

The guide asks sites to let Google's agent reach add-to-cart, booking and API endpoints, and describes a signed-request experiment using Web Bot Auth.

Source: Google for Developers

Google published a help document explaining how sites can allow traffic from Google-Agent, which it describes as the name used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. Google says explicitly allowing this traffic in a CDN, web application firewall or bot-detection service helps prevent legitimate requests from being blocked while preserving existing security controls.

According to the document, Google-Agent requests carry a User-Agent header containing "Google-Agent" and come from IP addresses listed in user-triggered-agents.json. Google is also experimenting with the Web Bot Auth protocol, signing some requests with HTTP Message Signatures (RFC 9421) under the agent.bot.goog identity. Google recommends that verified requests skip bot challenges.

The troubleshooting section asks sites to make sure allow rules take priority over data center ASN blocks, managed bot rules and category-level AI agent blocks, and that they skip CAPTCHAs and JavaScript challenges. Google says allowing should cover on-site search, filtering, detail pages, add-to-cart, booking and XHR, GraphQL or api.* endpoints, and notes that many users' requests share one egress pool, which can trip per-IP rate limits.

Search Engine Roundtable noted that Google-Agent is not new, having covered it in March, and quoted Google saying the guide is meant to help publishers that want to enable Google-Agent.

Read the full article at Google for Developers

Our take

Google's guide shows its agent is expected to search, filter, add to cart and book on merchant sites, so default bot defenses may already be turning away Google-driven shoppers.

  • Decide deliberately whether to allow Google-Agent, then test it through checkout.
  • Ask your CDN or bot vendor whether it verifies Google-Agent and Web Bot Auth signatures.
  • Watch shared-IP rate limits, which Google warns can block legitimate agent traffic.

Was our take useful?

Key takeaways

  1. Google-Agent is identified by its User-Agent and user-triggered-agents.json IPs
  2. Google says verified agent requests should skip CAPTCHAs and bot challenges
  3. Google asks sites to allow agent access to add-to-cart, booking and APIs