Retrieval, evidence and AI search
Choose controls by their documented purpose.
An AI access policy needs named providers, crawler roles and product scopes. A user agent token, a Search inclusion setting and a snippet instruction are different controls. Review current official documentation before deciding what a rule is meant to allow or restrict.
Free learning from Semantec SEO.
Inventory the uses before choosing a control.
List the uses the owner wants to assess: ordinary Search discovery, Search generative features, model training, grounding in a named product and user requested retrieval. Do not collapse them into a single “AI bot” category. One provider can document different agents and controls for different purposes.
Record the exact token or setting, official source, documentation check date, affected product, intended decision and known exclusions. Match the control to the stated use. A crawler log entry shows a request, not necessarily the identity of a trustworthy provider or the downstream purpose of every fetched byte.
This page uses Google’s official documents checked on 23 October 2026 for a full worked policy review. It is not a robots.txt generator and makes no live access changes. Other providers require their own current official review; Google’s names and scopes do not establish their behavior.
Distinguish the documented Google roles.
Google’s common crawler documentation, updated 15 October 2026, describes Googlebot as a Search crawler. Its Search role is broader than AI features alone. Restricting it cannot be described as a training only choice. Relevant image, video and resource access can also affect how content is processed.
Google-Extended is a robots product token with no separate HTTP user agent string. The current common crawler documentation assigns it specified Gemini Apps and Vertex AI training and grounding uses, and says it does not affect Google Search inclusion or ranking. Existing Google crawlers perform the fetches to which this token’s stated choices apply.
Google-InspectionTool is associated with Search testing tools and does not itself determine Search inclusion. Google-CloudVertexBot concerns site owner requested Vertex agent crawling, rather than ordinary Google Search crawling. These distinctions prevent a familiar name from being treated as a universal Search or AI switch.
Common Google crawlers respect robots rules according to the documentation, but a claimed HTTP user agent can be spoofed. Where identity matters, follow the official verification guidance, such as supported IP range or reverse and forward DNS checks. A matching text label alone is insufficient evidence.
Keep Search inclusion and snippet controls in their own scope.
The current Search generative AI setting in Search Console offers Include, Exclude and Inherit from parent. Its documented coverage is AI Overviews, AI Mode and generative AI features in Discover. The control help page says it is separate from model training and from other Search inclusion and ranking.
The AI Search documentation now includes this setting in eligibility requirements. The older AI features page describes no additional technical requirements; record the dates and follow the newer linked specific control documentation for the inclusion choice. An account’s effective setting, permission and inherited parent state still need direct observation.
Content instructions such as nosnippet, data-nosnippet, max-snippet and noindex serve different presentation or indexing purposes. Robots crawling restrictions and indexing instructions are not interchangeable. If a crawler cannot fetch a page, it may not see an indexing instruction on that page. Review the documented effect before combining controls.
No control promises a citation, erases all previously processed information or governs an unrelated provider merely because it uses AI. Document propagation periods, cache limits and the need to check the effective result. Leave unknown behavior unresolved rather than extending a rule beyond its published scope.
Worked example: a Google policy decision table.
The fictional owner of a public course explanation wants ordinary Search discovery, wants the page eligible for Google Search generative features, and wants a separate decision about the specified Gemini and Vertex uses. This is a policy review example, not the observed configuration of this website.
The table assigns each intention to its documented control and records what must be checked. Search eligibility and inclusion remain separate from a Google-Extended decision. The owner has not yet approved a final choice for Gemini and Vertex, so that row remains Pending rather than silently inheriting a broad permission or block.
An authorized technical reviewer would inspect the relevant property setting and parent inheritance, the applicable robots file and response, current page instructions and crawler identity evidence. They would retain the exact effective state, time and responsible owner. This example performs none of those live checks and supplies no deployable robots rules.
If the property says Inherit, inspect the controlling parent choice before judging the outcome. If logs show a purported Google agent, verify identity before treating the request as provider evidence. If a setting changes, distinguish the saved choice from later propagated behavior; neither a single request nor a missing citation closes every check.
| Intention | Documented control and role | Decision or observed state | Boundary and next check |
|---|---|---|---|
| Ordinary Search discovery | Googlebot and applicable Search crawl/index requirements | Intend to retain; effective state Unknown | Review robots/resources/index instructions; no display guarantee |
| Search generative eligibility | Search Console Search generative AI inclusion | Intend Include; effective choice and inheritance Unknown | Inspect authorized property; covers documented Search/Discover features, not training |
| Specified Gemini/Vertex uses | Google-Extended robots product token; no separate HTTP UA | Owner decision Pending; current state Unknown | Read current stated training/grounding scope; no Search ranking/inclusion effect |
| Search diagnostic fetches | Google-InspectionTool | No live observation | Testing role; not the inclusion control |
| Owner requested Vertex agent crawl | Google-CloudVertexBot | Not part of this fictional Search objective | Separate named product review; do not equate with Googlebot |
Review other providers independently and maintain the record.
For another provider, repeat the inventory using that provider’s official crawler and product documentation. Check training, search discovery, on demand user retrieval and grounding separately where the provider distinguishes them. Do not translate Google-Extended into an assumed equivalent token or copy Google’s setting semantics.
Record documentation dates and source URLs, decision owner, effective policy, tests, unresolved scope and the next review trigger. A provider can change agents or products; a past table is not permanent proof of current behavior. Retest after relevant policy, hosting, robots, account or content changes.
Keep access policy distinct from content quality and confidentiality. A publicly accessible resource should not be treated as confidential merely because one cooperative crawler is restricted. Restrict sensitive information with appropriate access controls rather than relying on a broad claim that “AI is blocked”.
A complete review explains the intended use, the exact documented control, the observed state and its limitations. It does not infer universal AI exclusion or guaranteed discovery from a single token.
Sources and further reading.
The linked Google documents distinguish crawler roles, Search inclusion and content controls. The examples and decision records here are editorial teaching material, not observations from a live audit.
Continue learning
Choose the next related task.
Use the passage retrieval and answer systems hub. Return to the semantic SEO learning overview to choose another subject.