We read the robots.txt of 30 leading UK fintechs. Almost none say anything about AI crawlers.

Methodology

The company list

The population is 30 companies from BusinessCloud, "FinTech 50 2025: UKs most innovative financial tech creators": the 2025 FinTech 50, BusinessClouds annual list of the UKs most innovative fintech companies, judged by an independent panel rather than ordered strictly by revenue. We fetched that list on 2026-07-14. This is an innovation-judged list, not a revenue ranking. It is the freely accessible published list available; firm-to-domain mapping was done by inspection, and a wrong domain would surface below as unreachable, never as a false directive. The list is judged on innovation, not company size or AI readiness, and nothing here should be read as one.

The fetch

On 2026-07-14 we requested https://<domain>/robots.txt and https://<domain>/llms.txt for each firm with curl, following redirects, and saved the exact response body for every firm as provenance. Three firms answered only on their www host, so we recorded the effective URL. Each firm's primary domain was mapped by inspection; a wrong domain would appear below as unreachable, never as a false directive.

Reading a file, and refusing to read a fake one

A response counts as a readable robots.txt only when its body is genuinely a robots file, that is, plain text carrying User-agent, Allow or Disallow lines. Several firms returned HTTP 200, 403 or 429 whose body was an HTML bot-protection challenge (Vercel, Cloudflare, Azure WAF, Incapsula). We record those as unreadable and never parse them for directives, because a challenge page is not a policy. The same discipline applies to llms.txt: a 200 whose body is an ordinary HTML page is a soft-404 and is marked as such, not counted as a genuine file.

How each token was classified

robots.txt was parsed with standard user-agent grouping: consecutive User-agent lines share the rules that follow them, until the next group. For each of the 12 AI tokens we then recorded:

• allow: the token is named and nothing disallows it.
• block: the token is named with Disallow: / and no Allow override, a full-site block request.
• partial: the token is named with some paths disallowed, or an allowlist pattern (Disallow: / alongside Allow: rules that re-permit specific sections).
• not mentioned: the token does not appear. The catch-all User-agent: * group, if any, still applies to it, but no AI-specific rule was set. This is not a block.

Those four labels describe syntax, deliberately. A token can be "partial" because it receives an allowlist, or because it shares the site's ordinary housekeeping exclusions; neither is evidence that the firm singled out AI. The judgement lives in a separate, derived field.

The stance judgement: stricter than Googlebot, or not

For each firm we derived one field, the AI-specific stance, computed from the archived raw files. The benchmark is Googlebot, the mainstream crawler every firm plainly wants. Following RFC 9309 semantics, each token obeys the group or groups naming it exactly; a token that is not named falls back to the User-agent: * group. We then ask one question: is any AI token's effective rule set clearly stricter than the same file's effective rule set for Googlebot? Bingbot sits in the per-token table because Copilot rides its index, but it is a mainstream search crawler, so it is not part of the AI-specific test.

• Restricts AI specifically: at least one AI token is fully disallowed while Googlebot is not equally disallowed. This is the only pattern we label a restriction.
• No AI-specific restriction: the AI tokens are unnamed (they inherit the same generic rules as every other unnamed crawler), or named with rules identical to or more permissive than Googlebot's. A restriction shared equally with Googlebot, or inherited from *, is a general crawling policy, not an AI decision.
• Indeterminate: we could not read the file, or the differences were neither clearly stricter nor clearly equal. Ambiguity is graded conservatively, never as a restriction.

Limitations

• This is a single snapshot on 2026-07-14. robots.txt and llms.txt change; the retained raw files fix what we saw on that date.
• The population is an innovation-judged list, not a random or complete sample of UK fintechs. It says nothing about companies outside this list.
• Firms behind bot protection (7 of 30) have an unreadable stance here. That is a finding about access, not evidence that they block or permit any crawler.
• robots.txt and llms.txt are voluntary requests. Compliance is at the operator's discretion, so a directive is a stated preference, not an enforced outcome, and we measure no outcome.
• We checked 12 named AI tokens. Other agents exist (for example Amazonbot, Bytespider, meta-externalagent, which appear in some of these files); they are out of scope and not counted.

The dataset behind this page is public: src/data/ai-crawler-index-fintech.json plus one raw file per company. Anyone can refetch a domain and check our reading against the saved response. If a company's file has changed since 2026-07-14, that is expected, and a fresh fetch is the way to confirm it. See also how we grade and the reproduction study.