We read the robots.txt of every top-50 UK law firm. Almost none say anything about AI crawlers.

Methodology

The firm list

The population is the top 50 firms from Lawyer Mag (lawyermag.co.uk), "Law Firm Rankings: 100 Leading Law Firms in UK in 2025": the top 100 UK law firms ranked primarily by revenue for 2025, of which we take the top 50. We fetched that ranking on 2026-07-14. This is a secondary compilation, not a primary revenue audit. The Lawyer UK 200 and Legal Business LB100 are the primary rankings but are paywalled. Firm-to-domain mapping was done by inspection; a wrong domain would surface below as unreachable, not as a false directive. The two primary rankings in this field, The Lawyer UK 200 and the Legal Business LB100, are the stronger sources but sit behind paywalls, so we used a freely accessible published compilation and state that plainly. The list orders firms by revenue; it is not a ranking of AI readiness, and nothing here should be read as one.

The fetch

On 2026-07-14 we requested https://<domain>/robots.txt and https://<domain>/llms.txt for each firm with curl, following redirects, and saved the exact response body for every firm as provenance. Three firms answered only on their www host, so we recorded the effective URL. Each firm's primary domain was mapped by inspection; a wrong domain would appear below as unreachable, never as a false directive.

Reading a file, and refusing to read a fake one

A response counts as a readable robots.txt only when its body is genuinely a robots file, that is, plain text carrying User-agent, Allow or Disallow lines. Several firms returned HTTP 200, 403 or 429 whose body was an HTML bot-protection challenge (Vercel, Cloudflare, Azure WAF, Incapsula). We record those as unreadable and never parse them for directives, because a challenge page is not a policy. The same discipline applies to llms.txt: a 200 whose body is an ordinary HTML page is a soft-404 and is marked as such, not counted as a genuine file.

How each token was classified

robots.txt was parsed with standard user-agent grouping: consecutive User-agent lines share the rules that follow them, until the next group. For each of the 12 AI tokens we then recorded:

• allow: the token is named and nothing disallows it.
• block: the token is named with Disallow: / and no Allow override, a full-site block request.
• partial: the token is named with some paths disallowed, or an allowlist pattern (Disallow: / alongside Allow: rules that re-permit specific sections).
• not mentioned: the token does not appear. The catch-all User-agent: * group, if any, still applies to it, but no AI-specific rule was set. This is not a block.

Those four labels describe syntax, deliberately. A token can be "partial" because it receives an allowlist, or because it shares the site's ordinary housekeeping exclusions; neither is evidence that the firm singled out AI. The judgement lives in a separate, derived field.

The stance judgement: stricter than Googlebot, or not

For each firm we derived one field, the AI-specific stance, computed from the archived raw files. The benchmark is Googlebot, the mainstream crawler every firm plainly wants. Following RFC 9309 semantics, each token obeys the group or groups naming it exactly; a token that is not named falls back to the User-agent: * group. We then ask one question: is any AI token's effective rule set clearly stricter than the same file's effective rule set for Googlebot? Bingbot sits in the per-token table because Copilot rides its index, but it is a mainstream search crawler, so it is not part of the AI-specific test.

• Restricts AI specifically: at least one AI token is fully disallowed while Googlebot is not equally disallowed. This is the only pattern we label a restriction.
• No AI-specific restriction: the AI tokens are unnamed (they inherit the same generic rules as every other unnamed crawler), or named with rules identical to or more permissive than Googlebot's. A restriction shared equally with Googlebot, or inherited from *, is a general crawling policy, not an AI decision.
• Indeterminate: we could not read the file, or the differences were neither clearly stricter nor clearly equal. Ambiguity is graded conservatively, never as a restriction.

Limitations

• This is a single snapshot on 2026-07-14. robots.txt and llms.txt change; the retained raw files fix what we saw on that date.
• The population is a revenue ranking, not a random or complete sample of UK firms. It says nothing about firms outside the top 50.
• Firms behind bot protection (5 of 50) have an unreadable stance here. That is a finding about access, not evidence that they block or permit any crawler.
• robots.txt and llms.txt are voluntary requests. Compliance is at the operator's discretion, so a directive is a stated preference, not an enforced outcome, and we measure no outcome.
• We checked 12 named AI tokens. Other agents exist (for example Amazonbot, Bytespider, meta-externalagent, which appear in some of these files); they are out of scope and not counted.

Why this index matters for competitive positioning

Law firm brand visibility in AI assistants is increasingly a client acquisition and business development channel. When in-house counsel, procurement teams, or prospective clients query ChatGPT or Perplexity for firm recommendations, recommendations, or case law analysis, firms that appear in those results gain an advantage over competitors who do not. This index reveals that 49 of the top 50 law firms by revenue are either not restricting AI access at all, or are doing so generically alongside other crawlers. Only 1 enforce AI-specific restrictions. For firms competing on thought leadership and visibility, this means most market leaders remain indexed and citable. A firm that restricts AI crawlers while all its peers remain visible is potentially disadvantaging itself, while a firm that remains citable while competitors block gains a structural advantage. The most credible law firms (those featured in Legal500, Chambers, The Lawyer) are already cited by AI systems regardless of robots.txt, because those rankings are licensed data sources. The real risk for mid-market and smaller firms is invisibility, not over-citation. This index is a tool for assessing your own competitive positioning: if you are not in the top 50 by revenue but aspire to be visible to AI-driven searches, audit your third-party presence before you audit your robots.txt.

AI visibility by practice area: where law firms are citeable

Law firms are not equally visible across all practice areas in AI answers. Perplexity and ChatGPT search cite firms more consistently in areas where third-party rankings and earned media are strongest: commercial law, dispute resolution, employment, and tax are well-covered in industry publications and analyst databases. Niche practice areas—immigration, construction defects, pension law—see lower citation frequency because fewer publications cover them routinely. A firm in a saturated category like corporate law is competing for citation against dozens of well-covered competitors; a firm in a smaller, vertical-specific niche may be the only citable authority on a question. This is structural advantage: being the only firm mentioned on a topic in AI results is worth more than being one of ten options, even in a larger practice area.

The corollary is that AI visibility strategies must be practice-area-specific. A US litigation boutique will find Perplexity cites the American Bar Association, state bar associations, and published case digests before it cites firm websites. A UK employment law practice will find it cited in employment news sites, tribunal decisions, and ACAS guidance before firm content. This is not a failure of the firm's SEO; it is the correct behavior of AI systems. The way to shift citation patterns is not to change robots.txt but to build presence in the sources AI systems are trained on: publication placement in sector-specific journals, contribution to practitioner databases, inclusion in curated legal directories and knowledge bases that OpenAI and Google have licensed for training or retrieval.

Practical AI visibility tiers for different firm sizes

Law firms in this index range from sole practitioner boutiques to 50-partner institutions, and their AI visibility strategy should scale accordingly. Top-50 magic circle and elite-circle firms (Clifford Chance, Freshfields, Linklaters, etc.) are already cited by AI systems because they appear in multiple licensed data sources: Chambers & Partners, the Legal Business 100 rankings, and dozens of press mentions. These firms can ignore AI visibility entirely—they are already visible—and their time is better spent protecting their rankings from newer competitors than optimizing for citation. Mid-market firms (20 to 100 partners, typically ranked 20-200 by revenue) have the most to gain: they are large enough to generate thought leadership and publish original research, but small enough that their visibility can shift meaningfully. For these firms, AI visibility is a by-product of: getting partners quoted in Legal Cheek, The Lawyer, and practice-area-specific publications; securing placement on Chambers Associates shortlists; publishing defensible, cited research in legal journals; and building a substantive presence in practitioner communities (LinkedIn groups, Reddit's r/law, specialist forums). Boutique and smaller practices should audit whether they are visible in practice-area-specific databases and directories first—a family law firm may find itself cited if it is ranked on a family law comparison site or included in a local legal directory that AI systems have crawled. For these firms, robots.txt optimization is premature. Citation comes from being known, not from removing barriers to crawling.

The measurement tier also scales. A top-50 firm needs no AI visibility reporting—partners will naturally hear if the firm stops being cited. A mid-market firm should audit quarterly by running 10-15 practice-area queries in ChatGPT and Perplexity and recording which competitors are cited; that takes an hour and costs nothing. A boutique firm should track referral traffic from ChatGPT and Perplexity if it has access to analytics that can parse HTTP referers; if referral traffic appears, the firm is citeable; if it does not, the lack of thought leadership is the blocker, not the robots.txt. Expensive GEO or AEO tools add no value at any of these tiers.

The cost-benefit of AI visibility tooling for UK law firms

A mid-market UK law firm (20 to 50 partners) with a £20,000 annual business development budget faces a choice when considering AI visibility monitoring. An entry-tier tool like Rankscale or Peec AI costs £200-300 monthly (£2,400-3,600 annually), leaving £16,400 to £17,600 for actual visibility work. A premium tool like Nightwatch costs £299+ monthly (£3,588+ annually), leaving roughly £16,400. The difference in cost is negligible; the difference in leverage is not. The measurement that actually matters is referral traffic from ChatGPT and Perplexity in your analytics logs (free, through UTM parameters or referer header parsing), not a subscription dashboard. For law firms in this revenue tier, a rational allocation would be: zero on premium tooling (Profound's opaque pricing and multi-year contracts create lock-in), £1,500 on a single analyst relations project (securing inclusion in Chambers Associates or Legal500 if not already listed, or in a specialised legal technology ranking for growth-stage firms), £3,000-5,000 on thought leadership placement (Getting published in Legal Cheek, The Lawyer, or practice-area-specific publications like Construction Law Journal or Employment Law Review), £2,000 on community presence (Reddit moderation, specialist forum participation, answering questions substantively in your practice areas), and if budget allows, £1,500 on a lightweight tracking tool like Peec AI entry tier to confirm referral traffic moved. A firm that follows this order—analyst relations and publication placement first, tools second—will have measurable citations within three months. A firm that reverses the order (tool subscription first) will watch a dashboard move noise for six months before cancelling.

The dataset behind this page is public: src/data/ai-crawler-index.json plus one raw file per firm. Anyone can refetch a domain and check our reading against the saved response. If a firm's file has changed since 2026-07-14, that is expected, and a fresh fetch is the way to confirm it. See also how we grade and the reproduction study.

If your firm is restricting AI crawlers: the cost of changing your policy

Among the 1 firms in this index that hold AI crawlers to stricter rules than Googlebot, most did so preemptively out of caution rather than deliberate strategy. For a firm realising that the restriction is costing visibility, reversing it is straightforward. The operational cost of updating robots.txt is zero: edit the file, remove or relax the AI-specific directives (Disallow: /GPTBot, Disallow: /ClaudeBot, etc.), and republish. The search engines and AI model operators will rescan your robots.txt within 48 to 72 hours, and crawlers will resume access. If your firm has not published your robots.txt file before, creating one with no AI restrictions takes one hour for a competent developer (DNS verification, HTTPS confirmation, syntax validation). The business cost is also zero: unlike removing a directory from Google's index, which can take months to recover from, permitting AI crawlers has no negative rank effects for traditional search. Googlebot was already crawling your site (most law firm sites receive 40-60% of their organic traffic from Google), and if it was indexed, AI model crawlers were never the blocker. The real cost—if any—is if your firm's website contains sensitive information (pending litigation strategy, confidential client models, internal client lists) that you do not want in AI training data. For firms in this situation, the answer is not to block AI crawlers site-wide; it is to block specific sensitive directories while permitting access to public-facing practice area descriptions, news and publications. This is a fine-grained approach that costs 2-4 hours of configuration but delivers the same privacy protection without the visibility loss. Most law firm websites do not contain secrets; they contain marketing, practice descriptions, and published thought leadership—exactly the content you want cited by AI systems.

The timing matters. If a firm has blocked AI crawlers for six months or longer, Google's index of the site is not stale, but AI model training is. The models that most recently trained (Gemini 2.0, GPT-4 in late 2024 / early 2025, Claude in Q1 2025) may not have access to your content if you blocked their crawlers during their training window. Reversing the block today will allow new content crawlers to access your site, but backfilling older content requires publishing a sitemap or manual outreach to model operators (most do not accept sitemaps, so the practical option is to focus on new content). The implication for mid-market firms is that the longer you delay reversing a restriction, the longer your site is effectively invisible in model training. A firm that has been blocked for twelve months and reverses the block today will not see new citations from pre-existing content for months; only new articles published after the block lift will be available to crawlers. This reinforces the recommendation: decide now whether the caution was justified. If not, reverse it. If yes, implement fine-grained blocking (allow public site access, block /confidential/ or /client-work/).

The strategic implication for mid-market firms is that permitting AI crawlers costs nothing and may have modest upside. The risk is real but quantifiable: if a firm publishes legal analysis or case commentary on its website and has not secured it in third-party publications first, blocking GPTBot prevents direct theft of that analysis by AI models. But the analysis was already not being cited if it was not published in a licensed source. A firm that writes original research and publishes it in Legal Cheek, Law.com or a legal journal first, then republishes it on the firm website, loses nothing by permitting crawlers—the journal citation takes precedence in AI training. The firm publishing the same analysis only on its website is invisible in AI regardless of its robots.txt stance. The cost of blocking is lost referral traffic; the benefit is protection against a threat (direct model training) that only matters if the content was worth stealing in the first place. Most firm websites contain templated service descriptions, not unique analysis. Blocking GPTBot for templated content is security theatre.

Regional variations: London and regional firms see different AI visibility patterns

The data shows a geographical pattern that deserves attention. London-based magic circle and elite firms (those in the top 20 by revenue) achieve higher AI citation rates because they benefit from cumulative coverage in Legal500, Chambers & Partners, The Lawyer, and international legal databases. These firms are cited not because of their robots.txt policies, but because AI systems have licensed access to directories that automatically include them. Regional mid-market and boutique firms face a different visibility challenge: they may be ranked by revenue but overlooked by the publications that train AI systems. A top-100 regional employment law firm might rank highly in Chambers Associates regionally, but that does not guarantee citation in ChatGPT if Chambers' training data in the model is weighted toward national firms. For regional firms competing against London peers, the visibility block is not technical—it is editorial. Building relationships with legal editors at practice-area-specific publications (Legal Cheek for employment law, Construction Law Journal for construction specialists, Pensions Aspects for pension law specialists) is more valuable than any robots.txt strategy.

What actually happens to firm referrals when AI visibility changes

The finding that 49 of the top 50 firms do not restrict AI crawlers is only meaningful if it drives measurable business outcome. The missing data point is referral traffic. A firm that appears in ChatGPT search results for "top commercial law firms in London" may receive five inquiries per month as a result, or none at all, depending on whether those searchers convert interest into action. A firm that blocks training crawlers while remaining visible in AI answers (because it is licensed-data-source prominent, such as being in Chambers & Partners) will likely see no change in referral volume. Conversely, a mid-market firm with weak third-party coverage that blocks GPTBot and ClaudeBot is optimising a variable that has no measurable impact on visibility in live AI systems (which use different tokens: OAI-SearchBot, Perplexity-User). The way to audit this is not through tool dashboards but through referrer logs. Set up a UTM parameter or referrer-header parsing rule in your analytics tool. Then, weekly for 12 weeks, record exact traffic originating from ChatGPT, Perplexity and Google AI Overviews. If referral traffic is zero, your visibility problem is editorial (you are not listed where AI systems look), not technical (robots.txt). If referral traffic is measurable and growing, you are citable and should track it quarterly rather than chasing month-on-month noise from tool dashboards. This is the one audit that matters, and no tool can measure it better than your own logs.

The cost of misaligned AI visibility strategy: why law firms waste money on the wrong tools

A typical law firm AI visibility tool costs £200-500 monthly. A premium tool like Nightwatch runs £300+. For a mid-market firm with thirty to fifty partners, this is a £3,600 to £6,000 annual commitment. The finding of this index—that 49 of the top 50 firms make no AI-specific restrictions—means most premium tools are measuring citation traffic from firms that are already visible or already blocked. For a firm paying for monthly reports that say "your citation share is 8%", the tool is returning noise. The real question is whether that firm appears at all in ChatGPT or Perplexity for practice-area-specific queries. A five-minute manual audit—running "employment law firm London", "corporate law advice UK", "tax law consultancy near me" in ChatGPT and Perplexity—answers this for free. If the firm appears zero times, a tool tells you nothing about why. If it appears two to three times, the firm has baseline visibility and should track referral traffic (free through analytics logs) rather than paying for a proprietary score. If it appears five or more times, the firm is already citable and should invest in maintaining visibility through thought leadership, not in tool spend.

What law firm size justifies AI visibility tool investment

The decision to buy an AI visibility tool should depend on firm revenue and development pipeline. A sole-practitioner boutique should not buy a tool under any circumstances: the cost is a fixed overhead on solo-scale revenue, and the visibility work is relationship-driven, not metrics-driven. A firm with five to ten partners should treat a tool as an experiment, not an operating expense. Set aside £1,500 for six months of Rankscale or Peec AI entry-tier tracking, run it, and audit referral traffic from ChatGPT and Perplexity during that window (free, through analytics logs). If referral traffic increased, renew; if it did not, the firm's block is editorial (not being mentioned where AI systems look), not measurement-related, and no tool spend will fix it. A mid-market firm (twenty to fifty partners) can justify Gumshoe (£349/mo) or Nightwatch (£299+/mo) if the firm is already getting press mentions and analyst inclusion, because then the tool is tracking a real phenomenon and watching for competitive shifts. An enterprise firm (100+ partners) should build an internal dashboard parsing referrer logs from ChatGPT and Perplexity, abandoning commercial tools entirely; the £3,600 annual cost of a tracker becomes pure waste when the firm's analytics platform (Mixpanel, Amplitude, GA4 with server-side tracking) can deliver the same data for zero additional cost.

The misallocation comes from applying SaaS or e-commerce buying logic to law firm business development. A SaaS company buying an AI visibility tracker makes sense if it is competing for a finite set of product-comparison keywords where citation ranking matters to deal flow. A law firm has a different visibility challenge: it is not ranked by an algorithm but filtered by editorial gatekeeping (Chambers, Legal500) and earned media (publications that mention it). No monthly tool report will move those levers. A firm spending £500 monthly on a tracker and nothing on getting featured in The Lawyer, Legal Cheek, or practice-area-specific publications is burning money on rear-view mirrors while ignoring the engine. The rational allocation for a mid-market firm is zero on tools, £2,000-3,000 monthly on analyst relations and publication placement, and £500-1,000 on community presence and Reddit participation in practice-area-specific forums. That budget moves citations. A tool does not.

AI visibility tool pricing for UK law firms by size and return

A typical UK law firm considering AI visibility investment faces a pricing decision with three options. Entry-level trackers like Rankscale (from $20 monthly, approximately £16) or Otterly ($29 monthly, approximately £23) cost £200 to £300 annually and track 25 to 100 branded keywords with monthly updates. For a solo practitioner or two-partner boutique, this cost is defensible only if the firm is already appearing in legal databases (Wikipedia's law categories, Avvo, state bar listings), publications (Legal Cheek, Law.com), or Reddit legal communities. If not, the tool confirms invisibility without explaining the cause. Mid-tier tools (Nightwatch at £239 monthly, approximately £287 annually for entry, Gumshoe at £279 monthly equivalent) cost £3,000 to £4,000 annually and refresh weekly, meaning you can correlate tool movements with publication placements or analyst inclusion in Chambers or Legal500. For a 10-20 partner regional firm with a £10,000 annual business development budget, splitting it as £3,500 on a mid-tier tool plus £6,500 on analyst relations (securing Chambers or Legal500 inclusion if not already listed) makes sense because the tool validates whether analyst placements move AI citations. Premium tier (Profound, typically £1,500+ monthly with multi-year contracts, approximately £18,000+ annually) is justifiable only for magic-circle firms that are already mentioned in multiple licensed sources and need to track competitive positioning. For most UK law firms, the cost-benefit inflection point is this: spend zero on premium tools and up to £3,000 annually on mid-tier only after you have secured analyst or publication placement in sources AI systems weight (Chambers & Partners, Legal500, law firm databases, industry publications). Without that underlying placement, the tool measures signal that does not exist, and the spend is wasted.

Partner specialties and practitioner visibility: where individual authority matters more than firm brand

This index ranks firms by revenue, but AI visibility for law firms often clusters around named practitioners rather than firm brand. A solo partner in a niche area (immigration law, pension disputes, construction defects) may drive more AI citations than a 50-partner commercial law generalist. This is because Perplexity and ChatGPT cite individuals as sources when they have public profiles, published research, or practitioner recognition in specialised forums. The implication for mid-market and boutique firms is that AI visibility strategy should be bottom-up: encourage senior practitioners to publish in practitioner journals, build profiles on LinkedIn and practitioner forums, contribute to open discussions in legal communities (Reddit's r/law, Legal Stack Exchange). A single well-known employment law partner will drive more ChatGPT citations to the firm than a generalised "employment law services" SEO push. This is asymmetric to traditional SEO, where firm brand matters more than individual reputation. In AI search, the opposite is true.

How to act on the index: a three-month law firm AI visibility roadmap

If you are in this top-50 ranking, your first action is not to change your robots.txt. It is to audit your third-party presence. Spend two hours this week fetching your firm's entry in Chambers & Partners, Legal500, The Lawyer's directory, and any practice-area-specific rankings (e.g., Boutique Law Firms Top 100). If you are listed, you are already likely cited by AI systems; if you are not listed, that is your blocker. Chambers and Legal500 exclusion is structural—no on-site optimisation will overcome it. The path is to secure inclusion in next year's round or to pursue analyst relations. For firms outside the top 50 by revenue or geographic scope, the same logic applies: identify which directories, publications and databases your ideal clients and business development contacts actually consult, and ensure you are included there. Second, assign one partner per practice area to build practitioner visibility: publish a piece of research (even 2,000 words), contribute to a legal journal or trade publication, or engage in open practitioner forums. This takes four to eight weeks and costs no money. Third, measure by referral traffic only. If you are getting chatbot referrals, track them. If you are not, the block is editorial (not listed where AI systems look), not technical (robots.txt).