91 of 225 of the most visited websites block at least one AI crawler in their robots.txt. Measured from each site's public robots.txt, re-read daily. Updated 2026-10-24. Change log · JSON (CC BY 4.0)
| GPTBot OpenAI (training) | 68 blocked (30%) · named in 69 robots.txt files |
| ChatGPT-User ChatGPT browsing | 52 blocked (23%) · named in 53 robots.txt files |
| OAI-SearchBot ChatGPT search | 40 blocked (18%) · named in 43 robots.txt files |
| ClaudeBot Anthropic (training) | 71 blocked (32%) · named in 68 robots.txt files |
| Claude-User Claude browsing | 46 blocked (20%) · named in 39 robots.txt files |
| Claude-SearchBot Claude search | 44 blocked (20%) · named in 39 robots.txt files |
| PerplexityBot Perplexity | 56 blocked (25%) · named in 63 robots.txt files |
| Google-Extended Gemini (training) | 59 blocked (26%) · named in 65 robots.txt files |
| Applebot-Extended Apple Intelligence | 51 blocked (23%) · named in 41 robots.txt files |
| CCBot Common Crawl | 77 blocked (34%) · named in 66 robots.txt files |
| Meta-ExternalAgent Meta AI | 60 blocked (27%) · named in 51 robots.txt files |
| Bytespider ByteDance | 76 blocked (34%) · named in 61 robots.txt files |
| Amazonbot Amazon | 45 blocked (20%) · named in 37 robots.txt files |
| cohere-ai Cohere | 61 blocked (27%) · named in 49 robots.txt files |
| asahi.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| baidu.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| dailymail.co.uk | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| daum.net | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| facebook.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| hihonorcloud.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| huffpost.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| instagram.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| kakao.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| mail.ru | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| mediacategory.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| msn.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| nbcnews.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| pinterest.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| reddit.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| soundcloud.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| zepto.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| cnbc.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| imdb.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| theglobeandmail.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| amazon.ca | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-23 |
| amazon.co.jp | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.co.uk | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.com.au | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.com.br | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-23 |
| amazon.com.mx | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.de | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.es | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.fr | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.in | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| amazon.it | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| bbc.co.uk | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| bbc.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| cnn.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Applebot-Extended, CCBot, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| tiktok.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider · 2026-10-24 |
| twitter.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Applebot-Extended, CCBot, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| x.com | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Applebot-Extended, CCBot, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| nature.com | blocks GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| newyorker.com | blocks ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| spiegel.de | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| theverge.com | blocks ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| vox.com | blocks ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| wired.com | blocks ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| buzzfeed.com | blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| lefigaro.fr | blocks GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, CCBot, Amazonbot, cohere-ai · 2026-10-24 |
| abc.net.au | blocks GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| corriere.it | blocks GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot · 2026-10-24 |
| forbes.com | blocks GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-23 |
| genius.com | blocks GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| lemonde.fr | blocks ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| newsweek.com | blocks GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Applebot-Extended, CCBot, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| theguardian.com | blocks ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot · 2026-10-24 |
| zigzag.kr | blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| 11st.co.kr | blocks GPTBot, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| techcrunch.com | blocks GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, cohere-ai · 2026-10-24 |
| yahoo.com | blocks GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, cohere-ai · 2026-10-24 |
| amazonvideo.com | blocks ChatGPT-User, Claude-User, PerplexityBot, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai · 2026-10-24 |
| elpais.com | blocks ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, CCBot, Bytespider, Amazonbot · 2026-10-24 |
| healthline.com | blocks GPTBot, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot · 2026-10-24 |
| repubblica.it | blocks GPTBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, cohere-ai · 2026-10-24 |
| t-online.de | blocks GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider · 2026-10-24 |
| aljazeera.com | blocks GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bytespider, cohere-ai · 2026-10-24 |
| aol.com | blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, cohere-ai · 2026-10-24 |
| bild.de | blocks PerplexityBot, Applebot-Extended, CCBot, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| canva.com | blocks GPTBot, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider · 2026-10-23 |
| figma.com | blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, cohere-ai · 2026-10-24 |
| netflix.com | blocks Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai · 2026-10-24 |
| tumblr.com | blocks ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot · 2026-10-24 |
| whatsapp.com | blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot · 2026-10-24 |
| espn.com | blocks GPTBot, ChatGPT-User, Google-Extended, CCBot, Bytespider · 2026-10-24 |
| vimeo.com | blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider · 2026-10-24 |
| airbnb.com | blocks GPTBot, ClaudeBot, Applebot-Extended · 2026-10-24 |
| businessinsider.com | blocks ClaudeBot, CCBot, Bytespider · 2026-10-24 |
| dictionary.com | blocks GPTBot, Applebot-Extended, CCBot · 2026-10-24 |
| globo.com | blocks GPTBot, Google-Extended, CCBot · 2026-10-24 |
| jio.com | blocks CCBot, Meta-ExternalAgent, Bytespider · 2026-10-24 |
| latimes.com | blocks GPTBot, ClaudeBot, Bytespider · 2026-10-24 |
| weather.com | blocks CCBot, Bytespider, Amazonbot · 2026-10-24 |
| webmd.com | blocks GPTBot, ClaudeBot, CCBot · 2026-10-24 |
| goodreads.com | blocks GPTBot, CCBot · 2026-10-24 |
| notion.com | blocks Meta-ExternalAgent, Amazonbot · 2026-10-24 |
| yahoo.co.jp | blocks GPTBot, Google-Extended · 2026-10-24 |
| amplitude.com | blocks Bytespider · 2026-10-24 |
| coursera.org | blocks Meta-ExternalAgent · 2026-10-24 |
| github.com | blocks Bytespider · 2026-10-24 |
| hindustantimes.com | blocks Meta-ExternalAgent · 2026-10-24 |
| indiatimes.com | blocks Meta-ExternalAgent · 2026-10-24 |
| kwai.com | blocks Bytespider · 2026-10-24 |
| notion.so | blocks Amazonbot · 2026-10-24 |
| straitstimes.com | blocks Google-Extended · 2026-10-24 |
| 163.com | allows all AI crawlers · 2026-10-24 |
| 360yield.com | allows all AI crawlers · 2026-10-24 |
| a-mo.net | no robots.txt (allows all) · 2026-10-24 |
| adobe.com | allows all AI crawlers · 2026-10-24 |
| adtrafficquality.google | no robots.txt (allows all) · 2026-10-24 |
| ahnlab.com | allows all AI crawlers · 2026-10-23 |
| ampproject.org | no robots.txt (allows all) · 2026-10-24 |
| android.com | allows all AI crawlers · 2026-10-24 |
| anthropic.com | allows all AI crawlers · 2026-10-24 |
| appier.net | allows all AI crawlers · 2026-10-24 |
| apple.com | allows all AI crawlers · 2026-10-24 |
| arxiv.org | allows all AI crawlers · 2026-10-24 |
| auction.co.kr | no robots.txt (allows all) · 2026-10-24 |
| bing.com | allows all AI crawlers · 2026-10-24 |
| blogspot.com | allows all AI crawlers · 2026-10-24 |
| booking.com | allows all AI crawlers · 2026-10-24 |
| brave.com | allows all AI crawlers · 2026-10-24 |
| braze.com | allows all AI crawlers · 2026-10-24 |
| cdc.gov | allows all AI crawlers · 2026-10-24 |
| charter.net | allows all AI crawlers · 2026-10-24 |
| cisco.com | allows all AI crawlers · 2026-10-24 |
| clarin.com | allows all AI crawlers · 2026-10-24 |
| claude.ai | no robots.txt (allows all) · 2026-10-23 |
| cloudflare.com | allows all AI crawlers · 2026-10-24 |
| coupang.com | no robots.txt (allows all) · 2026-10-23 |
| cursor.sh | allows all AI crawlers · 2026-10-24 |
| dell.com | allows all AI crawlers · 2026-10-24 |
| discord.com | allows all AI crawlers · 2026-10-24 |
| domaindetails.com | allows all AI crawlers · 2026-10-24 |
| dotomi.com | no robots.txt (allows all) · 2026-10-24 |
| doubao.com | no robots.txt (allows all) · 2026-10-24 |
| dropbox.com | allows all AI crawlers · 2026-10-24 |
| duolingo.com | allows all AI crawlers · 2026-10-24 |
| fast.com | allows all AI crawlers · 2026-10-24 |
| flashtalking.com | allows all AI crawlers · 2026-10-24 |
| flipkart.com | allows all AI crawlers · 2026-10-24 |
| foxnews.com | allows all AI crawlers · 2026-10-24 |
| google.ca | allows all AI crawlers · 2026-10-23 |
| google.co.in | allows all AI crawlers · 2026-10-23 |
| google.co.jp | allows all AI crawlers · 2026-10-23 |
| google.co.kr | allows all AI crawlers · 2026-10-23 |
| google.co.uk | allows all AI crawlers · 2026-10-23 |
| google.com | allows all AI crawlers · 2026-10-23 |
| google.com.au | allows all AI crawlers · 2026-10-23 |
| google.com.br | allows all AI crawlers · 2026-10-23 |
| google.com.mx | allows all AI crawlers · 2026-10-23 |
| google.de | allows all AI crawlers · 2026-10-23 |
| google.es | allows all AI crawlers · 2026-10-23 |
| google.fr | allows all AI crawlers · 2026-10-23 |
| google.it | allows all AI crawlers · 2026-10-23 |
| google.nl | allows all AI crawlers · 2026-10-24 |
| google.ru | allows all AI crawlers · 2026-10-23 |
| grammarly.com | allows all AI crawlers · 2026-10-24 |
| harvard.edu | allows all AI crawlers · 2026-10-24 |
| hcaptcha.com | allows all AI crawlers · 2026-10-24 |
| hotstar.com | no robots.txt (allows all) · 2026-10-24 |
| hp.com | allows all AI crawlers · 2026-10-24 |
| ibm.com | allows all AI crawlers · 2026-10-24 |
| icloud.com | allows all AI crawlers · 2026-10-24 |
| independent.co.uk | allows all AI crawlers · 2026-10-24 |
| intel.com | allows all AI crawlers · 2026-10-24 |
| intercom.io | allows all AI crawlers · 2026-10-24 |
| ip-api.com | allows all AI crawlers · 2026-10-24 |
| iubenda.com | allows all AI crawlers · 2026-10-24 |
| japantimes.co.jp | allows all AI crawlers · 2026-10-24 |
| kaspersky.com | allows all AI crawlers · 2026-10-24 |
| khanacademy.org | no robots.txt (allows all) · 2026-10-24 |
| launchdarkly.com | allows all AI crawlers · 2026-10-24 |
| line.me | allows all AI crawlers · 2026-10-24 |
| live.com | no robots.txt (allows all) · 2026-10-24 |
| microsoft.com | allows all AI crawlers · 2026-10-24 |
| mit.edu | allows all AI crawlers · 2026-10-24 |
| moengage.com | allows all AI crawlers · 2026-10-24 |
| mozilla.org | allows all AI crawlers · 2026-10-24 |
| nasa.gov | allows all AI crawlers · 2026-10-24 |
| nationalgeographic.com | allows all AI crawlers · 2026-10-24 |
| naver.com | no robots.txt (allows all) · 2026-10-24 |
| naver.net | no robots.txt (allows all) · 2026-10-24 |
| navercorp.com | allows all AI crawlers · 2026-10-24 |
| nerdwallet.com | allows all AI crawlers · 2026-10-24 |
| nexon.com | no robots.txt (allows all) · 2026-10-24 |
| nike.com | allows all AI crawlers · 2026-10-24 |
| nubank.com.br | allows all AI crawlers · 2026-10-24 |
| nvidia.com | allows all AI crawlers · 2026-10-24 |
| office.com | no robots.txt (allows all) · 2026-10-24 |
| onelink.me | no robots.txt (allows all) · 2026-10-24 |
| opera.com | allows all AI crawlers · 2026-10-24 |
| outlook.com | no robots.txt (allows all) · 2026-10-24 |
| paytm.com | allows all AI crawlers · 2026-10-24 |
| perplexity.ai | allows all AI crawlers · 2026-10-24 |
| phonepe.com | allows all AI crawlers · 2026-10-24 |
| publicnode.com | allows all AI crawlers · 2026-10-24 |
| rakuten.co.jp | allows all AI crawlers · 2026-10-23 |
| roblox.com | allows all AI crawlers · 2026-10-24 |
| salesforce.com | allows all AI crawlers · 2026-10-24 |
| samsung.com | allows all AI crawlers · 2026-10-24 |
| scmp.com | allows all AI crawlers · 2026-10-24 |
| shopee.com.br | allows all AI crawlers · 2026-10-24 |
| shopify.com | allows all AI crawlers · 2026-10-24 |
| slack.com | allows all AI crawlers · 2026-10-24 |
| smilewanted.com | no robots.txt (allows all) · 2026-10-24 |
| sonyliv.com | allows all AI crawlers · 2026-10-24 |
| spotify.com | allows all AI crawlers · 2026-10-24 |
| springer.com | no robots.txt (allows all) · 2026-10-24 |
| stackadapt.com | allows all AI crawlers · 2026-10-24 |
| stanford.edu | allows all AI crawlers · 2026-10-24 |
| stripe.com | allows all AI crawlers · 2026-10-24 |
| target.com | allows all AI crawlers · 2026-10-24 |
| teads.tv | no robots.txt (allows all) · 2026-10-24 |
| telegram.org | no robots.txt (allows all) · 2026-10-24 |
| time.com | allows all AI crawlers · 2026-10-24 |
| toss.im | allows all AI crawlers · 2026-10-24 |
| tossinvest.com | allows all AI crawlers · 2026-10-24 |
| truecaller.com | allows all AI crawlers · 2026-10-24 |
| twitch.tv | allows all AI crawlers · 2026-10-24 |
| uber.com | allows all AI crawlers · 2026-10-24 |
| ubuntu.com | allows all AI crawlers · 2026-10-24 |
| v-videoapp.com | no robots.txt (allows all) · 2026-10-24 |
| visualstudio.com | allows all AI crawlers · 2026-10-24 |
| vivoglobal.com | allows all AI crawlers · 2026-10-24 |
| vk.ru | allows all AI crawlers · 2026-10-24 |
| walmart.com | allows all AI crawlers · 2026-10-24 |
| weplayapp.com | allows all AI crawlers · 2026-10-24 |
| wikipedia.org | allows all AI crawlers · 2026-10-24 |
| wildberries.ru | allows all AI crawlers · 2026-10-24 |
| wordpress.com | allows all AI crawlers · 2026-10-24 |
| wps.com | allows all AI crawlers · 2026-10-24 |
| xero.com | allows all AI crawlers · 2026-10-24 |
| xhamster46.desi | allows all AI crawlers · 2026-10-24 |
| yandex.com | allows all AI crawlers · 2026-10-24 |
| yandex.ru | allows all AI crawlers · 2026-10-24 |
| yastatic.net | no robots.txt (allows all) · 2026-10-24 |
| youtube.com | allows all AI crawlers · 2026-10-24 |
| zoom.us | allows all AI crawlers · 2026-10-24 |
Method: RFC 9309 group rules. A crawler counts as blocked when the group naming it (or, if none names it, the * group) disallows the whole site without allowing it back. Partial rules (single folders) are not counted as blocking. A site appears once its homepage answers as a website. Corrections: the table re-reads every site daily.
Free, no account. Every result is measured live from the site's public files (homepage, robots.txt, llms.txt, sitemap.xml). An Agent Exchange tool.