{"id":12306,"date":"2024-03-11T09:43:11","date_gmt":"2024-03-11T09:43:11","guid":{"rendered":"https:\/\/www.20i.com\/blog\/?p=12306"},"modified":"2026-07-16T14:03:27","modified_gmt":"2026-07-16T13:03:27","slug":"how-to-prevent-ai-from-scraping-your-website","status":"publish","type":"post","link":"https:\/\/www.20i.com\/blog\/how-to-prevent-ai-from-scraping-your-website\/","title":{"rendered":"How to Prevent AI From Scraping Your Website"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Search engines such as Google use crawlers \u2013 otherwise known as bots \u2013 to tirelessly process millions of pages per day, creating a vast digital index which makes up their search results.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For the most part, this is a positive, as it means your content can appear in their search results and earn traffic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, it\u2019s not just search engines who use these bots and anyone with the skills and infrastructure can have their own crawler.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The ethicality of these bots can vary, from SEO tools like ahrefs collecting data for their reports, to spam bots which may look to harvest email address data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In recent years, a new, industrial scale breed of bot has emerged \u2013 the AI scraper.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"How to Prevent AI From Scraping Your Website\" width=\"1200\" height=\"675\" src=\"https:\/\/www.youtube.com\/embed\/FkxAMNmoGSg?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">What are AI scrapers?&nbsp;<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">These bots, used in popular tools such as ChatGPT and Google Gemini, go beyond collecting information to help with indexing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Their objective is to scoop up your website&#8217;s data to then train their AI models for everything from providing chatbot answers to generating text or creating photorealistic images.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This has begun to cause some controversy across the industry, with some publishers not happy with perceived privacy and intellectual property violations from bots taking their content.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Getty Images and a group of artists are making a <a href=\"https:\/\/www.theverge.com\/2023\/12\/4\/23988403\/getty-lawsuit-stability-ai-copyright-infringement\" target=\"_blank\" rel=\"noreferrer noopener\">legal challenge against AI image<\/a> generators, whilst in 2023 Google was hit with a class action suit for <a href=\"https:\/\/www.reuters.com\/legal\/litigation\/google-hit-with-class-action-lawsuit-over-ai-data-scraping-2023-07-11\/\" target=\"_blank\" rel=\"noreferrer noopener\">their use of AI scraped data<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Some companies such as Tumblr &amp; WordPress owner Automattic have seen the economic opportunities here and <a href=\"https:\/\/www.theverge.com\/2024\/2\/27\/24084884\/tumblr-midjourney-openai-training-data-deal-report\" target=\"_blank\" rel=\"noreferrer noopener\">agreed deals<\/a> to allow AI companies to use their data for training purposes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Why stop AI bots scraping your website?&nbsp;<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The biggest concern most publishers have regarding AI bots, is the devaluation of their own content, and a loss of traffic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once bots have your content, they can begin serving it within chatbot answers \u2013 often without providing credit to the source. This usually means that the user has less reason to ever visit the original place that content was published.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As mentioned above, there are also concerns surrounding intellectual property. If you publish anything particularly unique such as artwork or data that you have IP protections over, then you certainly don\u2019t want AI serving this data as its own.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">AI bots may also affect areas such as bandwidth. Every time someone accesses your website, data is transferred, which uses up bandwidth. AI bots with free reign to crawl a website may be a drain on resources if they become particularly aggressive and crawl content excessively \u2013 leading to slower loading times for legitimate visitors.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Should you block ChatGPT from scraping your website?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This all depends on your views on AI, and also the type of content that you are posting.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">By allowing ChatGPT to access your content, you are helping to train the AI and improve future versions \u2013 which is great if you\u2019re an AI fanatic like some members of the 20i team!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There is a risk however that your content may be served elsewhere, without credit to you. Therefore, if you are writing particularly unique or insightful content which can only be found on your website, it is better to keep it that way.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It is also worth considering whether your website contains any sensitive information which, again, you would prefer to keep out of AI\u2019s reach.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Blocking ChatGPT is becoming a more popular practice. As of September 2023, <a href=\"https:\/\/searchengineland.com\/more-popular-websites-blocking-gptbot-432531\" target=\"_blank\" rel=\"noreferrer noopener\">26% of the top 100 websites<\/a> in the world had blocks in place. News giant the New York Times is one of these many websites with the GPTBot blocked via robots.txt.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How to stop AI bots from scraping your website<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">There are multiple methods you can use to try and deter AI bots from scraping your website, including some specific to each crawler.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Preventing ChatGPT from scraping your website is done in the same way as blocking most other mainstream bots.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">ChatGPT&#8217;s developers, OpenAI, respect the robots.txt protocol. This means you can instruct their crawler, GPTBot, to not access your website by adding the following line to your robots.txt file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: GPTBot<br>Disallow: \/<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This tells GPTBot (the user-agent) to disallow crawling of all pages on your website. To specify blocking of specific pages or subfolders, just amend the \/ to your required URL.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong><em>Should you block Google AI from scraping your website?<\/em><\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Looking to avoid the number of websites which have blocked ChatGPT, Google has tried to appeal more to website owners by stating their commitment to transparency and showing a desire to provide website credit links as often as possible.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With that said, whether you allow Google AI to crawl your website still comes down to your overall view of AI.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Your data could be used to train AI tools to be more useful in the future, however, there remains the risk of your content becoming available without the need for people to visit your website.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s important to know that Google-extended will not block Googles SGE from crawling your website, and therefore blocking Google AI bots poses no risk to your organic search rankings.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>How to stop Google AI from scraping your website<\/strong>&nbsp;<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Similarly, to ChatGPT, Google\u2019s AI tools can be prevented from scraping your website using the robots.txt file.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In late 2023, Google announced a new standalone product token known as Google-Extended. By placing the following into your robots.txt file, Google AI bots will not be able to use your website content for learning purposes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: Google-Extended<br>Disallow: \/<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This will block both Google Bard, the conversational AI used for tools such as Gemini, and Vertex AI, which is used for building and deploying generative AI-powered search. Google-extended is now a part of <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/overview-google-crawlers\" target=\"_blank\" rel=\"noreferrer noopener\">Googles crawler overview documentation<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The above snippet is designed to disallow the crawler from your entire website. If you would only like to block certain pages \u2013 for example those with sensitive or valuable data &#8211; then replace the \/ with the URL or directory of your choice.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Blocking other AI bots<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">If you\u2019d rather keep your website copy away from AI, then on top of the \u201cbig two\u201d OpenAI and Google, you may also want to consider Common Crawl.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Common Crawl is one of the largest public datasets used by AI for training, with ChatGPT and other large language models all utilising this dataset, which is collected with a crawler housed in Amazons S3 service. If the Common Crawl bot (otherwise known as CCBot) has your data, then there&#8217;s a chance that AI bots will have access to that information too.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Because of this, CCBot is the 2nd most blocked AI bot.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As with GPT Bot and Google, you can prevent CCBot from scraping your content by using the robots.txt exclusion protocol. Add the lines below into your robots.txt file to halt its crawling activities:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: CCBot<br>Disallow: \/<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Similarly, Claude, another popular AI tool owned by Anthropic can also be blocked using the following within robots.txt<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: ClaudeBot<br>Disallow: \/<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It is worth noting that Claude previously used user agent names &#8216;ANTHROPIC-AI&#8217; and &#8216;CLAUDE-WEB&#8217;. Anthropic have stated that ClaudeBot is configured to acknowledge these directives, but it still may be worth updating your robots.txt file if you aren&#8217;t referencing ClaudeBot.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Whilst blocking the larger AI bots from crawling your content is relatively easy, new smaller bots are popping up all of the time, which means that blocking them via robots.txt isn\u2019t always the answer.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Other methods to restrict AI bot access to your content:<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>.htaccess<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Bots can be blocked within your htaccess file, by using rewritecond to forbid certain bots based on their User-agent header. As with the robots.txt method though, this is only effective if you know the exact name of the bot.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Web Application Firewall (WAF)<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Install and configure a WAF to filter website traffic. This will enable you to block requests from AI bots by targeting specific user agents or IP addresses. All websites <a href=\"https:\/\/www.20i.com\/managed-hosting\" target=\"_blank\" rel=\"noreferrer noopener\">hosted with 20i<\/a> receive automatic protection by our free WAF.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Use CAPTCHAs or Proof of Work<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Implementing these can deter automated bots by requiring a human-like response or computational proof.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>HTTP Authentication<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This adds a username and password layer to your website. While not foolproof, and unsuitable for a lot of websites, it deters basic bots designed to navigate publicly accessible websites.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>IP Blocking<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">IP blocking can feel like fighting a losing battle at times, as bots can easily switch IPs, however it can still be effective. Monitor your server logs or create honeypot crawler traps to identify specific IPs which may be crawling your site excessively, and then block them. IPs can also be blocked with the <a href=\"https:\/\/www.20i.com\/cdn\" target=\"_blank\" rel=\"noreferrer noopener\">20iCDN<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Legal Measures<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An extreme measure, likely only reserved for enterprise level publishers, but if you feel that AI crawling is infringing on any of your intellectual property rights, then consulting legal advice can be the best course of action.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Reminder:<\/strong> <em>AI bots may change name at any time, so if you are blocking any particular scrapers, we recommend keeping on top of the relevant documentation so that you can ensure you are referencing the correct user agents.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Final thoughts<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">AI provides opportunities and challenges like the web has never seen before &#8211; our <a href=\"https:\/\/www.20i.com\/ai-survey\/\">AI survey<\/a> showed that 89% of digital creatives use AI tools in their work. However by understanding the landscape and using the right strategies, you can ensure that you keep as much control of your content as possible.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What are your thoughts on AI using your content? Are you happy to help these bots learn and become more useful, or do you feel that it is a threat to content producers? Let us know in the comments below.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Want to know more about AI? Read our guide on <a href=\"https:\/\/www.20i.com\/blog\/using-ai-for-copy\/\" target=\"_blank\" rel=\"noreferrer noopener\">using AI for writing website copy<\/a>.<\/p>\n\n\n<div class='code-block code-block-5' style='margin: 8px 0; clear: both;'>\n<hr>\n<br \/><a href=\"https:\/\/www.20i.com\/managed-cloud-servers\" target=\"_blank\"><img decoding=\"async\" src=\"https:\/\/www.20i.com\/blog\/wp-content\/uploads\/2026\/03\/Blog-Ad-MCS-1200x625-1.png\" loading=\"lazy\" alt=\"Managed Cloud Hosting\"><\/a><\/div>\n\n","protected":false},"excerpt":{"rendered":"Search engines such as Google use crawlers \u2013 otherwise known as bots \u2013 to tirelessly process millions of&hellip;","protected":false},"author":33,"featured_media":20053,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"csco_singular_sidebar":"","csco_page_header_type":"","csco_page_load_nextpost":"","footnotes":""},"categories":[51,60],"tags":[268,57,79],"class_list":["post-12306","post","type-post","status-publish","format-standard","has-post-thumbnail","category-technology","category-web-design-dev","tag-ai","tag-web-design","tag-web-development","cs-entry"],"_links":{"self":[{"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/posts\/12306","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/comments?post=12306"}],"version-history":[{"count":12,"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/posts\/12306\/revisions"}],"predecessor-version":[{"id":18165,"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/posts\/12306\/revisions\/18165"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/media\/20053"}],"wp:attachment":[{"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/media?parent=12306"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/categories?post=12306"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.20i.com\/blog\/wp-json\/wp\/v2\/tags?post=12306"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}