|
Post by account_disabled on Jan 28, 2024 6:43:34 GMT
By using Google-Extended to control access to content on a site, a website administrator can choose whether to help these AI models become more accurate and capable over time.” – Google’s Danielle Romain, VP, Trust / What is Google-Extended. Google calls it “A standalone product token that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products. The new crawler has been added to the DB to Data Google Search Central documentation on web crawlers. What Google is saying. The company said Google-Extended gives publishers “choice and control”: “Making simple and scalable controls, like Google-Extended, available through robots.txt is an important step in providing transparency and control that we believe all providers of AI models should make available. However, as AI applications expand, web publishers will face the increasing complexity of managing different uses at scale.” Robots.txt. You can use robots.txt to block Google-Extended from accessing your content, or parts of it. To fully block Google-Extended, add the following to your site’s robots. User-agent: Google-Extended Disallow: / Why we care. We know 242 of the most popular 1,000 websites have already decided to block GPTBot, OpenAI’s web crawler, since it launched in August.
|
|