Mastodon

0 posts0 participants0 posts today

James RavenscroftUpdate on my <a href="https://social.lol/tags/anubis" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#anubis</a> deployment. I've been hit by about 15k "suspicious" browsers today and 20 of them validated... So 0.13% of my visitors are human (or at least sophisticated enough to fool anubis). Gee wizz... <a href="https://social.lol/tags/forgejo" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#forgejo</a> <a href="https://social.lol/tags/scrapingrestrictions" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapingrestrictions</a> <a href="https://social.lol/tags/scrapingbots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapingbots</a>

Liz ProbertHow crawlers impact the operations of the Wikimedia projects <a href="https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/</a> <a href="https://greennet.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a>, <a href="https://greennet.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Crawlers</a>, <a href="https://greennet.social/tags/Infrastructure" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Infrastructure</a>, <a href="https://greennet.social/tags/KnowledgeAsAService" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#KnowledgeAsAService</a>, <a href="https://greennet.social/tags/KnowledgeContent" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#KnowledgeContent</a>, <a href="https://greennet.social/tags/Operations" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Operations</a>, <a href="https://greennet.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a>, <a href="https://greennet.social/tags/ScrapingBots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ScrapingBots</a>, <a href="https://greennet.social/tags/Traffic" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Traffic</a>, <a href="https://greennet.social/tags/WikimediaFoundation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WikimediaFoundation</a>, <a href="https://greennet.social/tags/WikimediaProjects" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WikimediaProjects</a>

Admin JerryIf <a href="https://hear-me.social/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Cloudflare</a> is to be believed, <a href="https://hear-me.social/tags/Lemmy" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Lemmy</a> instances have a built-in AI scraping bot operating beneath the covers. Do you think the developers have snuck it in?Looking through my logs, these requests have all been blocked by Cloudflare because they are identified as "AI Bots". There are many more requests by Lemmy instances blocked in the logs. This is just a sample. Other Lemmy requests from these servers get through. Only a few are blocked as AI Bots. Cloudflare says they use AI to determine if a request is a legitimate request or an AI bot trying to scrape.207.204.58.144 AS19045 DIRECTCOM United States User agent: Lemmy/0.19.5; +<a href="https://lemmy.cryonex.net" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://lemmy.cryonex.net</a>23.127.223.238 AS7018 ATT-INTERNET4 United States User agent: Lemmy/0.19.3; +<a href="https://lemux.minnix.dev" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://lemux.minnix.dev</a>2a01:cb19:f85:ec00:82fa:5bff:fe51:ed4a AS3215 France Telecom - Orange France User agent: Lemmy/0.19.5; +<a href="https://lemmy.sidh.bzh" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://lemmy.sidh.bzh</a>50.247.53.42 AS7922 COMCAST-7922 United States User agent: Lemmy/0.19.5; +<a href="https://toast.ooo" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://toast.ooo</a>69.42.19.234 AS11404 AS-WAVE-1 United States User agent: Lemmy/0.19.5; +<a href="https://lemmy.schlunker.com" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://lemmy.schlunker.com</a>155.138.226.183 AS20473 AS-CHOOPA United States User agent: Lemmy/0.19.5; +<a href="https://lemmy.mbl.social" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://lemmy.mbl.social</a><a href="https://hear-me.social/tags/MastoAdmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MastoAdmin</a> <a href="https://hear-me.social/tags/AIBots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AIBots</a> <a href="https://hear-me.social/tags/Scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scrapers</a> <a href="https://hear-me.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://hear-me.social/tags/ScrapingBots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ScrapingBots</a> <a href="https://hear-me.social/tags/privacy" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#privacy</a>

Recent searches

Search options

Administered by:

Server stats:

#scrapingbots