Hacker News with Generative AI: Web Crawlers

Nepenthes is a tarpit to catch AI web crawlers (zadzmo.org)
This is a tarpit intended to catch web crawlers. Specifically, it's targetting crawlers that scrape data for LLM's - but really, like the plants it is named after, it'll eat just about anything that finds it's way inside.
Reddit's robots.txt disallows all web crawlers (reddit.com)