Hacker News with Generative AI: Web Crawling

25% of the top websites are blocking OpenAI from crawling (originality.ai)
Bots such as OpenAI’s GPTBot, the Applebot, CCBot, Google-Extended, and Bytespider analyze, store, or scrape your website’s data in order to provide data to train more advanced LLMs.
Web Crawling, Privacy, Artificial Intelligence, Data
4 points by behnamoh 13 days ago | 0 comments
Crawling More Politely Than Big Tech (cameronboehmer.com)
Dennis Schubert, engineer at Mozilla and noteworthy contributor to diapsora, a distributed, open-source social network, recently observed that 70% of the load on diaspora's servers was coming from poorly-behaved bots that feed the LLMs of a few big outfits.
Open Source, Artificial Intelligence, Web Crawling, Social Networks
43 points by pkghost 16 days ago | 17 comments
Web Crawler and Scraper for AI (spider.cloud)
Spider offers the finest data collecting solution. Engineered for speed and scalability, it allows you to elevate your AI projects.
Web Crawling, AI, Data Collection, Software, Tools
8 points by catskindleyou 30 days ago | 4 comments
Crawl4AI: Open-Source Web Crawler for Seamless AI Data Scraping (github.com/unclecode)
Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
Web Crawling, AI, Open Source
6 points by ProbeCraft 111 days ago | 0 comments
AI Has Created a Battle over Web Crawling (ieee.org)
AI, Web Crawling, Legal Issues
60 points by pseudolus 139 days ago | 52 comments
AI crawlers need to be more respectful (readthedocs.com)
Artificial Intelligence, Ethics, Web Crawling
226 points by pneff 177 days ago | 118 comments
Reddit has updated its robots.txt to block all web crawlers (stackdiary.com)
Web Crawling, Robots.txt, Reddit, Social Media
19 points by skilled 198 days ago | 15 comments
OpenAI and Anthropic are ignoring robots.txt (businessinsider.com)
Robotics, OpenAI, Anthropic, Web Crawling
18 points by Handy-Man 210 days ago | 6 comments
Show HN: Yomuco – A simple web crawling library for Node.js (github.com/andraindrops)
Web Crawling, Node.js, JavaScript, Software, Libraries
23 points by jtakahashi64 251 days ago | 3 comments