Hacker News with Generative AI: Web Crawling

Show HN: Crawlspace – A centralized web crawling platform built on Cloudflare (crawlspace.dev)
Crawlspace is a centralized platform for developers to build and deploy web crawlers. Gather fresh data for your apps and agents while contributing to a platform-wide cache for crawler traffic.
Web Crawling, Cloud Computing, Developer Tools, Software
9 points by andrethegiant 30 days ago | 1 comments
Amazon's AI crawler is making my Git server unstable (xeiaso.net)
Please, just stop.
Amazon, Git, AI, Web Crawling, Server Stability
607 points by OptionOfT 33 days ago | 246 comments
25% of the top websites are blocking OpenAI from crawling (originality.ai)
Bots such as OpenAI’s GPTBot, the Applebot, CCBot, Google-Extended, and Bytespider analyze, store, or scrape your website’s data in order to provide data to train more advanced LLMs.
Web Crawling, Privacy, Artificial Intelligence, Data
4 points by behnamoh 46 days ago | 0 comments
Crawling More Politely Than Big Tech (cameronboehmer.com)
Dennis Schubert, engineer at Mozilla and noteworthy contributor to diapsora, a distributed, open-source social network, recently observed that 70% of the load on diaspora's servers was coming from poorly-behaved bots that feed the LLMs of a few big outfits.
Open Source, Artificial Intelligence, Web Crawling, Social Networks
43 points by pkghost 50 days ago | 17 comments
Web Crawler and Scraper for AI (spider.cloud)
Spider offers the finest data collecting solution. Engineered for speed and scalability, it allows you to elevate your AI projects.
Web Crawling, AI, Data Collection, Software, Tools
8 points by catskindleyou 63 days ago | 4 comments
Crawl4AI: Open-Source Web Crawler for Seamless AI Data Scraping (github.com/unclecode)
Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐
Web Crawling, AI, Open Source
6 points by ProbeCraft 144 days ago | 0 comments
AI Has Created a Battle over Web Crawling (ieee.org)
AI, Web Crawling, Legal Issues
60 points by pseudolus 172 days ago | 52 comments
AI crawlers need to be more respectful (readthedocs.com)
Artificial Intelligence, Ethics, Web Crawling
226 points by pneff 210 days ago | 118 comments
Reddit has updated its robots.txt to block all web crawlers (stackdiary.com)
Web Crawling, Robots.txt, Reddit, Social Media
19 points by skilled 231 days ago | 15 comments
OpenAI and Anthropic are ignoring robots.txt (businessinsider.com)
Robotics, OpenAI, Anthropic, Web Crawling
18 points by Handy-Man 244 days ago | 6 comments
Show HN: Yomuco – A simple web crawling library for Node.js (github.com/andraindrops)
Web Crawling, Node.js, JavaScript, Software, Libraries
23 points by jtakahashi64 285 days ago | 3 comments