Scrapy needs to have sane defaults that do no harm
(github.com/scrapy)
Scrapy needs to have sane defaults that do no harm #6597
Scrapy needs to have sane defaults that do no harm #6597
The Web Is Broken – Botnet Part 2
(wildeboer.net)
I guess you have all heard about the growing problem of AI companies trying to aggressively collect whatever data they can get their hands on to train their models. This has caused an explosive surge in web crawlers relentlessly hitting servers big and small. But who runs these crawlers? Turns out — it could be you!
I guess you have all heard about the growing problem of AI companies trying to aggressively collect whatever data they can get their hands on to train their models. This has caused an explosive surge in web crawlers relentlessly hitting servers big and small. But who runs these crawlers? Turns out — it could be you!
Crawler operators, please stop destroying the commons
(lunnova.dev)
Goodwill for crawling/scraping has rapidly been depleted
Goodwill for crawling/scraping has rapidly been depleted
Abusive AI Web Crawlers: Get Off My Lawn
(mythic-beasts.com)
As many other folks have reported in the last few weeks, we have also been seeing a huge increase in the amount of traffic from abusive web crawlers.
As many other folks have reported in the last few weeks, we have also been seeing a huge increase in the amount of traffic from abusive web crawlers.
Cloudflare is luring web-scraping bots into an 'AI Labyrinth'
(theverge.com)
Cloudflare, one of the biggest network internet infrastructure companies in the world, has announced AI Labyrinth, a new tool to fight web-crawling bots that scrape sites for AI training data without permission.
Cloudflare, one of the biggest network internet infrastructure companies in the world, has announced AI Labyrinth, a new tool to fight web-crawling bots that scrape sites for AI training data without permission.
Improved ways to operate a rude crawler
(marginalia.nu)
Tech news is abuzz with rude AI crawlers that forge their user-agent and ignore robots.txt. In my opinion, if this is all the AI startups can muster, they’re losing their touch. wget can do this. You need to up your game, get that crawler really rolling coal. Flagrant disregard for externalities is an important signal to the investors that your AI startup is the one.
Tech news is abuzz with rude AI crawlers that forge their user-agent and ignore robots.txt. In my opinion, if this is all the AI startups can muster, they’re losing their touch. wget can do this. You need to up your game, get that crawler really rolling coal. Flagrant disregard for externalities is an important signal to the investors that your AI startup is the one.
Show HN: Hyperbrowser MCP Server – Connect AI agents to the web through browsers
(github.com/hyperbrowserai)
This is Hyperbrowser's Model Context Protocol (MCP) Server. It provides various tools to scrape, extract structured data, and crawl webpages. It also provides easy access to general purpose browser agents like OpenAI's CUA, Anthropic's Claude Computer Use, and Browser Use.
This is Hyperbrowser's Model Context Protocol (MCP) Server. It provides various tools to scrape, extract structured data, and crawl webpages. It also provides easy access to general purpose browser agents like OpenAI's CUA, Anthropic's Claude Computer Use, and Browser Use.
Fetch-MCP: Playwright-Based MCP Server with Batch URL Fetching Support
(github.com/jae-jae)
MCP server for fetch web page content using Playwright headless browser.
MCP server for fetch web page content using Playwright headless browser.
Show HN: I scrape Steam data every month and it's yours to download for free
(gginsights.io)
Leverage the power of AI to help answer your questions about the Steam market and become a data expert, transforming data into actionable insights.
Leverage the power of AI to help answer your questions about the Steam market and become a data expert, transforming data into actionable insights.
Show HN: Automatic Python scraper for xenforo, phpbb, invision, smf, vbulletin
(github.com/TUVIMEN)
forumscraper aims to be an universal, automatic and extensive scraper for forums.
forumscraper aims to be an universal, automatic and extensive scraper for forums.
Show HN: API Parrot – Automatically Reverse Engineer HTTP APIs
(apiparrot.com)
API Parrot is the tool specifically designed to reverese engineer the HTTP APIs of any website. Making life easier for developers looking to automate, integrate or scrape websites without public APIs.
API Parrot is the tool specifically designed to reverese engineer the HTTP APIs of any website. Making life easier for developers looking to automate, integrate or scrape websites without public APIs.
Show HN: DataFuel.dev – Turn websites into LLM-ready data
(datafuel.dev)
DataFuel API scrapes entire websites and knowledge bases in a single query. Get clean, markdown-structured web data instantly for your RAG systems and AI models. No complex scraping code needed.
DataFuel API scrapes entire websites and knowledge bases in a single query. Get clean, markdown-structured web data instantly for your RAG systems and AI models. No complex scraping code needed.
What Are the Latest Scraping APIs or Services/Websites?
(ycombinator.com)
I'm working on a large web scraping project and looking to avoid building custom scrapers for every site using something like BeautifulSoup.
I'm working on a large web scraping project and looking to avoid building custom scrapers for every site using something like BeautifulSoup.
Maxun: Open-Source No-Code Web Data Extraction Platform
(github.com/getmaxun)
Maxun lets you train a robot in 2 minutes and scrape the web on auto-pilot. Web data extraction doesn't get easier than this!
Maxun lets you train a robot in 2 minutes and scrape the web on auto-pilot. Web data extraction doesn't get easier than this!
Nearly 90% of our AI crawler traffic is from ByteDance
(haproxy.com)
This month, Fortune.com reported that TikTok’s web scraper — known as Bytespider — is aggressively sucking up content to fuel generative AI models. We noticed the same thing when looking at bot management analytics produced by HAProxy Edge — our global network that we ourselves use to serve traffic for haproxy.com. Some of the numbers we are seeing are fairly shocking, so let’s review the traffic sources and where they originate.
This month, Fortune.com reported that TikTok’s web scraper — known as Bytespider — is aggressively sucking up content to fuel generative AI models. We noticed the same thing when looking at bot management analytics produced by HAProxy Edge — our global network that we ourselves use to serve traffic for haproxy.com. Some of the numbers we are seeing are fairly shocking, so let’s review the traffic sources and where they originate.
Annoyed Redditors tanking Google Search results illustrates peril of AI scrapers
(arstechnica.com)
A trend on Reddit that sees Londoners giving false restaurant recommendations in order to keep their favorites clear of tourists and social media influencers highlights the inherent flaws of Google Search’s reliance on Reddit and Google's AI Overview.
A trend on Reddit that sees Londoners giving false restaurant recommendations in order to keep their favorites clear of tourists and social media influencers highlights the inherent flaws of Google Search’s reliance on Reddit and Google's AI Overview.
You-get: Dumb downloader that scrapes the web
(github.com/soimort)
You-Get is a tiny command-line utility to download media contents (videos, audios, images) from the Web, in case there is no other handy way to do it.
You-Get is a tiny command-line utility to download media contents (videos, audios, images) from the Web, in case there is no other handy way to do it.
Show HN: Epublifier – scrape pages (books, manuals) for offline reading
(github.com/maoserr)
Converts some webnovels to epub format
Converts some webnovels to epub format
Video scraping: extracting JSON from a 35s screen capture for 1/10th of a cent
(simonwillison.net)
The other day I found myself needing to add up some numeric values that were scattered across twelve different emails.
The other day I found myself needing to add up some numeric values that were scattered across twelve different emails.
Show HN: Web Scraping with AI
(hystruct.com)
Hystruct uses AI to help you scrape the web with ease. Get started with 4000 free credits today.
Hystruct uses AI to help you scrape the web with ease. Get started with 4000 free credits today.
ByteDance’s Bytespider is scraping at much higher rates than other platforms
(fortune.com)
ByteDance looks like it’s eager to make up for lost time when it comes to scraping the web for data needed to train its generative AI models.
ByteDance looks like it’s eager to make up for lost time when it comes to scraping the web for data needed to train its generative AI models.
Web scraping with your web browser: Why not?
(8chananon.github.io)
You can find plenty of tutorials on the Internet about the art of web scraping (for example, here and here) and the first things you will learn about are Python and Beautiful Soup. There is no tutorial on web scraping with Javascript in a web browser though you will find browser extensions that claim to do it all without any need for coding (this only works for simplistic and unprotected websites).
You can find plenty of tutorials on the Internet about the art of web scraping (for example, here and here) and the first things you will learn about are Python and Beautiful Soup. There is no tutorial on web scraping with Javascript in a web browser though you will find browser extensions that claim to do it all without any need for coding (this only works for simplistic and unprotected websites).
Show HN: A tool to import and manage "Who Is Hiring" posts
(github.com/gabfl)
This application allows you to scrape, store, and interactively explore job postings from Hacker News’s “Who is Hiring?” threads.
This application allows you to scrape, store, and interactively explore job postings from Hacker News’s “Who is Hiring?” threads.
Show HN: Pipet – CLI tool for scraping and extracting data online, with pipes
(github.com/bjesus)
Pipet is a command line based web scraper. It supports 3 modes of operation - HTML parsing, JSON parsing, and client-side JavaScript evaluation. It relies heavily on existing tools like curl, and it uses unix pipes for extending its built-in capabilities.
Pipet is a command line based web scraper. It supports 3 modes of operation - HTML parsing, JSON parsing, and client-side JavaScript evaluation. It relies heavily on existing tools like curl, and it uses unix pipes for extending its built-in capabilities.
The most accurate and cheapest AI for scraping
(ortutay.substack.com)
AI models have the potential to make web scraping 10x easier. Instead of writing complicated code like XPath and CSS selectors, you can scrape a website with plain English. That’s the idea behind FetchFox, a Chrome extension that scrapes any website using AI.
AI models have the potential to make web scraping 10x easier. Instead of writing complicated code like XPath and CSS selectors, you can scrape a website with plain English. That’s the idea behind FetchFox, a Chrome extension that scrapes any website using AI.
Show HN: Finic – Open source platform for building browser automations
(github.com/finic-ai)
Finic is a cloud platform designed to simplify the deployment and management of browser-based automation agents, with a focus on fault-tolerant execution. It’s designed for quickly launching bots, scrapers, RPA integrationsm and other jobs that depend on multiple authenticated web services.
Finic is a cloud platform designed to simplify the deployment and management of browser-based automation agents, with a focus on fault-tolerant execution. It’s designed for quickly launching bots, scrapers, RPA integrationsm and other jobs that depend on multiple authenticated web services.