Hacker News with Generative AI: Web Scraping

Maxun: Open-Source No-Code Web Data Extraction Platform (github.com/getmaxun)
Maxun lets you train a robot in 2 minutes and scrape the web on auto-pilot. Web data extraction doesn't get easier than this!

Open Source, Web Scraping, Data Extraction, Automation

58 points by thunderbong 12 days ago | 8 comments

Show HN: Convert any website into a React component (chromewebstore.google.com)

Web Development, React, Chrome Extensions, Web Scraping

326 points by alexdanilowicz 17 days ago | 62 comments

Nearly 90% of our AI crawler traffic is from ByteDance (haproxy.com)
This month, Fortune.com reported that TikTok’s web scraper — known as Bytespider — is aggressively sucking up content to fuel generative AI models. We noticed the same thing when looking at bot management analytics produced by HAProxy Edge — our global network that we ourselves use to serve traffic for haproxy.com. Some of the numbers we are seeing are fairly shocking, so let’s review the traffic sources and where they originate.

AI, Web Scraping, Traffic Analysis, Generative AI, TikTok

95 points by jcat123 21 days ago | 43 comments

Annoyed Redditors tanking Google Search results illustrates peril of AI scrapers (arstechnica.com)
A trend on Reddit that sees Londoners giving false restaurant recommendations in order to keep their favorites clear of tourists and social media influencers highlights the inherent flaws of Google Search’s reliance on Reddit and Google's AI Overview.

AI, Search Engines, Social Media, Web Scraping, Consumer Behavior

5 points by isaacfrond 23 days ago | 0 comments

You-get: Dumb downloader that scrapes the web (github.com/soimort)
You-Get is a tiny command-line utility to download media contents (videos, audios, images) from the Web, in case there is no other handy way to do it.

Command Line, Web Scraping, Downloading, Media

397 points by Anon84 25 days ago | 146 comments

Show HN: Epublifier – scrape pages (books, manuals) for offline reading (github.com/maoserr)
Converts some webnovels to epub format

Web Scraping, Ebooks, Software, Offline Reading, Webnovels

290 points by maoserr 31 days ago | 43 comments

Video scraping: extracting JSON from a 35s screen capture for 1/10th of a cent (simonwillison.net)
The other day I found myself needing to add up some numeric values that were scattered across twelve different emails.

Web Scraping, Data Extraction, Automation, Cost Optimization, Efficiency

309 points by simonw 35 days ago | 46 comments

Show HN: Web Scraping with AI (hystruct.com)
Hystruct uses AI to help you scrape the web with ease. Get started with 4000 free credits today.

Web Scraping, AI, Tools, Software

4 points by alexpate 44 days ago | 1 comments

ByteDance’s Bytespider is scraping at much higher rates than other platforms (fortune.com)
ByteDance looks like it’s eager to make up for lost time when it comes to scraping the web for data needed to train its generative AI models.

AI, Web Scraping, Data, ByteDance, Generative AI

141 points by wmstack 45 days ago | 93 comments

Web scraping with your web browser: Why not? (8chananon.github.io)
You can find plenty of tutorials on the Internet about the art of web scraping (for example, here and here) and the first things you will learn about are Python and Beautiful Soup. There is no tutorial on web scraping with Javascript in a web browser though you will find browser extensions that claim to do it all without any need for coding (this only works for simplistic and unprotected websites).

Web Scraping, JavaScript, Browser Extensions

150 points by 8chanAnon 51 days ago | 73 comments

Show HN: A tool to import and manage "Who Is Hiring" posts (github.com/gabfl)
This application allows you to scrape, store, and interactively explore job postings from Hacker News’s “Who is Hiring?” threads.

Show HN, Job Searching, Hacker News, Web Scraping, Tools

9 points by gabriel_b 51 days ago | 0 comments

Show HN: Pipet – CLI tool for scraping and extracting data online, with pipes (github.com/bjesus)
Pipet is a command line based web scraper. It supports 3 modes of operation - HTML parsing, JSON parsing, and client-side JavaScript evaluation. It relies heavily on existing tools like curl, and it uses unix pipes for extending its built-in capabilities.

Command Line, Web Scraping, Tools, Programming, Data Extraction

49 points by yoavm 52 days ago | 7 comments

The most accurate and cheapest AI for scraping (ortutay.substack.com)
AI models have the potential to make web scraping 10x easier. Instead of writing complicated code like XPath and CSS selectors, you can scrape a website with plain English. That’s the idea behind FetchFox, a Chrome extension that scrapes any website using AI.

Web Scraping, Artificial Intelligence, Chrome Extensions

20 points by marcell 57 days ago | 5 comments

Show HN: Finic – Open source platform for building browser automations (github.com/finic-ai)
Finic is a cloud platform designed to simplify the deployment and management of browser-based automation agents, with a focus on fault-tolerant execution. It’s designed for quickly launching bots, scrapers, RPA integrationsm and other jobs that depend on multiple authenticated web services.

Open Source, Browser Automation, Cloud Platform, Web Scraping

143 points by jasonwcfan 65 days ago | 77 comments

Show HN: I'm making an AI scraper called FetchFox (fetchfoxai.com)

Artificial Intelligence, Web Scraping, Software, Show HN

105 points by marcell 78 days ago | 59 comments

Unlock Articles with Paywallskip (paywallskip.com)

Web Development, Tools, Access Control, Web Scraping

55 points by francocanzani 80 days ago | 62 comments

Major Sites Are Saying No to Apple's AI Scraping (wired.com)

Artificial Intelligence, Web Scraping, Apple

86 points by marban 84 days ago | 83 comments

A web scraping CLI made for AI that is idempotent (github.com/clemlesne)