Hacker News with Generative AI: Web Scraping

Show HN: Automatic Python scraper for xenforo, phpbb, invision, smf, vbulletin (github.com/TUVIMEN)
forumscraper aims to be an universal, automatic and extensive scraper for forums.

Web Scraping, Python, Forums, Software, Open Source

3 points by TUVIMEN 10 days ago | 0 comments

Show HN: API Parrot – Automatically Reverse Engineer HTTP APIs (apiparrot.com)
API Parrot is the tool specifically designed to reverese engineer the HTTP APIs of any website. Making life easier for developers looking to automate, integrate or scrape websites without public APIs.

API Development, Web Development, Reverse Engineering, Automation, Web Scraping

456 points by pvarghav 17 days ago | 117 comments

Show HN: DataFuel.dev – Turn websites into LLM-ready data (datafuel.dev)
DataFuel API scrapes entire websites and knowledge bases in a single query. Get clean, markdown-structured web data instantly for your RAG systems and AI models. No complex scraping code needed.

Web Scraping, AI, Data

43 points by sachou 36 days ago | 34 comments

Show HN: App to discover job listings directly from company websites (unlistedjobs.com)

Job Hunting, Web Scraping, Software, Startups

67 points by Jabbs 42 days ago | 57 comments

What Are the Latest Scraping APIs or Services/Websites? (ycombinator.com)
I'm working on a large web scraping project and looking to avoid building custom scrapers for every site using something like BeautifulSoup.

Web Scraping, APIs, Services, Websites

6 points by jdcampolargo 44 days ago | 7 comments

Maxun: Open-Source No-Code Web Data Extraction Platform (github.com/getmaxun)
Maxun lets you train a robot in 2 minutes and scrape the web on auto-pilot. Web data extraction doesn't get easier than this!

Open Source, Web Scraping, Data Extraction, Automation

58 points by thunderbong 70 days ago | 8 comments

Show HN: Convert any website into a React component (chromewebstore.google.com)

Web Development, React, Chrome Extensions, Web Scraping

326 points by alexdanilowicz 74 days ago | 62 comments

Nearly 90% of our AI crawler traffic is from ByteDance (haproxy.com)
This month, Fortune.com reported that TikTok’s web scraper — known as Bytespider — is aggressively sucking up content to fuel generative AI models. We noticed the same thing when looking at bot management analytics produced by HAProxy Edge — our global network that we ourselves use to serve traffic for haproxy.com. Some of the numbers we are seeing are fairly shocking, so let’s review the traffic sources and where they originate.

AI, Web Scraping, Traffic Analysis, Generative AI, TikTok

95 points by jcat123 78 days ago | 43 comments

Annoyed Redditors tanking Google Search results illustrates peril of AI scrapers (arstechnica.com)
A trend on Reddit that sees Londoners giving false restaurant recommendations in order to keep their favorites clear of tourists and social media influencers highlights the inherent flaws of Google Search’s reliance on Reddit and Google's AI Overview.

AI, Search Engines, Social Media, Web Scraping, Consumer Behavior

5 points by isaacfrond 81 days ago | 0 comments

You-get: Dumb downloader that scrapes the web (github.com/soimort)
You-Get is a tiny command-line utility to download media contents (videos, audios, images) from the Web, in case there is no other handy way to do it.

Command Line, Web Scraping, Downloading, Media

397 points by Anon84 83 days ago | 146 comments

Show HN: Epublifier – scrape pages (books, manuals) for offline reading (github.com/maoserr)
Converts some webnovels to epub format

Web Scraping, Ebooks, Software, Offline Reading, Webnovels

290 points by maoserr 89 days ago | 43 comments

Video scraping: extracting JSON from a 35s screen capture for 1/10th of a cent (simonwillison.net)
The other day I found myself needing to add up some numeric values that were scattered across twelve different emails.

Web Scraping, Data Extraction, Automation, Cost Optimization, Efficiency

309 points by simonw 93 days ago | 46 comments

Show HN: Web Scraping with AI (hystruct.com)
Hystruct uses AI to help you scrape the web with ease. Get started with 4000 free credits today.

Web Scraping, AI, Tools, Software

4 points by alexpate 102 days ago | 1 comments

ByteDance’s Bytespider is scraping at much higher rates than other platforms (fortune.com)
ByteDance looks like it’s eager to make up for lost time when it comes to scraping the web for data needed to train its generative AI models.

AI, Web Scraping, Data, ByteDance, Generative AI

141 points by wmstack 103 days ago | 93 comments

Web scraping with your web browser: Why not? (8chananon.github.io)
You can find plenty of tutorials on the Internet about the art of web scraping (for example, here and here) and the first things you will learn about are Python and Beautiful Soup. There is no tutorial on web scraping with Javascript in a web browser though you will find browser extensions that claim to do it all without any need for coding (this only works for simplistic and unprotected websites).

Web Scraping, JavaScript, Browser Extensions

150 points by 8chanAnon 108 days ago | 73 comments

Show HN: A tool to import and manage "Who Is Hiring" posts (github.com/gabfl)
This application allows you to scrape, store, and interactively explore job postings from Hacker News’s “Who is Hiring?” threads.

Show HN, Job Searching, Hacker News, Web Scraping, Tools

9 points by gabriel_b 108 days ago | 0 comments

Show HN: Pipet – CLI tool for scraping and extracting data online, with pipes (github.com/bjesus)
Pipet is a command line based web scraper. It supports 3 modes of operation - HTML parsing, JSON parsing, and client-side JavaScript evaluation. It relies heavily on existing tools like curl, and it uses unix pipes for extending its built-in capabilities.

Command Line, Web Scraping, Tools, Programming, Data Extraction

49 points by yoavm 110 days ago | 7 comments

The most accurate and cheapest AI for scraping (ortutay.substack.com)
AI models have the potential to make web scraping 10x easier. Instead of writing complicated code like XPath and CSS selectors, you can scrape a website with plain English. That’s the idea behind FetchFox, a Chrome extension that scrapes any website using AI.

Web Scraping, Artificial Intelligence, Chrome Extensions

20 points by marcell 115 days ago | 5 comments

Show HN: Finic – Open source platform for building browser automations (github.com/finic-ai)
Finic is a cloud platform designed to simplify the deployment and management of browser-based automation agents, with a focus on fault-tolerant execution. It’s designed for quickly launching bots, scrapers, RPA integrationsm and other jobs that depend on multiple authenticated web services.

Open Source, Browser Automation, Cloud Platform, Web Scraping

143 points by jasonwcfan 123 days ago | 77 comments

Show HN: I'm making an AI scraper called FetchFox (fetchfoxai.com)

Artificial Intelligence, Web Scraping, Software, Show HN

105 points by marcell 136 days ago | 59 comments

Unlock Articles with Paywallskip (paywallskip.com)

Web Development, Tools, Access Control, Web Scraping

55 points by francocanzani 138 days ago | 62 comments

Major Sites Are Saying No to Apple's AI Scraping (wired.com)

Artificial Intelligence, Web Scraping, Apple

86 points by marban 142 days ago | 83 comments

A web scraping CLI made for AI that is idempotent (github.com/clemlesne)