Hacker News with Generative AI: Data Engineering

A Deep Dive into Ingesting Debezium Events from Kafka with Flink SQL (morling.dev)
Over the years, I’ve spoken quite a bit about the use cases for processing Debezium data change events with Apache Flink, such as metadata enrichment, building denormalized data views, and creating data contracts for your CDC streams.

Apache Flink, Kafka, Data Streaming, Data Engineering

6 points by code_reader 6 days ago | 0 comments

Data engineering to find domains pointing to certain CNAMEs (flaky.build)
When you’re using custom domains in web platforms you need to typically use CNAME records.

Data Engineering, Domains, CNAME Records, Web Platforms

13 points by onnimonni 51 days ago | 1 comments

Parser, Better, Faster, Stronger: A peek at the new dbt engine (getdbt.com)
Remember how dbt felt when you had a small project? You pressed enter and stuff just happened immediately? We're bringing that back.

Software, Data Engineering, Performance, dbt

9 points by data_ders 66 days ago | 1 comments

Building an Open, Multi-Engine Data Lakehouse with S3 and Python (tower.dev)
The idea of open, multi-engine data lakehouses is gaining momentum in the data industry.

Data Lakehouse, Open Source, Python, Data Engineering, Cloud Computing

71 points by bradhe 67 days ago | 11 comments

Logs Don't Lie: Debugging My Data Engineering Crisis in 2025 (datobra.com)
Every data pipeline has its breaking point. Mine came in late 2024, throwing errors I couldn’t ignore. Logs showed signs of stagnation, over-processing, and the need for a refreshed perspective.

Data Engineering, Debugging, Software Development, 2025

3 points by olgazju 107 days ago | 0 comments

Should you ditch Spark for DuckDB or Polars? (milescole.dev)
There’s been a lot of excitement lately about single-machine compute engines like DuckDB and Polars. With the recent release of pure Python Notebooks in Microsoft Fabric, the excitement about these lightweight native engines has risen to a new high. Out with Spark and in with the new and cool animal-themed engines— is it time to finally migrate your small and medium workloads off of Spark?

Data Analysis, Databases, Python, Data Engineering

169 points by RobinL 133 days ago | 114 comments

Data-engineer-handbook: everything to learn about data engineering (kifinity.com)

Data Engineering, Resources, Learning

61 points by codezerox 143 days ago | 1 comments

We need data engineering benchmarks for LLMs (structuredlabs.substack.com)
Tools like Copilot and GPT-based copilots promise to reduce the repetitive burden of data engineering tasks, suggest code, and even debug complex pipelines. But how do we measure whether they’re actually good at this? Frankly, the industry is lagging behind when it comes to evaluation methods. While SWE-bench offers a framework for software engineering, data engineering is just left out—no tailored benchmarks, no precise way to gauge their effectiveness. It’s time to change that.

Data Engineering, Evaluation Methods, Generative AI

12 points by amrutha_ 146 days ago | 4 comments

Dismantling ELT: The Case for Graphs, Not Silos (jack-vanlightly.com)
ELT is a bridge between silos. A world without silos is a graph.

Data Engineering, Data Architecture, Graph Databases

35 points by sebg 149 days ago | 15 comments

The Data Engineering Handbook (github.com/DataExpert-io)
This repo has all the resources you need to become an amazing data engineer!

Data Engineering, Resources, Software, GitHub

185 points by matthewhefferon 158 days ago | 20 comments

Understanding privacy risk with k-anonymity and l-diversity (marcusolsson.dev)
Imagine you’re a data analyst at a global company who’s been asked to provide employee statistics for a survey on remote working and distributed teams. You’ve extracted the relevant employee data, but sharing it as-is could violate privacy laws. How can you anonymize this data while ensuring it’s still useful? In this article, you’ll learn about k-anonymity and l-diversity—two valuable techniques in privacy engineering to help you reduce the privacy risk in datasets.

Data Privacy, Data Anonymization, Data Engineering, Privacy Engineering, Statistics

75 points by marols 172 days ago | 7 comments

Ask HN: How to learn UI/UX as a data/BE engineer? (ycombinator.com)
Hi HN,<p>coming from a data/ BE background I feel extremely familiar with reasoning about systems and performance from the cloud-infra to the pipeline stack level.

Career Advice, UI/UX, Data Engineering, Backend Engineering

51 points by thenaturalist 187 days ago | 28 comments

Dbt – Incremental but Incomplete (tobikodata.com)
Earlier this month, dbtTM launched microbatch incremental models in version 1.9, a highly requested feature since the experimental insert_by_period was introduced back in 2018. While it's certainly a step in the right direction, it has been a long time coming.

Software, Data Engineering, Data Warehousing

67 points by captaintobs 193 days ago | 18 comments

I spent 5 hours learning how ClickHouse built their internal data warehouse (vutr.substack.com)
My name is Vu Trinh, and I am a data engineer.

Data Warehousing, Databases, ClickHouse, Data Engineering, Substack

47 points by markhneedham 214 days ago | 5 comments

Data Engineering Vault: A 1000 Node Second Brain for DE Knowledge (ssp.sh)
Welcome to the Data Engineering Vault an integral part of my Second Brain. It’s a curated network of data engineering knowledge, designed to facilitate exploration and discovery. Here, you’ll find over 100+ interconnected terms, each serving as a gateway to deeper insights.

Data Engineering, Second Brain, Knowledge Management, Information Retrieval, Data Science

123 points by articsputnik 223 days ago | 19 comments

Rɐbbit Dynamic Datascapes (github.com/ryrobes)
As a long-time dashboard builder, data engineer, and UI hacker - I've always wanted something in-between Tableau & building bespoke web data products to ship answers to my users. The tools were too rigid at times, and building everything from scratch can be tiresome. The eternal push/pull of DE and SWE approaches, as many who work in BI can attest to. How could I have the flexibility & re-usability of code, but the compositional freedom & direct manipulation of a builder tool?

Data Visualization, Data Engineering, Software, User Interfaces

69 points by notarobot123 230 days ago | 16 comments

How to Build a Scalable Ingestion Pipeline for Enterprise GenAI Applications (enterprisebot.ai)

Generative AI, Enterprise Applications, Data Engineering, Scalability

7 points by ritzaco 235 days ago | 0 comments

Timeseries Indexing at Scale (krylysov.com)

Data Engineering, Databases, Time Series

95 points by gsky 237 days ago | 14 comments

Exploiting column chunks for faster ingestion and lower memory use (rerun.io)

Database Optimization, Data Engineering, Performance Optimization

15 points by Tycho87 243 days ago | 2 comments

Exploiting column chunks for faster ingestion and lower memory use (rerun.io)