Hacker News with Generative AI: Data Pipelines

Materialize Community Edition (materialize.com)
What happens when you give an engineering team a building block for integrating transactional data and incrementally transforming it into real-time data products and APIs? First, queries that once melted operational databases now return promptly and correctly. Then sprawling services begin to collapse into individual SQL queries. A data pipeline that ran hourly batch jobs now operates in real-time with minimal effort. An SRE has their first full night’s sleep in weeks.

Databases, Real-time Data, Data Pipelines, Software

13 points by nate_stewart 22 days ago | 0 comments

Adding concurrent read/write to DuckDB with Arrow Flight (definite.app)
We've been thinking a lot about latency, streaming and (near) real-time analytics lately. At Definite, we deal with a lot of data pipelines. In most cases (e.g. ingesting Stripe data), our customers are fine with batch processing (e.g. every hour). But as we've grown, we've seen more and more need for near real-time pipelines (e.g. ingesting events or CDC from Postgres).

Database, Data Pipelines, Real-Time Analytics, Streaming Data, Arrow Flight

94 points by mritchie712 63 days ago | 32 comments

Dbt Labs acquires SDF Labs (getdbt.com)
The TL;DR: today, I have the pleasure of announcing that dbt Labs has acquired SDF Labs. The two teams are already working side-by-side to bring SDF’s SQL comprehension technology into the hands of dbt users everywhere. SDF will be a massive upgrade to the very heart of the dbt user experience moving forward.

Software, Data Science, Acquisitions, Data Pipelines

109 points by karakanb 78 days ago | 48 comments

Show HN: I built an open-source data pipeline tool in Go (github.com/bruin-data)
Bruin is a data pipeline tool that brings together data ingestion, data transformation with SQL & Python, and data quality into a single framework.

Open Source, Data Pipelines, Go, Software, Tools

200 points by karakanb 105 days ago | 48 comments

Reducing the cost of a single Google Cloud Dataflow Pipeline by Over 60% (allegro.tech)
In this article we’ll present methods for efficiently optimizing physical resources and fine-tuning the configuration of a Google Cloud Platform (GCP) Dataflow pipeline in order to achieve cost reductions.

Cloud Computing, Data Pipelines, Optimization

9 points by jimminyx 138 days ago | 2 comments

Postgres Meets Analytics: CDC from Neon to ClickHouse via PeerDB (neon.tech)
Combining ClickHouse and Neon for real-time analytics on transactional data

Databases, Real-time Analytics, Cloud Computing, Data Pipelines

3 points by shenli3514 181 days ago | 0 comments

Understanding Airflow DAG and Task Concurrency on Google Cloud Composer (cloud.google.com)

Airflow, Concurrency, Data Pipelines, Cloud Computing

32 points by meysamazad 248 days ago | 3 comments

Large language model data pipelines and Common Crawl (christianperone.com)

Data Pipelines, Common Crawl

139 points by sonabinu 287 days ago | 12 comments

Koheesio: Nike's Python-based framework to build advanced data-pipelines (github.com/Nike-Inc)

Python, Data Pipelines, Open Source, Nike, Frameworks

217 points by betacar 302 days ago | 73 comments

Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines (github.com/DAGWorks-Inc)