Hacker News with Generative AI: Data Pipelines

Dbt Labs acquires SDF Labs (getdbt.com)
The TL;DR: today, I have the pleasure of announcing that dbt Labs has acquired SDF Labs. The two teams are already working side-by-side to bring SDF’s SQL comprehension technology into the hands of dbt users everywhere. SDF will be a massive upgrade to the very heart of the dbt user experience moving forward.

Software, Data Science, Acquisitions, Data Pipelines

108 points by karakanb 4 days ago | 48 comments

Show HN: I built an open-source data pipeline tool in Go (github.com/bruin-data)
Bruin is a data pipeline tool that brings together data ingestion, data transformation with SQL & Python, and data quality into a single framework.

Open Source, Data Pipelines, Go, Software, Tools

200 points by karakanb 32 days ago | 48 comments

Reducing the cost of a single Google Cloud Dataflow Pipeline by Over 60% (allegro.tech)
In this article we’ll present methods for efficiently optimizing physical resources and fine-tuning the configuration of a Google Cloud Platform (GCP) Dataflow pipeline in order to achieve cost reductions.

Cloud Computing, Data Pipelines, Optimization

9 points by jimminyx 64 days ago | 2 comments

Postgres Meets Analytics: CDC from Neon to ClickHouse via PeerDB (neon.tech)
Combining ClickHouse and Neon for real-time analytics on transactional data

Databases, Real-time Analytics, Cloud Computing, Data Pipelines

3 points by shenli3514 107 days ago | 0 comments

Understanding Airflow DAG and Task Concurrency on Google Cloud Composer (cloud.google.com)

Airflow, Concurrency, Data Pipelines, Cloud Computing

32 points by meysamazad 174 days ago | 3 comments

Large language model data pipelines and Common Crawl (christianperone.com)

Data Pipelines, Common Crawl

139 points by sonabinu 213 days ago | 12 comments

Koheesio: Nike's Python-based framework to build advanced data-pipelines (github.com/Nike-Inc)

Python, Data Pipelines, Open Source, Nike, Frameworks

217 points by betacar 228 days ago | 73 comments

Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines (github.com/DAGWorks-Inc)