Hacker News with Generative AI: Data Pipelines

Bauplan – Git-for-data pipelines on object storage (bauplanlabs.com)
Bauplan is a Pythonic data platform that provides functions as a service for large-scale data pipelines and git-for-data over S3 data lakes.

Data Pipelines, Python, Object Storage, Git, Cloud Storage

83 points by barabbababoon 10 days ago | 45 comments

Ask HN: Why don't we have a functional DSL for data+embedding+API pipelines? (ycombinator.com)
I’ve been working on a pretty common problem: <p><pre> - I have structured data in JSONL files (in.jsonl, out.jsonl) - I match lines by a key - I transform them into (text, embedding) pairs - I optionally filter/map them - I batch them (into chunks of 50) - I push each batch into an external system (e.g. vector DB, Chroma) </pre> That’s it. Sounds trivial.

Data Pipelines, Machine Learning, API

21 points by codingmoh 12 days ago | 21 comments

Materialize Community Edition (materialize.com)
What happens when you give an engineering team a building block for integrating transactional data and incrementally transforming it into real-time data products and APIs? First, queries that once melted operational databases now return promptly and correctly. Then sprawling services begin to collapse into individual SQL queries. A data pipeline that ran hourly batch jobs now operates in real-time with minimal effort. An SRE has their first full night’s sleep in weeks.

Databases, Real-time Data, Data Pipelines, Software

13 points by nate_stewart 46 days ago | 0 comments

Adding concurrent read/write to DuckDB with Arrow Flight (definite.app)
We've been thinking a lot about latency, streaming and (near) real-time analytics lately. At Definite, we deal with a lot of data pipelines. In most cases (e.g. ingesting Stripe data), our customers are fine with batch processing (e.g. every hour). But as we've grown, we've seen more and more need for near real-time pipelines (e.g. ingesting events or CDC from Postgres).

Database, Data Pipelines, Real-Time Analytics, Streaming Data, Arrow Flight

94 points by mritchie712 87 days ago | 32 comments

Dbt Labs acquires SDF Labs (getdbt.com)
The TL;DR: today, I have the pleasure of announcing that dbt Labs has acquired SDF Labs. The two teams are already working side-by-side to bring SDF’s SQL comprehension technology into the hands of dbt users everywhere. SDF will be a massive upgrade to the very heart of the dbt user experience moving forward.

Software, Data Science, Acquisitions, Data Pipelines

109 points by karakanb 102 days ago | 48 comments

Show HN: I built an open-source data pipeline tool in Go (github.com/bruin-data)
Bruin is a data pipeline tool that brings together data ingestion, data transformation with SQL & Python, and data quality into a single framework.

Open Source, Data Pipelines, Go, Software, Tools

200 points by karakanb 130 days ago | 48 comments

Reducing the cost of a single Google Cloud Dataflow Pipeline by Over 60% (allegro.tech)
In this article we’ll present methods for efficiently optimizing physical resources and fine-tuning the configuration of a Google Cloud Platform (GCP) Dataflow pipeline in order to achieve cost reductions.

Cloud Computing, Data Pipelines, Optimization

9 points by jimminyx 162 days ago | 2 comments

Postgres Meets Analytics: CDC from Neon to ClickHouse via PeerDB (neon.tech)
Combining ClickHouse and Neon for real-time analytics on transactional data

Databases, Real-time Analytics, Cloud Computing, Data Pipelines

3 points by shenli3514 205 days ago | 0 comments

Understanding Airflow DAG and Task Concurrency on Google Cloud Composer (cloud.google.com)

Airflow, Concurrency, Data Pipelines, Cloud Computing

32 points by meysamazad 272 days ago | 3 comments

Large language model data pipelines and Common Crawl (christianperone.com)

Data Pipelines, Common Crawl

139 points by sonabinu 311 days ago | 12 comments

Koheesio: Nike's Python-based framework to build advanced data-pipelines (github.com/Nike-Inc)

Python, Data Pipelines, Open Source, Nike, Frameworks

217 points by betacar 326 days ago | 73 comments

Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines (github.com/DAGWorks-Inc)