Hacker News with Generative AI: Data Processing

Differential Dataflow for the Masses (github.com/brurucy)
This library provides an implementation of the DBSP language for incremental streaming computations.

Data Processing, Programming, Software, Big Data, Streaming

42 points by rebanevapustus 13 days ago | 11 comments

How to Flatten nested JSON arrays (datazip.io)
Flattening nested JSON or MongoDB’s BSON or normalizing semi-structured data and writing queries on it for analytics or regular queries, is a common challenge in data processing.

Data Processing, JSON, Data Structures, Data Analytics

14 points by pkhodiyar 15 days ago | 2 comments

pg_flo – Stream, transform, and re-route PostgreSQL data in real-time (pgflo.io)
The easiest way to move and transform data between PostgreSQL databases

PostgreSQL, Data Processing, Real-time Data, Databases, Software

239 points by shayonj 18 days ago | 54 comments

Drasi: Microsoft's open source data processing platform for event-driven systems (github.com/drasi-project)
Drasi is a data processing platform that simplifies detecting changes in data and taking immediate action.

Data Processing, Open Source, Microsoft, Event-Driven Systems, Data Streaming

331 points by benocodes 32 days ago | 67 comments

No such thing as exactly-once delivery (sequinstream.com)
We say Sequin is system with "at-least-once delivery" and "exactly-once processing" guarantees.

Distributed Systems, Data Processing, Guarantees, Software Architecture

99 points by todsacerdoti 52 days ago | 114 comments

Parsing Gigabytes of JSON per Second (arxiv.org)
JavaScript Object Notation or JSON is a ubiquitous data exchange format on the Web. Ingesting JSON documents can become a performance bottleneck due to the sheer volume of data. We are thus motivated to make JSON parsing as fast as possible.

Performance Optimization, Data Processing, JSON, Web Development, Algorithms

5 points by ibobev 71 days ago | 1 comments

Sail – Unify stream processing, batch processing and compute-intensive workloads (github.com/lakehq)
LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

Data Processing, Stream Processing, Batch Processing, Big Data, Artificial Intelligence

79 points by chenxi9649 72 days ago | 15 comments

I use Nim instead of Python for data processing (2021) (benjamindlee.com)
Lazy programmers often prefer to substitute computing effort for programming effort. I am just such a programmer. For my research, I often need to design and run algorithms over large datasets ranging into the scale of terabytes. As a fellow at the NIH, I have access to Biowulf, a 100,000+ processor cluster, so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big [MapReduce](https://en.wikipedia.org/wiki/MapReduce).

Programming Languages, Data Processing, Research, Performance Optimization

95 points by archargelod 76 days ago | 80 comments

Reflection-based JSON in C++ at Gigabytes per Second (lemire.me)

C++, JSON, Performance, Optimization, Data Processing

14 points by jandeboevrie 98 days ago | 1 comments

Pipes: A spiritual successor to Yahoo Pipes (pipes.digital)

Web Development, Social Media, Data Processing, RSS

466 points by sea-gold 143 days ago | 130 comments

Show HN: Qq: like jq, but can transcode between many formats (github.com/JFryy)

Data Processing, Software, Open Source

115 points by arandomhuman 149 days ago | 27 comments

TSV – Alternative to CSV (wikipedia.org)