Hacker News with Generative AI: Data Processing

Show HN: Open-Source Document Extraction Tool (github.com/harishdeivanayagam)
Rowfill helps extract, analyze, and process data from complex documents, images, PDFs and more with advanced AI capabilities.

Open Source, Document Extraction, AI, Data Processing

13 points by harishd30 3 days ago | 2 comments

Apache DataFusion (apache.org)
DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format.

Query Engines, Apache, Rust, Data Processing

153 points by thebuilderjr 5 days ago | 47 comments

Show HN: Pyper – Concurrent Python Made Simple (github.com/pyper-dev)
Pyper is a flexible framework for concurrent and parallel data-processing, based on functional programming patterns. Used for 🔀 ETL Systems, ⚙️ Data Microservices, and 🌐 Data Collection

Python, Data Processing, Functional Programming, Concurrency

149 points by pyper-dev 6 days ago | 35 comments

Apache DataFusion: Fast, Embeddable, Modular Analytic Query Engine [pdf] (nerdnetworks.org)

Apache DataFusion, Big Data, Query Engines, Data Processing, Open Source

12 points by hambandit 11 days ago | 0 comments

Show HN: Bodo – high-performance compute engine for Python data processing (github.com/bodo-ai)
Bodo is a cutting edge compute engine for large scale Python data processing. Powered by an innovative auto-parallelizing just-in-time compiler, Bodo transforms Python programs into highly optimized, parallel binaries without requiring code rewrites, which makes Bodo 20x to 240x faster compared to alternatives!

Python, Data Processing, Performance Optimization, Open Source, Software

11 points by ehsantn 30 days ago | 2 comments

Sail 0.2: Spark replacement in Rust, runs 4x faster, drop-in PySpark compatible (lakesail.com)
LakeSail is thrilled to unveil a preview release of Sail 0.2, our latest milestone in the journey to redefine distributed data processing.

Rust, Distributed Computing, Data Processing, Big Data, Open Source

14 points by chenxi9649 57 days ago | 2 comments

Differential Dataflow for the Masses (github.com/brurucy)
This library provides an implementation of the DBSP language for incremental streaming computations.

Data Processing, Programming, Software, Big Data, Streaming

42 points by rebanevapustus 70 days ago | 11 comments

How to Flatten nested JSON arrays (datazip.io)
Flattening nested JSON or MongoDB’s BSON or normalizing semi-structured data and writing queries on it for analytics or regular queries, is a common challenge in data processing.

Data Processing, JSON, Data Structures, Data Analytics

14 points by pkhodiyar 73 days ago | 2 comments

pg_flo – Stream, transform, and re-route PostgreSQL data in real-time (pgflo.io)
The easiest way to move and transform data between PostgreSQL databases

PostgreSQL, Data Processing, Real-time Data, Databases, Software

239 points by shayonj 76 days ago | 54 comments

Drasi: Microsoft's open source data processing platform for event-driven systems (github.com/drasi-project)
Drasi is a data processing platform that simplifies detecting changes in data and taking immediate action.

Data Processing, Open Source, Microsoft, Event-Driven Systems, Data Streaming

331 points by benocodes 90 days ago | 67 comments

No such thing as exactly-once delivery (sequinstream.com)
We say Sequin is system with "at-least-once delivery" and "exactly-once processing" guarantees.

Distributed Systems, Data Processing, Guarantees, Software Architecture

99 points by todsacerdoti 110 days ago | 114 comments

Parsing Gigabytes of JSON per Second (arxiv.org)
JavaScript Object Notation or JSON is a ubiquitous data exchange format on the Web. Ingesting JSON documents can become a performance bottleneck due to the sheer volume of data. We are thus motivated to make JSON parsing as fast as possible.

Performance Optimization, Data Processing, JSON, Web Development, Algorithms

5 points by ibobev 129 days ago | 1 comments

Sail – Unify stream processing, batch processing and compute-intensive workloads (github.com/lakehq)
LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

Data Processing, Stream Processing, Batch Processing, Big Data, Artificial Intelligence

79 points by chenxi9649 130 days ago | 15 comments

I use Nim instead of Python for data processing (2021) (benjamindlee.com)
Lazy programmers often prefer to substitute computing effort for programming effort. I am just such a programmer. For my research, I often need to design and run algorithms over large datasets ranging into the scale of terabytes. As a fellow at the NIH, I have access to Biowulf, a 100,000+ processor cluster, so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big [MapReduce](https://en.wikipedia.org/wiki/MapReduce).

Programming Languages, Data Processing, Research, Performance Optimization

95 points by archargelod 134 days ago | 80 comments

Reflection-based JSON in C++ at Gigabytes per Second (lemire.me)

C++, JSON, Performance, Optimization, Data Processing

14 points by jandeboevrie 155 days ago | 1 comments

Pipes: A spiritual successor to Yahoo Pipes (pipes.digital)

Web Development, Social Media, Data Processing, RSS

466 points by sea-gold 201 days ago | 130 comments

Show HN: Qq: like jq, but can transcode between many formats (github.com/JFryy)

Data Processing, Software, Open Source

115 points by arandomhuman 207 days ago | 27 comments

TSV – Alternative to CSV (wikipedia.org)