Hacker News with Generative AI: Big Data

The CDC, Palantir and the AI-Healthcare Revolution (unlimitedhangout.com)
The CDC’s Center for Forecasting and Outbreak Analytics (CFA) has partnered with the CIA-linked Palantir to cement the public-private model of invasive surveillance in “public health,” all while pushing the U.S. national security state and Silicon Valley even closer together.

Public Health, Surveillance, Artificial Intelligence, Government, Big Data

7 points by TheFreim 4 days ago | 0 comments

Parquet and ORC's many shortfalls for machine learning, and what to do about it? (starburst.io)
At the turn of the century (around a quarter of a decade ago), over 99% of the data management industry used row-oriented storage to store data for all workloads involving structured data — including transactional and analytical workloads.

Machine Learning, Data Storage, Data Management, Big Data

6 points by abadid 10 days ago | 1 comments

Apache DataFusion: Fast, Embeddable, Modular Analytic Query Engine [pdf] (nerdnetworks.org)

Apache DataFusion, Big Data, Query Engines, Data Processing, Open Source

12 points by hambandit 11 days ago | 0 comments

Apache Hudi 1.0 released with secondary indexes for data lakehouses (apache.org)
We are thrilled to announce the release of Apache Hudi 1.0, a landmark achievement for our vibrant community that defines what the next generation of data lakehouses should achieve.

Data Lakehouses, Apache Hudi, New Releases, Software, Big Data

11 points by v5c6 32 days ago | 9 comments

How big data created the modern dairy cow (worksinprogress.co)
What do cryogenics, butterfat tests, and genetic data have in common? They’re some of the reasons behind the world’s most productive dairy cows. Here’s how it all started.

Agriculture, Big Data, Genetics, Dairy

48 points by surprisetalk 36 days ago | 58 comments

New Amazon S3 Tables: Storage optimized for analytics workloads (amazon.com)
Amazon S3 Tables give you storage that is optimized for tabular data such as daily purchase transactions, streaming sensor data, and ad impressions in Apache Iceberg format, for easy queries using popular query engines like Amazon Athena, Amazon EMR, and Apache Spark.

Cloud Storage, Data Analytics, Big Data

19 points by craigkerstiens 46 days ago | 3 comments

Sail 0.2: Spark replacement in Rust, runs 4x faster, drop-in PySpark compatible (lakesail.com)
LakeSail is thrilled to unveil a preview release of Sail 0.2, our latest milestone in the journey to redefine distributed data processing.

Rust, Distributed Computing, Data Processing, Big Data, Open Source

14 points by chenxi9649 57 days ago | 2 comments

Differential Dataflow for the Masses (github.com/brurucy)
This library provides an implementation of the DBSP language for incremental streaming computations.

Data Processing, Programming, Software, Big Data, Streaming

42 points by rebanevapustus 70 days ago | 11 comments

Write your first MapReduce program in 20 minutes (deepnote.com)
This notebook focuses on MapReduce, which processes large datasets. It is designed to find the maximum transaction value by store using sales data. For those who want to learn MapReduce from scratch, this notebook covers the basics. For more information, here is a detailed article.

Programming, MapReduce, Data Analysis, Big Data

11 points by andrewg4445 94 days ago | 6 comments

Ask HN: How big of a problem is unstructured data for companies? (ycombinator.com)
I read somewhere that 90% of companies have data like documents, PDFs, videos, images, audio clips, and other content that are unstructured that will be a big obstacle for ai.

Data Management, Artificial Intelligence, Business, Big Data, Unstructured Data

16 points by dark7 96 days ago | 16 comments

COBOL's Map Reduce (2022) (ztoz.blog)
COBOL is for Big Data. Well, sort of. Awhile back, I noticed that the COBOL SORT verb was overpowered. Rather than sorting an array of items or even sorting a file, it included a generalized ability to stream in arbitrary inputs — () => Stream[T] —, where T is a key/value pair, and process the outputs in order — SortedStream[T] => (). This power is useful if you are writing a map-reduce program, but excessive for sorting.

Programming, COBOL, Big Data, Map Reduce

5 points by thunderbong 102 days ago | 0 comments

6 Powerful Databricks Alternatives for Data Lakes and Lakehouses (definite.app)
Databricks has established itself as a leader in the data lake and lakehouse space, offering a powerful platform for big data processing and analytics.

Data Lakes, Data Warehousing, Big Data, Cloud Computing, Analytics

13 points by TenMatrix 129 days ago | 2 comments

Sail – Unify stream processing, batch processing and compute-intensive workloads (github.com/lakehq)
LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

Data Processing, Stream Processing, Batch Processing, Big Data, Artificial Intelligence

79 points by chenxi9649 130 days ago | 15 comments

Apache Zeppelin (apache.org)

Apache, Software, Data Visualization, Big Data

144 points by saikatsg 137 days ago | 31 comments

The Imperial Origins of Big Data (yalebooks.yale.edu)

History, Big Data, Data Science

20 points by drdee 141 days ago | 7 comments

Amazon's exabyte-scale migration from Apache Spark to Ray on EC2 (amazon.com)