Hacker News with Generative AI: Data Science

Non-elementary group-by aggregations in Polars vs pandas (quansight.org)
I attended PyData Berlin 2024 in April, and it was a blast! I met so many colleagues, collaborators, and friends. There was quite some talk of Polars - some people even gathered together for a Polars-themed dinner! It's certainly nice to see people talking about it, and the focus tends to be on features such as:
Data Analysis, Pandas, Polars, Python, Data Science
133 points by rbanffy 5 days ago | 74 comments
Image-Text Curation for 1B+ Data: Faster, Better, Smaller Clip Models (datologyai.com)
Computer Vision, Data Science, Generative AI, Language Models
12 points by hurrycane 7 days ago | 0 comments
The Corpus of United States State Statutes–Design, Construction and Use (ssrn.com)
There is a need for more publicly available corpora of legal language.
Legal Research, Data Science, Legal Language, Public Data
19 points by cwmoore 7 days ago | 4 comments
The geometry of data: the missing metric tensor and the Stein score [Part II] (christianperone.com)
I’m writing this second part of the series because I couldn’t find any formalisation of this metric tensor that naturally arises from the Stein score (especially when used with learned models), and much less blog posts or articles about it, which is surprising given its deep connection between score-based generative models, diffusion models and the geometry of the data manifold.
Machine Learning, Data Science, Generative Models, Geometry
63 points by perone 7 days ago | 7 comments
New textbook teaches students about matrix methods and their real world apps (engin.umich.edu)
A new textbook, Linear Algebra for Data Science, Machine Learning, and Signal Processing, is being unveiled for use in classes that explore the many applications of matrix methods to real world data.
Linear Algebra, Data Science, Machine Learning, Signal Processing, Education
77 points by teleforce 9 days ago | 33 comments
45% of U.S. Data Scientists Work at Just 20 Companies (index42.com)
For most companies, data scientists are indispensable. These specialized professionals are at the core of artificial intelligence (AI) and data-driven projects, helping companies uncover insights, optimize operations, and unlock new business opportunities.
Data Science, Employment, Artificial Intelligence, Business
44 points by Gaussian 10 days ago | 9 comments
Timing-Sensitive Analysis in Python (deepnote.com)
Time consistency is critical in many fields, especially in sensitive applications like cryptography.
Python, Data Science, Cryptography
9 points by nature556 10 days ago | 2 comments
Segmenting Credit Card Customers with K-Means (medium.com)
Ever wondered how credit card companies categorize their clients? Here’s a behind-the-scenes look at how they do it using data science! In this project, we’ll explore how to segment credit card customers with K-Means clustering, one of the most popular machine learning techniques. Let’s dive into the key steps and make sense of the data!
Data Science, Machine Learning, Customer Segmentation, K-Means Clustering
56 points by thunderbong 15 days ago | 29 comments
Rd-TableBench – Accurately evaluating table extraction (reducto.ai)
RD-TableBench is an open benchmark to help teams evaluate extraction performance for complex tables.
Data Science, Open Source, Benchmarking
29 points by raunakchowdhuri 16 days ago | 6 comments
DataChain: DBT for Unstructured Data (github.com/iterative)
DataChain is a modern Pythonic data-frame library designed for artificial intelligence.
Python, Data Science, Artificial Intelligence, Data Warehousing
163 points by shcheklein 17 days ago | 26 comments
131M American Buildings (marksblogg.com)
In May, Nature published an article detailing Oak Ridge National Laboratory's (ORNL) new, AI-generated US Building Dataset.
Artificial Intelligence, Data Science, Research, Buildings, United States
169 points by marklit 19 days ago | 37 comments
Python has overtaken JavaScript on GitHub (infoworld.com)
Python has overtaken JavaScript as the most popular language on GitHub, while the use of Jupyter Notebooks also has skyrocketed on the site. The rise of both underscore the surge in data science, artificial intelligence, and machine learning on the code-sharing platform, according to GitHub’s just-released Octoverse 2024 report.
Programming Languages, GitHub, Data Science, Artificial Intelligence, Machine Learning
6 points by e2e4 19 days ago | 1 comments
The overlooked GenAI use case: cleaning, processing, and analyzing data (sumble.com)
Job post data reveals what companies plan to do with GenAI. The biggest use case is cleaning, processing, and analyzing data.
Generative AI, Data Science, Business
31 points by antgoldbloom 20 days ago | 5 comments
Data Commons (datacommons.org)
Data, Data Science, Tools, Research
68 points by zerojames 21 days ago | 4 comments
Probability-generating functions (entropicthoughts.com)
I have long struggled with understanding what probability-generating functions are and how to intuit them. There were two pieces of the puzzle missing for me, and we’ll go through both in this article.
Probability, Math, Statistics, Data Science
207 points by todsacerdoti 21 days ago | 48 comments
Mistakes from building a model to scalp concert tickets (datastream.substack.com)
In November 2022, Taylor Swift fans crashed Ticketmaster. Not just the website - they crashed the entire company's facade of competence. Over 14 million people tried to buy tickets meant for 1.5 million fans. The servers melted down, the presale collapsed, and Ticketmaster had to cancel the general public sale entirely. Swift herself said watching her fans struggle to get tickets was "excruciating."
Concert Tickets, Scalping, Data Science, Ticketmaster, Software
52 points by racketracer 22 days ago | 66 comments
Using Survival Analysis to estimate product lifetime (jumpdata.co.uk)
We needed to estimate the average number of years after purchase various consumer products broke down from a large dataset containing:
Data Science, Statistics, Product Management, Survival Analysis
18 points by sebg 24 days ago | 3 comments
ModelKit: Transforming AI/ML artifact sharing and management across lifecycles (kitops.ml)
ModelKit revolutionizes the way AI/ML artifacts are shared and managed throughout the lifecycle of AI/ML projects.
AI/ML, Software, Machine Learning, Data Science, Tools
12 points by dsaed 25 days ago | 1 comments
Goodhart’s law isn’t as useful as you might think (2023) (commoncog.com)
Goodhart’s Law is a famous adage that goes “when a measure becomes a target, it ceases to be a good measure.”
Goodhart's Law, Data Science, Metrics, Optimization, Business
140 points by yagizdegirmenci 26 days ago | 81 comments
KitOps: Only Standards-Based Packaging and Versioning Tool for AI/ML Projects (kitops.ml)
KitOps is an innovative open-source project designed to enhance collaboration among data scientists, application developers, and SREs working on integrating or managing self-hosted AI/ML models.
AI/ML, Open Source, Software, Data Science, Version Control
17 points by dsaed 27 days ago | 2 comments
Data viz project that maps all earthquakes by magnitude (concord.org)
Data Visualization, Earthquakes, Mapping, Data Science
86 points by therabbithole 28 days ago | 18 comments
Fast and scalable dataset preparation and curation tool from Nvidia (github.com/NVIDIA)
🚀 The GPU-Accelerated Open Source Framework for Efficient Large Language Model Data Curation 🚀
Open Source, Data Science, Nvidia, Machine Learning
12 points by shcheklein 29 days ago | 0 comments
BMI-type measure for a place's "goodness of weather" (ycombinator.com)
Weather, Climate, Data Science, Measurement
7 points by profsummergig 30 days ago | 17 comments
My NumPy year: Creating a DType for the next generation of scientific computing (quansight.com)
From no CPython C API experience to shipping a new DType in NumPy 2.0.
NumPy, Scientific Computing, Programming, Data Science
85 points by sebg 30 days ago | 9 comments
Is It Pokémon or Big Data? (pixelastic.github.io)
Made by @pixelastic, inspired by this google form. Source code available on GitHub.
Data Visualization, Games, Programming, Data Science
6 points by bruh2 30 days ago | 2 comments
Free Python Course – A friendly, modern hands-on introduction to Python (fabridamicelli.github.io)
This is an open, free short introduction to the Python programming language, emphasizing practical over theoretical aspects and with a focus on data-related tasks.
Python, Programming, Education, Data Science
9 points by fbrdm 30 days ago | 0 comments
Understanding Gaussians (gestalt.ink)
The Gaussian distribution, or normal distribution is a key subject in statistics, machine learning, physics, and pretty much any other field that deals with data and probability. It’s one of those subjects, like $\pi$ or Bayes’ rule, that is so fundamental that people treat it like an icon.
Statistics, Machine Learning, Data Science, Probability
152 points by lapnect 30 days ago | 34 comments
Why Rust is an increasingly beloved part of my programming toolbox (idyll.org)
Hi, I'm Titus. I'm a prof at UC Davis who writes scientific software for biological data analysis of very, very large data sets. And I really like Rust, especially in combination with Python. This blog post is about why.
Programming, Rust, Data Science, Python
7 points by selimthegrim 31 days ago | 0 comments
Machine Learning to Computational Plasma Physics Reduced-Order Plasma Modeling (arxiv.org)
Machine learning (ML) provides a broad spectrum of tools and architectures that enable the transformation of data from simulations and experiments into useful and explainable science, thereby augmenting domain knowledge.
Machine Learning, Plasma Physics, Computational Physics, Data Science, Scientific Computing
20 points by sandwichsphinx 31 days ago | 1 comments
Waking up science's sleeping beauties (2023) (worksinprogress.co)
Many scientific papers receive little attention initially but become highly cited years later. What groundbreaking discoveries might have already been made, and how can we uncover them faster?
Science, Research, Citation Analysis, Data Science
70 points by bookofjoe 31 days ago | 30 comments