Hacker News with Generative AI: Data Science

Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser (hyperparam.app)
Hyperparam was founded to address a critical gap in the machine learning ecosystem: the lack of a user-friendly, scalable UI for exploring and curating massive datasets.
Machine Learning, Data Science, Open Source, Web Development, Tools
73 points by platypii 2 days ago | 21 comments
Dataframely: A polars-native data frame validation library (quantco.com)
At QuantCo, we are constantly trying to improve the quality of our code bases to ensure that they remain easily maintainable. More recently, this often involved migrating data pipelines from pandas to polars in order to achieve significant performance gains.
Data Science, Libraries, Performance Optimization
39 points by sito42 3 days ago | 8 comments
Relational Graph Transformers (kumo.ai)
In the world of enterprise data, the most valuable insights often lie not in individual tables, but in the complex relationships between them. Customer interactions, product hierarchies, transaction histories—these interconnected data points tell rich stories that traditional machine learning approaches struggle to fully capture. Enter Relational Graph Transformers: a breakthrough architecture that's transforming how we extract intelligence from relational databases.
Machine Learning, Data Science, Enterprise Data, Relational Databases
66 points by gk1 5 days ago | 6 comments
Stuffed-Na(a)N: stuff your NaNs (github.com/si14)
Have you ever done this by mistake?
Programming, Software, Software Development, Data Science, Python
154 points by dgroshev 7 days ago | 58 comments
Are polynomial features the root of all evil? (2024) (alexshtf.github.io)
It turns out that it’s just a MYTH. There’s nothing inherently wrong with high degree polynomials, and in contrast to what is typically taught, high degree polynomials are easily controlled using standard ML tools, like regularization. The source of the myth stems mainly from two misconceptions about polynomials that we will explore here. In fact, not only they are great non-linear features, certain representations also provide us with powerful control over the shape of the function we wish to learn.
Machine Learning, Data Science, Myths, Mathematics, Polynomial Regression
188 points by Areibman 11 days ago | 77 comments
Introduction to Graph Transformers (kumo.ai)
Graphs are everywhere. From modeling molecular interactions and social networks to detecting financial fraud, learning from graph data is powerful—but inherently challenging.
Machine Learning, Data Science
44 points by gk1 11 days ago | 0 comments
The value of a dedicated data science approach in HR (gorelik.net)
This document outlines why HR departments in large organizations benefit from a dedicated data science approach, highlighting impacts beyond recruitment. In short, my thesis is as follows: as organizations scale, so does the complexity of understanding their internal dynamics. Data tools become essential to analyzing large organizations, as they enable HR to identify patterns and insights that can drive strategic improvements across key areas.
Data Science, Human Resources, Business Intelligence, Organizational Dynamics, Strategy
13 points by luu 20 days ago | 3 comments
Show HN: LLM Based Spark Profiler (datasre.ai)
Spark, Data Science, Profiling, Show HN
27 points by ambrood 24 days ago | 5 comments
Functional connectomics spanning multiple areas of mouse visual cortex (nature.com)
Understanding the brain requires understanding neurons’ functional responses to the circuit architecture shaping them. Here we introduce the MICrONS functional connectomics dataset with dense calcium imaging of around 75,000 neurons in primary visual cortex (VISp) and higher visual areas (VISrl, VISal and VISlm) in an awake mouse that is viewing natural and synthetic stimuli.
Neuroscience, Brain Research, Visual Cortex, Mice, Data Science
6 points by rntn 24 days ago | 0 comments
FUTO open-sources 1M row keyboard swipe dataset (huggingface.co)
Data Science, Machine Learning, Datasets, Open Source, Keyboard Input
12 points by ReadEvalPost 29 days ago | 1 comments
Nvidia's latest AI PC boxes sound great – for data scientists with $3k to spare (theregister.com)
Nvidia's latest AI PC boxes sound great – if you're a data scientist with $3,000 to spare
Nvidia, Artificial Intelligence, Hardware, Data Science, Finance
44 points by rntn 33 days ago | 55 comments
UK govt data people not 'technical,' says ex-Downing St data science head (theregister.com)
A former director of data science at the UK prime minister's office has told MPs that people working with data in government are not typically technical and would be unlikely to get a similar job in the private sector.
Government, Data Science, UK, Technology
3 points by rntn 36 days ago | 1 comments
Compress Better, Compute Bigger (ironarray.io)
Have you ever experienced the frustration of not being able to analyze a dataset because it's too large to fit in memory? Or perhaps you've encountered the memory wall, where computation is hindered by slow memory access? These are common challenges in data science and high-performance computing. The developers of Blosc and Blosc2 have consistently focused on achieving compression and decompression speeds that approach or even exceed memory bandwidth limits.
Data Compression, Data Science
20 points by todsacerdoti 36 days ago | 0 comments
Study of Lyft rideshare data confirms minorities get more tickets (arstechnica.com)
It's no secret that "driving while black" is a real phenomenon. Study after study has shown that minority drivers are ticketed at a higher rate, and data from speed cameras suggests that it's not because they commit traffic violations more frequently. But this leaves open the question of why. Bias is an obvious answer, but it's hard to eliminate an alternative explanation: Minority groups may engage in more unsafe driving, and the police are trying to deter that.
Racial Bias, Transportation, Data Science, Law Enforcement
9 points by AlexErrant 36 days ago | 0 comments
Show HN: Xorq – open-source Python-first Pandas-style pipelines (github.com/xorq-labs)
xorq is a deferred computational framework that brings the replicability and performance of declarative pipelines to the Python ML ecosystem.
Open Source, Python, Data Science, Machine Learning, Pipelines
15 points by secretasiandan 37 days ago | 7 comments
(yet another) two kinds of data scientists (artofdatascience.substack.com)
We’re hiring data scientists. I mean, we hired one last week, but we need more data scientists here at Babbage.
Data Science, Hiring, Jobs
3 points by amrrs 41 days ago | 0 comments
Stop using the elbow criterion for k-means (arxiv.org)
A major challenge when using k-means clustering often is how to choose the parameter k, the number of clusters.
Machine Learning, K-Means Clustering, Data Science
79 points by Anon84 41 days ago | 28 comments
Notebooks as reusable Python programs (marimo.io)
AI developers, data engineers, data scientists, and researchers do a lot of their work in Python notebooks like Jupyter.
Python, Data Science, Programming, Software Development
117 points by akshayka 45 days ago | 60 comments
The clustering behavior of sliding windows (arxiv.org)
Things can go spectacularly wrong when clustering timeseries data that has been preprocessed with a sliding window.
Data Science, Machine Learning
97 points by mathfan 45 days ago | 20 comments
Why extracting data from PDFs is still a nightmare for data experts (arstechnica.com)
For years, businesses, governments, and researchers have struggled with a persistent problem: How to extract usable data from Portable Document Format (PDF) files.
Data Science, Data Extraction, PDF, Software, Business
14 points by ilamont 46 days ago | 1 comments
Predictive Data Selection: The Data That Predicts Is the Data That Teaches (arxiv.org)
Language model pretraining involves training on extensive corpora, where data quality plays a pivotal role.
Machine Learning, Data Science, Generative AI
5 points by PaulHoule 49 days ago | 0 comments
Changing Forecasts for Python Questions on Stack Overflow (win-vector.com)
I recently conducted a small time series workshop session for AI+ training hosted by ODSC. It went really well, and I’d be happy to offer longer interactive workshops going forward (please reach out if your team would like one!).
Programming, Data Science, Python, Stack Overflow, Time Series
25 points by jmount 49 days ago | 5 comments
Why extracting data from PDFs is still a nightmare for data experts (arstechnica.com)
For years, businesses, governments, and researchers have struggled with a persistent problem: How to extract usable data from Portable Document Format (PDF) files.
Data Extraction, PDFs, Data Science, Technology, Business
9 points by rntn 53 days ago | 0 comments
Building Deep Research Agent from Scratch (swirlai.com)
👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in GenAI, MLOps, Data Engineering, Machine Learning and overall Data space.
Artificial Intelligence, Machine Learning, Data Science, Research, Newsletter
10 points by AurimasGr 53 days ago | 3 comments
Nexar Dashcam Crash Prediction Challenge (kaggle.com)
Computer Vision, Machine Learning, Data Science, Competitions
10 points by dmoura 57 days ago | 4 comments
Extracting time series features: a powerful method from a obscure paper [pdf] (rcin.org.pl)
Data Science, Machine Learning, Research Papers
49 points by tmshapland 57 days ago | 5 comments
McKinsey's Analysis of the Gender Pay Gap: Data Exploration Gone Wrong (substack.com)
Analysis, Data Science, McKinsey, Business
11 points by sarahweaver 60 days ago | 1 comments
The power of interning: making a time series database smaller (gendignoux.com)
This week-end project started by browsing the open-data repository of Paris’ public transport network, which contains various APIs to query real-time departures, current disruptions, etc.
Data Science, Databases, Time Series, Open Data, Public Transportation
273 points by todsacerdoti 61 days ago | 66 comments
Data Incest: When AI Breeds with Itself (defragzone.substack.com)
Today, we’re diving into a problem that’s as bizarre as it is concerning: data incest. No, this isn’t some dystopian sci-fi horror plot. It’s a real issue creeping into AI and machine learning. And trust me, it’s as messy as it sounds.
Artificial Intelligence, Machine Learning, Data Science
3 points by frag 61 days ago | 0 comments
Show HN: NumPy+Jax Except with Named Axes (github.com/justindomke)
NumPy+Jax with named axes and an uncompromising attitude
Python, Machine Learning, Software, Libraries, Data Science
15 points by justindomke 67 days ago | 1 comments