Hacker News with Generative AI: Data Science

Nvidia's latest AI PC boxes sound great – for data scientists with $3k to spare (theregister.com)
Nvidia's latest AI PC boxes sound great – if you're a data scientist with $3,000 to spare
Nvidia, Artificial Intelligence, Hardware, Data Science, Finance
44 points by rntn 3 days ago | 55 comments
UK govt data people not 'technical,' says ex-Downing St data science head (theregister.com)
A former director of data science at the UK prime minister's office has told MPs that people working with data in government are not typically technical and would be unlikely to get a similar job in the private sector.
Government, Data Science, UK, Technology
3 points by rntn 5 days ago | 1 comments
Compress Better, Compute Bigger (ironarray.io)
Have you ever experienced the frustration of not being able to analyze a dataset because it's too large to fit in memory? Or perhaps you've encountered the memory wall, where computation is hindered by slow memory access? These are common challenges in data science and high-performance computing. The developers of Blosc and Blosc2 have consistently focused on achieving compression and decompression speeds that approach or even exceed memory bandwidth limits.
Data Compression, Data Science
20 points by todsacerdoti 5 days ago | 0 comments
Study of Lyft rideshare data confirms minorities get more tickets (arstechnica.com)
It's no secret that "driving while black" is a real phenomenon. Study after study has shown that minority drivers are ticketed at a higher rate, and data from speed cameras suggests that it's not because they commit traffic violations more frequently. But this leaves open the question of why. Bias is an obvious answer, but it's hard to eliminate an alternative explanation: Minority groups may engage in more unsafe driving, and the police are trying to deter that.
Racial Bias, Transportation, Data Science, Law Enforcement
9 points by AlexErrant 6 days ago | 0 comments
Show HN: Xorq – open-source Python-first Pandas-style pipelines (github.com/xorq-labs)
xorq is a deferred computational framework that brings the replicability and performance of declarative pipelines to the Python ML ecosystem.
Open Source, Python, Data Science, Machine Learning, Pipelines
15 points by secretasiandan 6 days ago | 7 comments
(yet another) two kinds of data scientists (artofdatascience.substack.com)
We’re hiring data scientists. I mean, we hired one last week, but we need more data scientists here at Babbage.
Data Science, Hiring, Jobs
3 points by amrrs 10 days ago | 0 comments
Stop using the elbow criterion for k-means (arxiv.org)
A major challenge when using k-means clustering often is how to choose the parameter k, the number of clusters.
Machine Learning, K-Means Clustering, Data Science
78 points by Anon84 11 days ago | 28 comments
Notebooks as reusable Python programs (marimo.io)
AI developers, data engineers, data scientists, and researchers do a lot of their work in Python notebooks like Jupyter.
Python, Data Science, Programming, Software Development
117 points by akshayka 14 days ago | 60 comments
The clustering behavior of sliding windows (arxiv.org)
Things can go spectacularly wrong when clustering timeseries data that has been preprocessed with a sliding window.
Data Science, Machine Learning
97 points by mathfan 14 days ago | 20 comments
Why extracting data from PDFs is still a nightmare for data experts (arstechnica.com)
For years, businesses, governments, and researchers have struggled with a persistent problem: How to extract usable data from Portable Document Format (PDF) files.
Data Science, Data Extraction, PDF, Software, Business
14 points by ilamont 15 days ago | 1 comments
Predictive Data Selection: The Data That Predicts Is the Data That Teaches (arxiv.org)
Language model pretraining involves training on extensive corpora, where data quality plays a pivotal role.
Machine Learning, Data Science, Generative AI
5 points by PaulHoule 19 days ago | 0 comments
Changing Forecasts for Python Questions on Stack Overflow (win-vector.com)
I recently conducted a small time series workshop session for AI+ training hosted by ODSC. It went really well, and I’d be happy to offer longer interactive workshops going forward (please reach out if your team would like one!).
Programming, Data Science, Python, Stack Overflow, Time Series
25 points by jmount 19 days ago | 5 comments
Why extracting data from PDFs is still a nightmare for data experts (arstechnica.com)
For years, businesses, governments, and researchers have struggled with a persistent problem: How to extract usable data from Portable Document Format (PDF) files.
Data Extraction, PDFs, Data Science, Technology, Business
9 points by rntn 23 days ago | 0 comments
Building Deep Research Agent from Scratch (swirlai.com)
👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in GenAI, MLOps, Data Engineering, Machine Learning and overall Data space.
Artificial Intelligence, Machine Learning, Data Science, Research, Newsletter
10 points by AurimasGr 23 days ago | 3 comments
Nexar Dashcam Crash Prediction Challenge (kaggle.com)
Computer Vision, Machine Learning, Data Science, Competitions
10 points by dmoura 26 days ago | 4 comments
Extracting time series features: a powerful method from a obscure paper [pdf] (rcin.org.pl)
Data Science, Machine Learning, Research Papers
49 points by tmshapland 26 days ago | 5 comments
McKinsey's Analysis of the Gender Pay Gap: Data Exploration Gone Wrong (substack.com)
Analysis, Data Science, McKinsey, Business
11 points by sarahweaver 29 days ago | 1 comments
The power of interning: making a time series database smaller (gendignoux.com)
This week-end project started by browsing the open-data repository of Paris’ public transport network, which contains various APIs to query real-time departures, current disruptions, etc.
Data Science, Databases, Time Series, Open Data, Public Transportation
273 points by todsacerdoti 30 days ago | 66 comments
Data Incest: When AI Breeds with Itself (defragzone.substack.com)
Today, we’re diving into a problem that’s as bizarre as it is concerning: data incest. No, this isn’t some dystopian sci-fi horror plot. It’s a real issue creeping into AI and machine learning. And trust me, it’s as messy as it sounds.
Artificial Intelligence, Machine Learning, Data Science
3 points by frag 30 days ago | 0 comments
Show HN: NumPy+Jax Except with Named Axes (github.com/justindomke)
NumPy+Jax with named axes and an uncompromising attitude
Python, Machine Learning, Software, Libraries, Data Science
15 points by justindomke 36 days ago | 1 comments
The best way to use text embeddings portably is with Parquet and Polars (minimaxir.com)
Text embeddings, particularly modern embeddings generated from large language models, are one of the most useful applications coming from the generative AI boom.
Text Embeddings, Generative AI, Data Science, Tools
247 points by minimaxir 37 days ago | 59 comments
You can’t build a moat with AI (redux) (frontierai.substack.com)
Last spring, we wrote an article called You can’t build a moat with AI. That post argued that prompt engineering, while important, would be difficult to defend over time given how easy it is to experiment with LLMs. As a result, you have to focus on the quality of data your application has access to and your use of that data to differentiate yourself.
Artificial Intelligence, Generative AI, Business Strategy, Data Science
134 points by cgwu 41 days ago | 102 comments
USDS Engineering Director Resigns: 'This Is Not the Mission I Came to Serve' (wired.com)
The director of data science and engineering for the United States Digital Service—which Elon Musk rebranded as the US DOGE Service—has resigned from her position.
Resignation, Elon Musk, Data Science
10 points by jay_kyburz 42 days ago | 7 comments
Weird Kaggle and the Superiority of Books (griffens.net)
I recently entered a Kaggle competition to brush up on some modeling skills.1 The analysis problem is a pretty typical clinical prediction question. My final product is, well, "final product" is an extremely charitable description of it.2 But despite this it was a fascinating and worthwhile experience, full of interesting questions to ponder, such as "what is the fundamental difference between statistics and machine learning?" and "do they realize their evaluation metric is pretty silly?"
Machine Learning, Data Science, Books, Kaggle
65 points by ngriffiths 42 days ago | 2 comments
Tristan Davey's Punch Card Archive (tristandavey.com)
Punched cards were once a ubiquitous part of accounting, data collection and early computing.
History, Computing, Data Science, Archive, Art
18 points by NaOH 43 days ago | 3 comments
Show HN: Bag of words – Build and share smart data apps using AI (github.com/bagofwords1)
Bag of words enables users to create comprehensive dashboards with a single prompt and refine them iteratively.
AI, Data Visualization, Data Science, Software, Show HN
12 points by bow14 44 days ago | 8 comments
Physics Informed Neural Networks (pages.dev)
I mentioned in a previous post that as part of my position as junior DS for IKEA, I also get the opportunity to take part in their training program (called AI accelerator program). It’s a six months long program in which we are trained on all aspects of data science and AI by both professionals but also through various courses.
Machine Learning, Artificial Intelligence, Data Science
96 points by nchagnet 45 days ago | 20 comments
Basketball has evolved into a game of calculated decision-making (nabraj.com)
Basketball has evolved from a game of unpredictability into a game of calculated decision-making with the use of data and analytics.
Basketball, Sports, Analytics, Data Science
126 points by nabaraz 46 days ago | 261 comments
Classic Data science pipelines built with LLMs (github.com/Pravko-Solutions)
Below you’ll find an overview of each file, what it showcases, and links to the actual code. You can run these examples (assuming you’ve installed FlashLearn and set your “OPENAI_API_KEY”) simply by using:
Data Science, Generative AI, Programming, GitHub
196 points by galgia 53 days ago | 86 comments
Stop using zip codes for geospatial analysis (2019) (carto.com)
Uncover deeper insights beyond ZIP codes with geospatial analysis. Explore the limitations of ZIP codes and discover alternatives for spatial understanding.
Geospatial Analysis, Data Visualization, Data Science, Cartography
184 points by voxadam 54 days ago | 129 comments