Hacker News with Generative AI: Data Science

PyViz – Overview of the Python visualization landscape (pyviz.org)
The Python visualization landscape can seem daunting at first. These overviews attempt to shine light on common patterns and use cases, comparing or discussing multiple plotting libraries. Note that some of the projects discussed in the overviews are no longer maintained, so be sure to check the list of dormant projects before choosing that library.
Python, Data Visualization, Libraries, Tools, Data Science
26 points by leonry 21 hours ago | 3 comments
Directory of job boards by category (remote only, data science, etc.) (jobsearchdb.com)
Find an active job board in your career niche
Job Searching, Remote Work, Data Science
22 points by dougjudytobin 2 days ago | 2 comments
Don't use cosine similarity carelessly (migdal.pl)
Midas turned everything he touched into gold. Data scientists turn everything into vectors. We do it for a reason — as gold is the language of merchants, vectors are the language of AI1.
Cosine Similarity, Data Science, AI, Machine Learning
426 points by stared 3 days ago | 95 comments
Dbt Labs acquires SDF Labs (getdbt.com)
The TL;DR: today, I have the pleasure of announcing that dbt Labs has acquired SDF Labs. The two teams are already working side-by-side to bring SDF’s SQL comprehension technology into the hands of dbt users everywhere. SDF will be a massive upgrade to the very heart of the dbt user experience moving forward.
Software, Data Science, Acquisitions, Data Pipelines
107 points by karakanb 4 days ago | 48 comments
Show HN: SemHash – Fast Semantic Text Deduplication for Cleaner Datasets (github.com/MinishLab)
Data Science, Machine Learning, Software, GitHub
19 points by Pringled 5 days ago | 6 comments
How to become a Data Scientist? My journey, overview of skill set, practice tips (mljar.com)
In recent years, many people have been drawn to the field of data science, often believing it to be a fast track to wealth.
Data Science, Career Advice, Machine Learning
12 points by pplonski86 9 days ago | 4 comments
Small Data [video] (youtube.com)
Data, Videos, Data Science
85 points by spacechild1 11 days ago | 11 comments
Time-Series Anomaly Detection: A Decade Review (arxiv.org)
Recent advances in data collection technology, accompanied by the ever-rising volume and velocity of streaming data, underscore the vital need for time series analytics.
Time-Series Analysis, Data Science, Machine Learning, Anomaly Detection, Research
446 points by belter 12 days ago | 80 comments
JPL Horizons on-line solar system data and ephemeris computation service (nasa.gov)
The JPL Horizons on-line solar system data and ephemeris computation service provides access to key solar system data and flexible production of highly accurate ephemerides for solar system objects ( loading… ). ( unable to load body counts… please try later ).
Space Exploration, Astronomy, Data Science, Solar System
125 points by perihelions 19 days ago | 25 comments
Name Collision of the Year: Vector (crunchydata.com)
I can’t get through a zoom call, a conference talk, or an afternoon scroll through LinkedIn without hearing about vectors. Do you feel like the term vector is everywhere this year? It is.
Data Science, Technology, Trends, AI, Machine Learning
12 points by winslett 22 days ago | 0 comments
Optimality of Frequency Moment Estimation (weizmann.ac.il)
Data Science, Algorithms, Statistics
8 points by cryptohell 23 days ago | 1 comments
Model Evaluation with RandomForest and AdaBoost (deepnote.com)
This notebook demonstrates how to train and evaluate machine learning models, specifically focusing on the Random Forest Classifier and AdaBoost Classifier. We will use synthetic data and visualize key metrics, including accuracy, precision, recall, and F1-score.
Machine Learning, Data Science, Classification, Random Forest, AdaBoost
18 points by QuantumCoder111 32 days ago | 2 comments
Maximum likelihood estimation and loss functions (rish-01.github.io)
When I started learning about loss functions, I could always understand the intuition behind them. For example, the mean squared error (MSE) for regression seemed logical—penalizing large deviations from the ground-truth makes sense. But one thing always bothered me: I could never come up with those loss functions on my own. Where did they come from? Why do we use these specific formulas and not something else?
Machine Learning, Statistics, Mathematics, Data Science
119 points by snprajwal 33 days ago | 31 comments
Ask HN: Why is Ilya saying data is limited when the whole world is data? (ycombinator.com)
In his recent talk Ilya S. said that the data running out is a fundamental constraint on the scaling laws.
Generative AI, Data Science
12 points by georgestrakhov 34 days ago | 16 comments
Array Languages for Clojurians (2020) (appliedscience.studio)
A discussion in the Clojure data science Zulip led me to Slobodan Blazeski’s enlightening article Array languages for Lisp programmers in the journal of the British APL Association. He quotes Alan Perlis:
Clojure, Data Science, Programming Languages, Functional Programming
106 points by simonpure 35 days ago | 13 comments
Datasaurus dozen – Different datasets with the same descriptive statistics (wikipedia.org)
The Datasaurus dozen comprises thirteen data sets that have nearly identical simple descriptive statistics to two decimal places, yet have very different distributions and appear very different when graphed.[1] It was inspired by the smaller Anscombe's quartet that was created in
Data Visualization, Statistics, Datasets, Data Science
14 points by yread 35 days ago | 1 comments
Soylent Green Is People (xeiaso.net)
Recently a group of data scientists at Hugging Face created a dataset of curated Bluesky posts. The publication of this data has made a lot of people very angry and has been widely regarded as a bad move. The dataset contained one million posts from the Bluesky firehose with the intent that this could be a standard dataset to evaluate the effectiveness of various moderation tooling.
Social Media, Data Science, Privacy, AI
5 points by hasheddan 36 days ago | 0 comments
1,600 days of a failed hobby data science project (lellep.xyz)
⭐ I spent >1,600 days working on a data science project that then failed because I lost interest. This article is to cope with the failure and maybe help you (and me) to finish successful data science projects by summarising a few learnings into a checklist, see below.
Data Science, Personal Projects, Failure, Productivity, Learning
158 points by millimacro 40 days ago | 70 comments
It's not enough to be right as an engineer (pedramnavid.com)
Early in my career, I was rewarded for being right. As a data scientist, I worked hard to understand some business processes, make predictions, model some outcomes, and help forecast some future states. As a data engineer, my ability to properly forecast future usage of a system and build systems that are resilient and scalable to that future state while being easy to maintain helped me grow in my career.
Engineering, Data Science, Career Advice
18 points by randyzwitch 44 days ago | 1 comments
PyMyFlySpy: Track your flight using its headrest data (robertheaton.com)
“Where are we daddy?” asked my five-year-old.
Software, Privacy, Travel, Data Science
9 points by khalby786 45 days ago | 1 comments
Ask HN: Freelancer? Seeking freelancer? (December 2024) (ycombinator.com)
I'm a data scientist and I've been absolutely slaughtered by this recession. I lost a big project in July and haven't picked anything up since. Everything is "on hold until new year". So I'm looking for a project or it's coal for Christmas.
Freelancing, Data Science, Economy, Job Market
26 points by whoishiring 46 days ago | 159 comments
From PDFs to AI-ready structured data: a deep dive (explosion.ai)
PDFs are ubiquitous in industry and daily life. Paper is scanned, documents are sent and received as PDF, and they’re often kept as the archival copy. Unfortunately, processing PDFs is hard. In this blog post, I’ll present a new modular workflow for converting PDFs and similar documents to structured data and show how to build end-to-end document understanding and information extraction pipelines for industry use cases.
Data Science, AI, Machine Learning, Document Processing
23 points by will-burner 47 days ago | 0 comments
Vesuvius Challenge: First letters found in new scroll (scrollprize.substack.com)
Today we are thrilled to share a new scroll dataset that is already full of exciting findings. In collaboration with our partners at the Bodleian Library, the University of Oxford, Diamond Light Source, and EduceLab, we present: PHerc. 172!
History, Archaeology, Research, Data Science, Technology
278 points by leocassarani 51 days ago | 27 comments
A Non-Technical Guide to Interpreting SHAP Analyses (aidancooper.co.uk)
With interpretability becoming an increasingly important requirement for machine learning projects, there's a growing need to communicate the complex outputs of model interpretation techniques to non-technical stakeholders.
Machine Learning, Interpretability, Data Science, Non-technical
14 points by reqo 54 days ago | 0 comments
Heuristics that almost always work (deepnote.com)
RunCopyInclude input parameters in the link.
Heuristics, Software Development, Programming, Data Science
15 points by ByteExplorer 57 days ago | 4 comments
Two Parameter Model for Running Performance (normalizingconstant.com)
Human running performance from real-world big data is a super cool paper from 2020 that analyzes 14,000 people’s running activities.
Sports, Data Science, Machine Learning, Performance Analysis
7 points by brendanashworth 57 days ago | 0 comments
Why can't we separate YAML from ML? (ethanrosenthal.com)
I’m coming up on 10 years, and half as many jobs, in data science and machine learning. No matter what, in every role, I find myself reinventing a programming language on top of YAML in order to train machine learning models.
Machine Learning, YAML, Data Science, Programming Languages
8 points by alexmolas 58 days ago | 1 comments
Non-elementary group-by aggregations in Polars vs pandas (quansight.org)
I attended PyData Berlin 2024 in April, and it was a blast! I met so many colleagues, collaborators, and friends. There was quite some talk of Polars - some people even gathered together for a Polars-themed dinner! It's certainly nice to see people talking about it, and the focus tends to be on features such as:
Data Analysis, Pandas, Polars, Python, Data Science
135 points by rbanffy 63 days ago | 74 comments
Image-Text Curation for 1B+ Data: Faster, Better, Smaller Clip Models (datologyai.com)
Computer Vision, Data Science, Generative AI, Language Models
12 points by hurrycane 64 days ago | 0 comments
The Corpus of United States State Statutes–Design, Construction and Use (ssrn.com)
There is a need for more publicly available corpora of legal language.
Legal Research, Data Science, Legal Language, Public Data
19 points by cwmoore 64 days ago | 4 comments