Hacker News with Generative AI: Information Retrieval

VideoRAG: Retrieval-Augmented Generation over Video Corpus (arxiv.org)
Retrieval-Augmented Generation (RAG) is a powerful strategy to address the issue of generating factually incorrect outputs in foundation models by retrieving external knowledge relevant to queries and incorporating it into their generation process.
Video, Generative AI, Machine Learning, Information Retrieval
4 points by t55 5 days ago | 0 comments
How outdated information hides in LLM token generation probabilities (anj.ai)
The internet usually has the correct answer somewhere, but it’s also full of conflicting and outdated information. How do large language models (LLMs) such as ChatGPT, trained on internet scale data, handle cases where there’s conflicting or outdated information? (Hint: it’s not always the most recent answer as of the knowledge cutoff date; think about what LLMs are trained to do)
Generative AI, Information Retrieval, Data Bias
122 points by anjsimmo 8 days ago | 53 comments
Wikipedia searches reveal differing styles of curiosity (scientificamerican.com)
Mapping explorers of Wikipedia rabbit holes revealed three different styles of human inquisitiveness: the “busybody,” the “hunter” and the “dancer”
Wikipedia, Curiosity, Research, Psychology, Information Retrieval
71 points by ripe 8 days ago | 37 comments
Embedding Models for Information Retrieval in 2025 (datastax.com)
The just-released Voyage-3-large is the surprise leader in embedding relevance
Information Retrieval, Embedding Models, Generative AI, Machine Learning
12 points by tjake 8 days ago | 1 comments
RAG a 40GB Outlook inbox – Long term Staff member leaving, keeping knowledge (reddit.com)
I've been fascinated by this concept since the early days of AI, and using ChatGPT has made it feel incredibly achievable and only just understood the concept of RAG. The idea is to leverage a local LLM paired with an open web UI to create vector or other databases of the inbox
Artificial Intelligence, Information Retrieval, Email
13 points by miles 16 days ago | 3 comments
Unifying Generative and Dense Retrieval for Sequential Recommendation (arxiv.org)
Sequential dense retrieval models utilize advanced sequence learning techniques to compute item and user representations, which are then used to rank relevant items for a user through inner product computation between the user and all item representations.
Generative AI, Recommendation Systems, Machine Learning, Information Retrieval
4 points by ashvardanian 18 days ago | 0 comments
Prism: Manipulating Concepts in Latent Space (thesephist.com)
Foundation models gesture at a way of interacting with information that’s at once more natural and powerful than “classic” knowledge tools. But to build the kind of rich, directly interactive information interfaces I imagine, current foundation models and embeddings are far too opaque to humans.
Artificial Intelligence, Machine Learning, Information Retrieval
3 points by diwank 18 days ago | 0 comments
The Anatomy of a Large-Scale Hypertextual Web Search Engine (1998) (infolab.stanford.edu)
The web creates new challenges for information retrieval. The amount of information on the web is growing rapidly, as well as the number of new users inexperienced in the art of web research.
Search Engines, Information Retrieval, Web Development, Computer Science
16 points by gone35 22 days ago | 1 comments
A new product that solves your tab hoarding problem and forgotten saved items (getstasher.com)
Stasher saves what you browse and brings up related links just when you need them
Web Browsing, Productivity Tools, Software, Bookmarking, Information Retrieval
101 points by aymanalrifai 28 days ago | 7 comments
Information Batteries (2021) [pdf] (raghavan.usc.edu)
Information Retrieval, Research Papers, Computer Science, Databases
21 points by compressedgas 49 days ago | 4 comments
The Tao of Topic Maps (2000) (ontopia.net)
Someone once said that “a book without an index is like a country without a map”.
Information Retrieval, Knowledge Representation, Semantic Web
31 points by Tomte 57 days ago | 5 comments
Understanding the BM25 full text search algorithm (emschwartz.me)
BM25, or Best Match 25, is a widely used algorithm for full text search. It is the default in Lucene/Elasticsearch and SQLite, among others. Recently, it has become common to combine full text search and vector similarity search into "hybrid search". I wanted to understand how full text search works, and specifically BM25, so here is my attempt at understanding by re-explaining.
Search Engines, Algorithms, Information Retrieval
12 points by emschwartz 60 days ago | 1 comments
Ask HN: Niche technical knowledge not found on the internet? (ycombinator.com)
What niche subjects are you interested in for which the knowledge is hard to come by on the internet?
Online Learning, Technical Skills, Information Retrieval
10 points by andrecarini 75 days ago | 3 comments
Ask HN: The Web Post ChatGPT? (ycombinator.com)
The humble chat thread is rapidly becoming the defacto interface to information for so many right now.
ChatGPT, Online Communities, Information Retrieval
7 points by canterburry 77 days ago | 8 comments
Ask HN: Local RAG with private knowledge base (ycombinator.com)
Looking for a free, local, open source RAG solution for running a reference library with 1000s of technical PDFs and word docs.
Information Retrieval, Open Source
108 points by zephodb 82 days ago | 40 comments
The Knowledge Graph: things, not strings (2012) (google)
Search is a lot about discovery—the basic human need to learn and broaden your horizons. But searching still requires a lot of hard work by you, the user. So today I’m really excited to launch the Knowledge Graph, which will help you discover new information quickly and easily.
Search Engines, Knowledge Graphs, Google, Information Retrieval
7 points by 2-3-7-43-1807 83 days ago | 2 comments
Bridging Search and Recommendation in Generative Retrieval (dl.acm.org)
Generative retrieval for search and recommendation is a promising paradigm for retrieving items, offering an alternative to traditional methods that depend on external indexes and nearest-neighbor searches.
Generative AI, Search, Recommendation, Information Retrieval
23 points by saeedesmaili 83 days ago | 0 comments
NotebookLM launches feature to customize and guide audio overviews (google)
NotebookLM is a tool for understanding, built with Gemini 1.5. When you upload your sources, it instantly becomes an expert, grounding its responses in your material and giving you powerful ways to transform information. And since it’s your notebook, your personal data is never used to train NotebookLM.
AI, Productivity Tools, Information Retrieval
335 points by alphabetting 93 days ago | 127 comments
Bloomberg Terminal – The Keys (libguides.nyit.edu)
The Bloomberg Terminal is Color Coded:
Finance, Bloomberg Terminal, Tools, Information Retrieval, Business
4 points by tosh 100 days ago | 1 comments
Phrase matching in Marginalia Search (marginalia.nu)
Marginalia Search now properly supports phrase matching. This not only permits a more robust implementation of quoted search queries, but also helps promote results where the search terms occur in the document exactly in the same order as they do in the query.
Search Engines, Information Retrieval
192 points by marginalia_nu 110 days ago | 24 comments
A new semantic chunking approach for RAG (gpt3experiments.substack.com)
As we saw in my last blog post, there is a shape for stories.
Artificial Intelligence, Text Generation, Information Retrieval
49 points by nutanc 115 days ago | 27 comments
Two kinds of LLM responses: Informational vs. Instructional (shabie.github.io)
When thinking of LLM evals especially in the context of RAGs, it occurred to me that there are two kinds of distinct responses people get from LLMs: informational and instructional.
Generative AI, Artificial Intelligence, Information Retrieval
75 points by shabie 116 days ago | 8 comments
"As We May Think" by Vannevar Bush (1945) (theatlantic.com)
As Director of the Office of Scientific Research and Development, Dr. Vannevar Bush has coordinated the activities of some six thousand leading American scientists in the application of science to warfare. In this significant article he holds up an incentive for scientists when the fighting has ceased. He urges that men of science should then turn to the massive task of making more accessible our bewildering store of knowledge.
Science, Technology, History, Research, Information Retrieval
19 points by joebig 116 days ago | 3 comments
Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (arxiv.org)
Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be "over-compressed" in the embeddings.
Information Retrieval, Text Embeddings, Machine Learning
20 points by mfiguiere 120 days ago | 3 comments
Contextual Retrieval (anthropic.com)
For an AI model to be useful in specific contexts, it often needs access to background knowledge. For example, customer support chatbots need knowledge about the specific business they're being used for, and legal analyst bots need to know about a vast array of past cases.
AI, Contextualization, Information Retrieval
309 points by loganfrederick 120 days ago | 72 comments
AI as an Information Interface (zeynepevecen.dev)
Artificial Intelligence, Human-Computer Interaction, Information Retrieval
29 points by zeynepevecen 121 days ago | 11 comments
STORM: Get a Wikipedia-like report on your topic (genie.stanford.edu)
I have carefully read the above and accepted all the terms and conditions
Information Retrieval, Research Tools, AI, Summaries
143 points by fragmede 124 days ago | 72 comments
Data Engineering Vault: A 1000 Node Second Brain for DE Knowledge (ssp.sh)
Welcome to the Data Engineering Vault an integral part of my Second Brain. It’s a curated network of data engineering knowledge, designed to facilitate exploration and discovery. Here, you’ll find over 100+ interconnected terms, each serving as a gateway to deeper insights.
Data Engineering, Second Brain, Knowledge Management, Information Retrieval, Data Science
123 points by articsputnik 124 days ago | 19 comments
Integrating Vision into RAG Applications (pamelafox.org)
AI, Information Retrieval
26 points by pamelafox 131 days ago | 0 comments
Long Context vs. RAG (jonathanadly.com)
One of the projects I have built is a long-standing retrieval-augmented generation (RAG) application. Documents are saved in a database, chunked into a reasonable amount of text that a large language model (LLM) can handle, and turned into numerical representation (vectors).
Retrieval-Augmented Generation, Information Retrieval
4 points by jonathan-adly 135 days ago | 0 comments