Hacker News with Generative AI: Information Retrieval

Understanding the BM25 full text search algorithm (emschwartz.me)
BM25, or Best Match 25, is a widely used algorithm for full text search. It is the default in Lucene/Elasticsearch and SQLite, among others. Recently, it has become common to combine full text search and vector similarity search into "hybrid search". I wanted to understand how full text search works, and specifically BM25, so here is my attempt at understanding by re-explaining.

Search Engines, Algorithms, Information Retrieval

10 points by emschwartz 2 days ago | 1 comments

Ask HN: Niche technical knowledge not found on the internet? (ycombinator.com)
What niche subjects are you interested in for which the knowledge is hard to come by on the internet?

Online Learning, Technical Skills, Information Retrieval

10 points by andrecarini 17 days ago | 3 comments

Ask HN: The Web Post ChatGPT? (ycombinator.com)
The humble chat thread is rapidly becoming the defacto interface to information for so many right now.

ChatGPT, Online Communities, Information Retrieval

7 points by canterburry 19 days ago | 8 comments

Ask HN: Local RAG with private knowledge base (ycombinator.com)
Looking for a free, local, open source RAG solution for running a reference library with 1000s of technical PDFs and word docs.

Information Retrieval, Open Source

108 points by zephodb 24 days ago | 40 comments

The Knowledge Graph: things, not strings (2012) (google)
Search is a lot about discovery—the basic human need to learn and broaden your horizons. But searching still requires a lot of hard work by you, the user. So today I’m really excited to launch the Knowledge Graph, which will help you discover new information quickly and easily.

Search Engines, Knowledge Graphs, Google, Information Retrieval

7 points by 2-3-7-43-1807 25 days ago | 2 comments

Bridging Search and Recommendation in Generative Retrieval (dl.acm.org)
Generative retrieval for search and recommendation is a promising paradigm for retrieving items, offering an alternative to traditional methods that depend on external indexes and nearest-neighbor searches.

Generative AI, Search, Recommendation, Information Retrieval

23 points by saeedesmaili 25 days ago | 0 comments

NotebookLM launches feature to customize and guide audio overviews (google)
NotebookLM is a tool for understanding, built with Gemini 1.5. When you upload your sources, it instantly becomes an expert, grounding its responses in your material and giving you powerful ways to transform information. And since it’s your notebook, your personal data is never used to train NotebookLM.

AI, Productivity Tools, Information Retrieval

335 points by alphabetting 35 days ago | 127 comments

Bloomberg Terminal – The Keys (libguides.nyit.edu)
The Bloomberg Terminal is Color Coded:

Finance, Bloomberg Terminal, Tools, Information Retrieval, Business

4 points by tosh 42 days ago | 1 comments

Phrase matching in Marginalia Search (marginalia.nu)
Marginalia Search now properly supports phrase matching. This not only permits a more robust implementation of quoted search queries, but also helps promote results where the search terms occur in the document exactly in the same order as they do in the query.

Search Engines, Information Retrieval

192 points by marginalia_nu 52 days ago | 24 comments

A new semantic chunking approach for RAG (gpt3experiments.substack.com)
As we saw in my last blog post, there is a shape for stories.

Artificial Intelligence, Text Generation, Information Retrieval

49 points by nutanc 57 days ago | 27 comments

Two kinds of LLM responses: Informational vs. Instructional (shabie.github.io)
When thinking of LLM evals especially in the context of RAGs, it occurred to me that there are two kinds of distinct responses people get from LLMs: informational and instructional.

Generative AI, Artificial Intelligence, Information Retrieval

75 points by shabie 58 days ago | 8 comments

"As We May Think" by Vannevar Bush (1945) (theatlantic.com)
As Director of the Office of Scientific Research and Development, Dr. Vannevar Bush has coordinated the activities of some six thousand leading American scientists in the application of science to warfare. In this significant article he holds up an incentive for scientists when the fighting has ceased. He urges that men of science should then turn to the massive task of making more accessible our bewildering store of knowledge.

Science, Technology, History, Research, Information Retrieval

19 points by joebig 59 days ago | 3 comments

Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (arxiv.org)
Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be "over-compressed" in the embeddings.

Information Retrieval, Text Embeddings, Machine Learning

20 points by mfiguiere 62 days ago | 3 comments

Contextual Retrieval (anthropic.com)
For an AI model to be useful in specific contexts, it often needs access to background knowledge. For example, customer support chatbots need knowledge about the specific business they're being used for, and legal analyst bots need to know about a vast array of past cases.

AI, Contextualization, Information Retrieval

309 points by loganfrederick 62 days ago | 72 comments

AI as an Information Interface (zeynepevecen.dev)

Artificial Intelligence, Human-Computer Interaction, Information Retrieval

29 points by zeynepevecen 63 days ago | 11 comments

STORM: Get a Wikipedia-like report on your topic (genie.stanford.edu)
I have carefully read the above and accepted all the terms and conditions

Information Retrieval, Research Tools, AI, Summaries

143 points by fragmede 66 days ago | 72 comments

Data Engineering Vault: A 1000 Node Second Brain for DE Knowledge (ssp.sh)
Welcome to the Data Engineering Vault an integral part of my Second Brain. It’s a curated network of data engineering knowledge, designed to facilitate exploration and discovery. Here, you’ll find over 100+ interconnected terms, each serving as a gateway to deeper insights.

Data Engineering, Second Brain, Knowledge Management, Information Retrieval, Data Science

123 points by articsputnik 67 days ago | 19 comments

Integrating Vision into RAG Applications (pamelafox.org)

AI, Information Retrieval

26 points by pamelafox 73 days ago | 0 comments

Long Context vs. RAG (jonathanadly.com)
One of the projects I have built is a long-standing retrieval-augmented generation (RAG) application. Documents are saved in a database, chunked into a reasonable amount of text that a large language model (LLM) can handle, and turned into numerical representation (vectors).

Retrieval-Augmented Generation, Information Retrieval

4 points by jonathan-adly 77 days ago | 0 comments

Why is this a research paper? HybridRAG = VectorRAG context and GraphRAG context (arxiv.org)

Research, AI, Information Retrieval

4 points by blizzard_knight 91 days ago | 1 comments

BMX: A Freshly Baked Take on BM25 (mixedbread.ai)

Information Retrieval, Search Engines, AI

91 points by breadislove 98 days ago | 11 comments

Ask HN: Best resources to learn about Search and Information retrieval (ycombinator.com)

Search, Information Retrieval, Learning Resources

20 points by Heloseaa 118 days ago | 2 comments

Find who knows what on HN (wilsonl.in)