Hacker News with Generative AI: Text Embeddings

The best way to use text embeddings portably is with Parquet and Polars (minimaxir.com)
Text embeddings, particularly modern embeddings generated from large language models, are one of the most useful applications coming from the generative AI boom.
Text Embeddings, Generative AI, Data Science, Tools
247 points by minimaxir 37 days ago | 59 comments
Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (arxiv.org)
Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be "over-compressed" in the embeddings.
Information Retrieval, Text Embeddings, Machine Learning
20 points by mfiguiere 195 days ago | 3 comments