Hacker News with Generative AI: Evaluation

Strengthening AI Agent Hijacking Evaluations (nist.gov)
Large AI models are increasingly used to power agentic systems, or “agents,” which can automate complex tasks on behalf of users.

Artificial Intelligence, Security, Agents, Evaluation

43 points by StatsAreFun 20 days ago | 15 comments

Chatbots Are Cheating on Their Benchmark Tests (theatlantic.com)
AI programs train on questions they’re later tested on. So how do we know if they’re getting smarter?

Artificial Intelligence, Machine Learning, Research, Evaluation

6 points by askl 27 days ago | 1 comments

Can We Trust AI Benchmarks? A Review of Current Issues in AI Evaluation (arxiv.org)
Quantitative Artificial Intelligence (AI) Benchmarks have emerged as fundamental tools for evaluating the performance, capability, and safety of AI models and systems.

Artificial Intelligence, Evaluation, Benchmarks, Research

23 points by rntn 46 days ago | 4 comments

How do we evaluate vector-based code retrieval? (voyageai.com)
Despite the widespread use of vector-based code retrieval, evaluating the retrieval quality of embedding models for code retrieval is a common pain point.

Code Retrieval, Evaluation, Machine Learning

55 points by fzliu 58 days ago | 0 comments

You're Not Testing Your AI Well Enough (tryreva.com)
Large Language Models (LLMs) have revolutionised machine learning, offering unprecedented versatility across various tasks. However, this flexibility poses a significant challenge: how do we effectively evaluate LLMs to ensure they’re suitable for specific applications?

Generative AI, Machine Learning, Evaluation

9 points by alexkirwan 140 days ago | 2 comments

Generated Checklists Improve LLM Evaluation and Generation (arxiv.org)
Given the widespread adoption and usage of Large Language Models (LLMs), it is crucial to have flexible and interpretable evaluations of their instruction-following ability.

Generative AI, Evaluation, AI, Checklists

22 points by hdvr 177 days ago | 1 comments

We're approaching LLM prompt evaluation at QA.tech (qa.tech)
The development of autonomous agents poses a unique challenge that other types of applications don’t typically grapple with: heavy reliance on inherently non-deterministic dependencies at multiple points within the system.

Artificial Intelligence, Software Development, Evaluation, Autonomous Agents

12 points by daniel_mp 197 days ago | 2 comments

Show HN: Anton the Search Relevance Evaluator (objective.inc)

Search, AI, Tools, Evaluation

13 points by pablomendes 210 days ago | 1 comments

IRL 25: Evaluating Language Models on Life's Curveballs (alignedhq.ai)

Generative AI, Language Models, Artificial Intelligence, Evaluation

4 points by pmmucsd 246 days ago | 0 comments

Everyone Is Judging AI by These Tests. Experts Say They're Close to Meaningless (themarkup.org)

Artificial Intelligence, Machine Learning, Evaluation, Bias

28 points by billybuckwheat 258 days ago | 19 comments

eBook on building LLM system evals (forestfriends.tech)