AI · Article or course
Modern Information Retrieval Evaluation In The RAG Era
Free video lecture on modern IR benchmarks and evaluation for RAG pipelines.
This free 45-minute Maven Lightning Lesson explains why traditional IR benchmarks like BEIR fail to reflect real-world RAG demands. Led by Nandan Thakur, Hamel Husain, and Shreya Shankar, the session introduces FreshStack, an evaluation benchmark built from recent StackOverflow and GitHub content.
What it's for
- Learn why standard IR leaderboards fail to predict real-world RAG retrieval quality
- Evaluate RAG retrieval pipelines using modern relevance, grounding, and diversity metrics
- Understand the methodology behind the FreshStack benchmark
Features
- Analysis of traditional IR paradigms vs. modern RAG systems
- Detailed breakdown of BEIR benchmark limitations and test collection staleness
- Introduction to FreshStack, a benchmark based on StackOverflow and GitHub queries
- Overview of modern RAG metrics covering diversity, grounding, and relevance
- Full video recording with timestamped chapters and Q&A session
Pricing
Free with Maven registration
Details
- Type
- Article or course
- Category
- AI
- Topic
- AI Courses
- Open source
- No
- Platforms
- Web
- For
- Machine learning engineers, AI researchers, and developers building RAG systems
- Tags
- rag, evals, benchmarks, information retrieval, llm
Why it was saved
- @mavenhq: Solves the critical bottleneck of evaluating RAG retrieval quality and answer accuracy using academic rigor and real-world evaluation metrics.
Related
Anthropic's official online learning portal for Claude and AI development
Free 30-minute video lesson on production RAG architecture, security, and scaling.
Free workshop on architecting enterprise-grade AI agent systems.
Free Maven video lesson on evaluating and benchmarking AI agents
Free 45-minute lesson on foundational building blocks for AI agents.
A free 30-minute workshop comparing RAG and fine-tuning architectures.
Updated