web02.fireside.fm Tue, 07 Apr 2026 07:39:22 -0500 Fireside (https://fireside.fm) Vanishing Gradients - Episodes Tagged with “Evals” https://vanishinggradients.fireside.fm/tags/evals Tue, 30 Sep 2025 17:30:00 +1000 A podcast about all things data, brought to you by data scientist Hugo Bowne-Anderson. It's time for more critical conversations about the challenges in our industry in order to build better compasses for the solution space! To this end, this podcast will consist of long-format conversations between Hugo and other people who work broadly in the data science, machine learning, and AI spaces. We'll dive deep into all the moving parts of the data world, so if you're new to the space, you'll have an opportunity to learn from the experts. And if you've been around for a while, you'll find out what's happening in many other parts of the data world. en-us episodic a data podcast with hugo bowne-anderson Hugo Bowne-Anderson A podcast about all things data, brought to you by data scientist Hugo Bowne-Anderson. It's time for more critical conversations about the challenges in our industry in order to build better compasses for the solution space! To this end, this podcast will consist of long-format conversations between Hugo and other people who work broadly in the data science, machine learning, and AI spaces. We'll dive deep into all the moving parts of the data world, so if you're new to the space, you'll have an opportunity to learn from the experts. And if you've been around for a while, you'll find out what's happening in many other parts of the data world. no data science, machine learning, AI Hugo Bowne-Anderson hugobowne@hey.com Episode 60: 10 Things I Hate About AI Evals with Hamel Husain https://vanishinggradients.fireside.fm/60 0fbc2a65-3bfc-4f8a-83ac-d370f1a30e13 Tue, 30 Sep 2025 17:30:00 +1000 Hugo Bowne-Anderson full Hugo Bowne-Anderson Most AI teams find "evals" frustrating, but ML Engineer Hamel Husain argues they’re just using the wrong playbook. In this episode, he lays out a data-centric approach to systematically measure and improve AI, turning unreliable prototypes into robust, production-ready systems. 1:13:15 no Most AI teams find "evals" frustrating, but ML Engineer Hamel Husain argues they’re just using the wrong playbook. In this episode, he lays out a data-centric approach to systematically measure and improve AI, turning unreliable prototypes into robust, production-ready systems. Drawing from his experience getting countless teams unstuck, Hamel explains why the solution requires a "revenge of the data scientists." He details the essential mindset shifts, error analysis techniques, and practical steps needed to move beyond guesswork and build AI products you can actually trust. We talk through: The 10(+1) critical mistakes that cause teams to waste time on evals Why "hallucination scores" are a waste of time (and what to measure instead) The manual review process that finds major issues in hours, not weeks A step-by-step method for building LLM judges you can actually trust How to use domain experts without getting stuck in endless review committees Guest Bryan Bischof's "Failure as a Funnel" for debugging complex AI agents If you're tired of ambiguous "vibe checks" and want a clear process that delivers real improvement, this episode provides the definitive roadmap. LINKS Hamel's website and blog (https://hamel.dev/) Hugo speaks with Philip Carter (Honeycomb) about aligning your LLM-as-a-judge with your domain expertise (https://vanishinggradients.fireside.fm/51) Hamel Husain on Lenny's pocast, which includes a live demo of error analysis (https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill) The episode of VG in which Hamel and Hugo talk about Hamel's "data consulting in Vegas" era (https://vanishinggradients.fireside.fm/9) Upcoming Events on Luma (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk) Watch the podcast video on YouTube (https://youtube.com/live/QEk-XwrkqhI?feature=share) Hamel's AI evals course, which he teaches with Shreya Shankar (UC Berkeley): starts Oct 6 and this link gives 35% off! (https://maven.com/parlance-labs/evals?promoCode=GOHUGORGOHOME) https://maven.com/parlance-labs/evals?promoCode=GOHUGORGOHOME 🎓 Learn more: Hugo's course: Building LLM Applications for Data Scientists and Software Engineers (https://maven.com/s/course/d56067f338) — https://maven.com/s/course/d56067f338 AI, GenAI, LLMs, data science, machine learning, evals Most AI teams find "evals" frustrating, but ML Engineer Hamel Husain argues they’re just using the wrong playbook. In this episode, he lays out a data-centric approach to systematically measure and improve AI, turning unreliable prototypes into robust, production-ready systems.

Drawing from his experience getting countless teams unstuck, Hamel explains why the solution requires a "revenge of the data scientists." He details the essential mindset shifts, error analysis techniques, and practical steps needed to move beyond guesswork and build AI products you can actually trust.

We talk through:

  • The 10(+1) critical mistakes that cause teams to waste time on evals
  • Why "hallucination scores" are a waste of time (and what to measure instead)
  • The manual review process that finds major issues in hours, not weeks
  • A step-by-step method for building LLM judges you can actually trust
  • How to use domain experts without getting stuck in endless review committees
  • Guest Bryan Bischof's "Failure as a Funnel" for debugging complex AI agents

If you're tired of ambiguous "vibe checks" and want a clear process that delivers real improvement, this episode provides the definitive roadmap.

LINKS

🎓 Learn more:

]]> Most AI teams find "evals" frustrating, but ML Engineer Hamel Husain argues they’re just using the wrong playbook. In this episode, he lays out a data-centric approach to systematically measure and improve AI, turning unreliable prototypes into robust, production-ready systems.

Drawing from his experience getting countless teams unstuck, Hamel explains why the solution requires a "revenge of the data scientists." He details the essential mindset shifts, error analysis techniques, and practical steps needed to move beyond guesswork and build AI products you can actually trust.

We talk through:

  • The 10(+1) critical mistakes that cause teams to waste time on evals
  • Why "hallucination scores" are a waste of time (and what to measure instead)
  • The manual review process that finds major issues in hours, not weeks
  • A step-by-step method for building LLM judges you can actually trust
  • How to use domain experts without getting stuck in endless review committees
  • Guest Bryan Bischof's "Failure as a Funnel" for debugging complex AI agents

If you're tired of ambiguous "vibe checks" and want a clear process that delivers real improvement, this episode provides the definitive roadmap.

LINKS

🎓 Learn more:

]]> Episode 50: A Field Guide to Rapidly Improving AI Products -- With Hamel Husain https://vanishinggradients.fireside.fm/50 3851d92b-389c-4690-90c3-8a54ad73b7d8 Tue, 17 Jun 2025 18:30:00 +1000 Hugo Bowne-Anderson full 1 Hugo Bowne-Anderson Hugo talks with Hamel Hussain (ex-Airbnb, GitHub, DataRobot) about how to improve AI products through evaluation, error analysis, and iteration. They discuss why most teams overlook debugging LLM systems, how to prioritize what to fix, and why evals are not just metrics—but a full development process. 27:42 no If we want AI systems that actually work, we need to get much better at evaluating them, not just building more pipelines, agents, and frameworks. In this episode, Hugo talks with Hamel Hussain (ex-Airbnb, GitHub, DataRobot) about how teams can improve AI products by focusing on error analysis, data inspection, and systematic iteration. The conversation is based on Hamel’s blog post A Field Guide to Rapidly Improving AI Products, which he joined Hugo’s class to discuss. They cover: 🔍 Why most teams struggle to measure whether their systems are actually improving 📊 How error analysis helps you prioritize what to fix (and when to write evals) 🧮 Why evaluation isn’t just a metric — but a full development process ⚠️ Common mistakes when debugging LLM and agent systems 🛠️ How to think about the tradeoffs in adding more evals vs. fixing obvious issues 👥 Why enabling domain experts — not just engineers — can accelerate iteration If you’ve ever built an AI system and found yourself unsure how to make it better, this conversation is for you. LINKS * A Field Guide to Rapidly Improving AI Products by Hamel Husain (https://hamel.dev/blog/posts/field-guide/) * Vanishing Gradients YouTube Channel (https://www.youtube.com/channel/UC_NafIo-Ku2loOLrzm45ABA) * Upcoming Events on Luma (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk) * Hugo's recent newsletter about upcoming events and more! (https://hugobowne.substack.com/p/ai-as-a-civilizational-technology) 🎓 Learn more: Hugo's course: Building LLM Applications for Data Scientists and Software Engineers (https://maven.com/s/course/d56067f338) — next cohort starts July 8: https://maven.com/s/course/d56067f338 Hamel & Shreya's course: AI Evals For Engineers & PMs (https://maven.com/parlance-labs/evals?promoCode=GOHUGORGOHOME) — use code GOHUGORGOHOME for $800 off 📺 Watch the video version on YouTube: YouTube link (https://youtu.be/rWToRi2_SeY) data science, machine learning, AI, LLMs, evas If we want AI systems that actually work, we need to get much better at evaluating them, not just building more pipelines, agents, and frameworks.

In this episode, Hugo talks with Hamel Hussain (ex-Airbnb, GitHub, DataRobot) about how teams can improve AI products by focusing on error analysis, data inspection, and systematic iteration. The conversation is based on Hamel’s blog post A Field Guide to Rapidly Improving AI Products, which he joined Hugo’s class to discuss.

They cover:
🔍 Why most teams struggle to measure whether their systems are actually improving

📊 How error analysis helps you prioritize what to fix (and when to write evals)

🧮 Why evaluation isn’t just a metric — but a full development process

⚠️ Common mistakes when debugging LLM and agent systems

🛠️ How to think about the tradeoffs in adding more evals vs. fixing obvious issues

👥 Why enabling domain experts — not just engineers — can accelerate iteration

If you’ve ever built an AI system and found yourself unsure how to make it better, this conversation is for you.

LINKS

🎓 Learn more:

📺 Watch the video version on YouTube: YouTube link

]]> If we want AI systems that actually work, we need to get much better at evaluating them, not just building more pipelines, agents, and frameworks.

In this episode, Hugo talks with Hamel Hussain (ex-Airbnb, GitHub, DataRobot) about how teams can improve AI products by focusing on error analysis, data inspection, and systematic iteration. The conversation is based on Hamel’s blog post A Field Guide to Rapidly Improving AI Products, which he joined Hugo’s class to discuss.

They cover:
🔍 Why most teams struggle to measure whether their systems are actually improving

📊 How error analysis helps you prioritize what to fix (and when to write evals)

🧮 Why evaluation isn’t just a metric — but a full development process

⚠️ Common mistakes when debugging LLM and agent systems

🛠️ How to think about the tradeoffs in adding more evals vs. fixing obvious issues

👥 Why enabling domain experts — not just engineers — can accelerate iteration

If you’ve ever built an AI system and found yourself unsure how to make it better, this conversation is for you.

LINKS

🎓 Learn more:

📺 Watch the video version on YouTube: YouTube link

]]>