Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft

Hi everyone. Uh my name is Nick and I'm

here with Ashe to give a talk about

ethos.

Uh we are from we're from Lyft and we've

been building lift customer support AI

agent for a year or two now and gave a

lot of a lot of thoughts about how to

build eval that actually matters and

scale um our AI agents multi multiAI

agent system. Uh just a little bit of

quick introductions. My name is Nick.

I'm a data science manager. I've been at

lift for six years a long time lifter.

uh really excited to talk to you a

little bit more about euph.

>> Hi everyone. Uh I'm Aka. I'm in Nick's

team and we've been working together

uh on customer support agents for Lyft.

Uh improving the hardness, improving the

evals, things like that. and I've been

at Lyft for almost four years now and

very excited to be here and talk about

uh building events that actually matter.

>> Super excited to be here and super

honored to be to be on the online track

for AI engineer warfare. Um yeah and

let's dive in. Um for the agenda of

today uh our talk will primarily focus

on eval uh we will start by sharing how

we think about the end to end pipeline

for our evaluations for building support

customer support AI agent system. Uh

we'll go into deep dive into each

component uh more deeply as we go. Uh

we'll start by talking about offline

evaluations, online evaluations,

eval harness as well as what we are

planning to build going forward.

All right, let's dive in. Um I want to

quickly explain the

you know high level system of how we

think about evaluation system for AI

agents. Uh so here you can see we have

the development phase and the production

phase. Uh so during development if

you're building agents you should be

very familiar with uh you know managing

contacts building rep pipeline to give

your agents educational context uh

defining your tool building your agentic

graph as well as writing a system

prompt. Uh so once all of that engine

agent engineering process is done you

have an you have an AI agents

uh the way we think about this is before

we launch this AI agents to productions

we want to go through a rigorous offline

evaluation process to make sure that

this agent uh actually has sufficient

performance before we launch this to a

live uh users. So coming from you know

data science and machine learning

background uh we've been building model

machine learning model for for for for a

while and I think the way that we think

about agent development is very similar

to building machine learning models as

well. If we are running um offline

evaluations for our machine learning

model before that goes to productions, I

think we should do the same for AI

agents as well or any other agentic

platform agentic applications.

uh but what I think you know offline

evaluation how that how that is

different than uh traditional machine

learning model is that you know we

typically we're building specifically

for customer support AI use case we're

building a agent that's multi-turn uh so

for offline evaluation there will be a

component of simulated conversations so

you want you typically want to have a

data set synthetic data set that's

represent representative of your

productions traffic. Um, have a use LM

that plays plays out the complete

multi-turn simulated conversation. Um,

as well as a having a grader such as a

LM as a judge um to be able to evaluate

how good that interactions was. And then

we have a launch gate, right? We want to

make sure that we have certain criteria

on our offline eval and we're meeting

that criteria before we decide to launch

this AI agent to productions.

And so the the real imperative here

really is that we don't want to use our

live user as you know test data for our

AI agents. And I think in any any cases

that that is not not good practice. So

we really want to emphasize the

importance of having a evaluation

process.

So once the agent pick productions we

also have an online evaluation pipeline

as well. We have our own favorite

tracing tools to uh trace all the

executions and context that the AI agent

use to respondse to a real user in in

productions environment. We have our

online grader as well that grades how

well our AI agent is doing in

productions. And as far as having a

human in the loop pipeline to do error

analysis, identify failure mode and

feedback that insights to the

development teams to continuously

improve our AI agents.

I want to quickly go over I think three

of the most common reasons why we think

uh evaluation typically fail in uh for

for different teams. So the first reason

is that you know the the greater that we

create the scores that we create needs

to be meaningfully gating something. Um

this is what we really emphasized on in

the previous slide that we need to have

a launch gate. If your LM is a judge is

just floating out there that there's a

score but no one is really using that

score as a meaningful gate uh for your

development and productions environment

then that LM is a judge is not not

available.

Uh we have also seen a lot of a lot of

mishap people have when they're creating

the LM judge. Um there's a lot of

different opinion out there on in terms

of how do you create a good LM judge and

typically uh and unfortunately also very

early on in our journey the LM is the LM

judge that we created are very noisy too

generic uh it will output a score and

but people don't really believe in uh in

what the LM judge is doing or they don't

think the LM judge insight is actionable

And finally, I think when something

regresses in productions, we need to

have clear mechanism to be able to to to

catch that regression as well as

identify clear owners to be able to take

actions on uh the insights of our

graders and regression gates.

Very cool. I want to sequence into

talking about our offline evaluation

system. And as we as I touched on early

on, I think this is the most critical

piece uh going from development cycle to

to productions, we really want to have a

robust offline evaluation system to be

able to get more confidence in uh in the

AI agents that we are shipping to

productions. We took a lot of

inspiration from um this paper called

Taobench uh which is developed by the

wonderful people at Sierra AI. uh and

this is specifically for customer

support AI agent but we also think this

is applicable for any userf facing

agendic applications.

Uh so here as you can see this is a

offline offline simulations where you

have the AI agent as well as a user LLM

that are interacting with each others to

produce the uh multi-turn traces.

uh you have agent domain policy which is

essentially what the you know for each

customer support use case there is

instructions policy on how to handle

different different customer support

issues.

So taking inspiration from towbench this

is sort of a high level approach that we

have in creating our offline simulator.

uh as we mentioned earlier we have our

lang graph agents that we've built and

we have defined

uh an instructions for our our user

length right as you can see for

simulation wise we define the user

intent we define you know what this

intent is supposed to uh to represents

and we define the user data point or the

world state of of our user. For example,

you know, the driver that are coming to

us might be a luxury driver that might

have been driving for us for for a

couple of years and so and so forth. And

finally, to define the user user

behavior, user personas as we would

typically see with our real uh you know

real lift user as well, we also created

these different persona for our user. uh

one example is you know this can be a

loyal longtime lift customer but they

are frustrated with uh with a lift

earning systems.

So in our offline simulator we have this

lang graph agent that are you know

interacting with our user ln model and

generating this multi-turn trajectories

uh of uh this multi-turnic

uhic trajectory and we also built our

offline offline grader or offline

evaluator uh L1 judge is a big component

of that. we will dive a little a little

a lot more deeper into how to build a

great LM as a judge.

uh and apart from you know alm is a

judge we also have more deterministic uh

evaluator as well and these usually

looks like a code assertion as you will

see in traditional unit test uh and for

example here some some of the

deterministic criteria that we have

created so far looks something like this

right if you know if in this in this

specific interactions the AI agent is

supposed to grant concession we'll write

like a rules like this whether or not

the AI agent has indeed grand

concessions uh and we will compare the

agent uh agent tool calls with the

expected outcome to make sure that uh we

h we can measure the accuracy of the

agent instruction following behavior.

So one of the big gotcha that we face

with running our offline evaluation is

the creating syntactic data is is one of

the key challenges in making sure our

offline data set is representative of

our productions um data. So uh here we

you know um here we have a meme here. Um

what ideally what you don't want to be

doing is just to simply prompt an LM

model to generate 50 different test

query for your offline data sets. Uh so

here's a couple of suggestions where you

can approach this you know much more

much more realistically and assemble a

data set that that closely resemble to

your production data. So this is also

something that we we try to do for

customer support AI agent as well. We

take uh we we we we take some sample

from our production data. Uh so uh lift

user has been reaching out to support

for a long time. We take a sample of our

real production ensemble and uh

supplement our offline data set with

that. and you and the second thing that

we can we we do is we mutate uh

different criterias for from our offline

um data sets to be able to cover

different golden path and edge cases.

So

one problem with our offline evaluator

is that

as as we mentioned in the previous

slide, if you were simply uh sampling

different 50 50 test use uh test query

from uh by using a lm model. Uh another

thing another big gotshot that we we

would face in in building this offline

simulator was that we were using a

frontier lab LM uh model to roleplay

lift user in our offline evaluations.

Um and for the most part, you know,

frontier lab model are trained to be

helpful assistant rather than a uh lift

user that might not uh that might not

sound as nice as always. So I I think uh

for everyone that have tried customer

support before, you know, you don't

typically reach out to customer support

agent uh with a very nice verbin. Um so

in our first pass at running our offline

evaluation what we've noticed is that

our LM user sounds almost too nice and

as you can see on on the right here

these verbadiums are very very very very

use the LM user are very patiently

explaining the issues that they are

facing in productions

and our first attemp at our offline

evaluation gave us 90 fast pass rate or

accuracy rate right uh this almost

sounds too good to be true and I think

it indeed is too good to be true so in

reality these are the real user

verbating that we get in productions

um as you can see here most user they

are they're impatient they're already

frustrated so the berating they they

they don't want to explain their issues

like a LM user

Uh so typically what we see in

production is is something like this. Uh

and in fact in reality the this makes

you know AI agents much more difficult

to evaluate.

So what can we do here? How how can we

make sure our lift user LLM uh simulate

this real life lift user much closely.

Right. Uh so

what we what we how we approach this is

we fine-tune a LM model with lift user

verbatim. So instead of speaking like

this very verbose and very nicely

impatiently explaining their issues uh

our LM user will

produce verbatim that's resemble this

much closely. And the benefit of this is

while you know we we did see our

evaluation score goes down after we have

fine-tuned a LM model that speaks more

like our lift user and therefore making

our evaluation more difficult. Uh but in

reality, this is really what you want uh

when you're building this user

simulator, right? If you have an EAL

that's too easy, that doesn't give you

any real uh production insights into how

your AI agent is actually going to

perform. Um and this also give you a lot

more room to be able to tweak your AI

agents to deal with quote unquote

difficult user. And as I as I gave a

little bit of sneak peek earlier as

well, we also did a lot of work to

define lift user personas. Uh this can

really ground our LM user to adopt a

specific lift user persona and therefore

uh simulate our real life user much more

closely. Uh a couple of different user

persona that we've defined here uh are

you know bypasser user who just want to

escalate to agent regardless of and not

giving AI a chance refund seeker uh AI

skeptics.

So the

we we really also took inspiration from

this paper uh from Microsoft user alen

uh Microsoft paper uh and I think they

adopt a very similar similar approach as

well they fine-tune a uh user outlet

model and so evaluation score goes down

um but I think in reality this is really

really what's expected and and good for

for your applications.

>> All right, I I'll hand it off to Ash to

talk more about LM judge.

>> All right. Uh, hi everyone. So, I'm

going to take from here. I'll talk about

second problem which we usually face

when we are doing eval with LLM as a

judge. And here you can see uh this is

how pretty much everyone is using LLM as

a judge to evaluate their agents. The

focus can be slightly different based on

the use case. Someone can focus more on

safety. Someone can focus more on cost

and latency. Someone can focus more on

quality. But people are going to measure

these sort of metrics more or less. So

uh we want to detect leaks, we want to

detect safety issues um and and things

like that. So some parts are

deterministic which can be evaluated by

code and some are not which are

evaluated by the LLM.

Now the problem with this approach is uh

that these metrics are too generic and

not actionable. So for example, we also

started with our eval journey using

pre-built metrics from deep eval which

were uh which were measuring tool usage,

appropriateness, response, helpfulness,

conversation, naturalenness,

completeness and things like that and we

did see those metrics but the problem

was these metrics were not actionable.

They were not giving us any actionable

insights. If something if let's say

response helpfulness is 0.5 then what do

we do with it?

So uh things like and other other scores

like toxicity score, bias, fairness,

conciseness, all these are kind of

relevant. But if the metrics are just uh

scores, we we don't know what to do with

them. So we can use these pre-built eval

metrics as a baseline but we shouldn't

use them as our core eval metrics

because we want eval metrics to be

actionable and tied to the business

outcome or the product which we are

focusing on.

Uh

okay. So what LLM as a judge should be

uh we collaborate very closely with u

domain experts and utilize their

insights. So eval should be framed

around a task success or failure and a

binary outcome is very easy to calibrate

and train um LLM jets that can

consistently score your agentic

trajectory. When we partner with domain

experts and data scientists to build

metrics which are actionable and aligned

with business goals, we can see much

more meaningful results uh and

actionable insights. Not only this is

more consistent uh but when an agent

fails an interaction, we can

systematically analyze the error pattern

and have and actually know what what we

can do to fix this. Here is an example

of an uh action enable metric for our

use case which is called education

rubric. Here the AI we we define the

metric how it should be and we define

success and fail criteria for LLM judge.

So for example the AI agent tries too

many times to educate the user uh if it

could have escalate escalated the the

issue

or if it's escalating too soon without

giving any chance to educate the user.

So things like that comes under failure

category. So we we mark this as fail. Uh

but if it's an expected behavior, we

mark it as as pass.

Uh okay. So

with this how do we validate if our uh

LLM judge is working as expected?

First thing is we need to treat it as a

classifier. So how we train our

classification models in machine

traditional machine learning, we can

also treat our evaluation judges as as

those traditional ML classifiers with

binary outputs. Once we have binary

outputs for every metric or a task based

on our business goals and functional

requirements, we can handle label uh

around 100 examples with pass fail

labels and then split the data into uh

train dev and validation sets like how

we used to do with machine learning

models.

Uh then we score precision and recall uh

for our judge based on human label

ground truths which will give us an

actual report on how good our uh judge

is performing.

Uh

and how do we split our data to do to

calculate precision and recall is this.

So we split the data similar to how we

used to do in training machine learning

models but the difference is the

percentage of splits. Uh and here we are

not actually training model weights. So

we are just using the data to inform

judges prompt. So the percentages are a

little bit different. For example, in

training data, we we pick a few short

examples from that from this set uh for

the judges prompt and then we iterate

the prompt against the dev set uh and

improve our hardness or our prompt and

then finally we validate against the

test set uh to see that we didn't

overfit on the dev examples.

This is the practical uh

way of like splitting the data and then

calculating precision and recall uh

scores for your judge to actually know

that the judge is working as as

expected.

All right. Next slide. Okay. So

this is another thing which uh

this is another thing which most of us

ignore when we are doing evaluations uh

which is criteria drift and validating

the validators. The key idea is that we

actually discover what our evaluation

criteria is by looking at the data and

grading our outputs.

uh our and our sense of uh quality will

also evol evolve uh with with new data

we see and more uh examples we grade. So

the evaluations should not be decoupled

from model observations. In fact they

should be developed uh they should be

co-developed with the model when we are

testing the evaluator and calculating

the precision and recall scores.

So uh we there there's always a gap when

we when we talk about LLM as a judge. We

cannot define the criteria beforehand

and then evaluate agents against them.

Uh I our criteria should also be

evolving as and when we see more

examples and then we should refine our

metrics uh for our judges and then

evaluate judges on top of those metrics.

Okay.

So the next step uh

one of the things which which can make

the numbers we are reporting more

meaningful is to add some statistical

rigor to it. Right? Uh we report

alignment rates as bare point estimates.

So if we add confidence intervals, we do

some calibration and proper sampling, uh

the same numbers can become more

meaningful.

We should definitely reserve the

expensive rigor for the moments. Uh

number actually gates something or like

like a shipping decision or we are

reporting numbers to u company leaders.

But depending on the use case we should

definitely have uh confidence intervals

for the for the numbers we report

because every score needs an interval.

So this is just a small example to give

you uh an insight on what this actually

means. Uh let's say we have two

evaluators and one scores 84% and one

and the other scores 88% and the number

of samples we number [clears throat] of

traces which we have uh used is let's

say 50. So to show that like this is

this is a very small gain and we we need

much more than 50 examples to actually

uh show that this gain gain is real. Uh

with just 50 examples and only four

percent point percentage gains gain we

we don't actually know if this gain is

real or not. So bigger gains and bigger

gains and paired designs need far less

uh and we can reserve the rigor

statistical rigor for for things which

matter the most.

Okay. So this is a non-exhaustive list

of observed eval anti patterns. I'm not

going to read all of them but uh these

definitely contain some lowhanging

fruits. We need to put in time and

effort if we if we need meaningful

evals. So we cannot rely on LLMs for

everything yet. uh and what I can say is

ignoring the data is one of the most

important uh things which we shouldn't

do uh and which we sometimes don't focus

on due to lack of time or resourcing but

it acts as the foundation for meaningful

eval evaluations. If you don't look at

the data, you won't be able to create

meaningful criteria or labels. And if

you don't have labels, you won't be able

to evaluate your judges. And if you are

not evaluating your judges, you don't

know if your u agentic pipeline is

working as as expected. So this acts as

a base. So one of the most important

things we should not ignore.

Okay. uh we we said that we want to make

metrics more actionable

and standardize the pipeline but how

actually we should do it. So this this

gives you like a template to to do an

error analysis loop and it is important

to know that this loop is something

which runs continuously. It's not uh not

an one-off audit. Uh so we deep dive

into raw traces. So once we have logged

our traces for our agentic flows, multi-

aent systems or whatever we have in our

uh use case then we pinpoint failure

modes. So basically we try to identify

what exactly is failing and we only keep

the metrics that change a decision. We

remove all the noise. We only prioritize

on the metrics which are tied to

business use cases, functional uh

requirements and actually something

which changes a decision. Then we form a

fresh premise

to re-evaluate and then we repeat it. So

we can have like a regular cadence of

doing this pipeline. It can be weekly,

it can be bi-weekly but this is

something which needs to run uh

continuously and it's it's not a one-off

audit.

Okay. So, tracing tracing as we said is

uh one of the most important things and

everything kind of depends on it. For

diving deep into raw traces, we

definitely need to log them first. So,

we can use tools like Langmith,

Langfuse, etc. to log and view the

traces. Uh and here each trace captures

the full graph execution. which nodes

ran, what LLM saw, which tools were

called, uh what was the token usage,

what was the latency for every call and

things like that. We can also enrich

traces with metadata if you want, which

gives you more insights than uh than the

actual data.

Okay. And we also have annotation cues.

Annotation cues are nothing but uh an

interface which is very helpful for

domain experts to label or give feedback

to evaluators and in an easy to

understand UI. So they don't have to

look at the raw traces uh JSONs and

stuff like that to figure out what to

focus on. They can use this annotation

queue and they can give feedback or

label examples easily. We can then add

these traces to data sets for offline

evaluation or we can use this for our

calculating our precision and recall for

our judges. So this forms the basis to

validate the evaluation with ground

truth labels.

>> Okay. I now I'll hand it over to Nick to

to close the evaluation loop.

Thank you Ash. Uh and I think the the

goal of having eval is to be able to

feed uh our evaluation insights back

into improving the model performance or

the agents performance. So here uh I'll

introduce a couple way that we think

about continual learning for our AI

agent and closing the evaluation loop

and here you have model learning,

context learning and harness learning.

Model learning is really about you know

post training updating the underlying

model weights and training a custom LM

model. Uh context learning and harness

learning is really more about improving

uh everything else other than the model.

Uh so context is improving what

information the agent actually sees.

This can be document. This can be uh the

store memories of the user. Two outputs

and so and so forth. Harness uh that

means updating the model system prompt

and two schemas control flow routing

retries and and so and so forth.

So I think from you know the error

analysis that actually has shared

earlier I think that has really helped

us to understand how can we improved uh

our agent prom as well as updating u

updating our knowledge base and tune the

context management strategy that we have

our AI agent the identify failure mode

really help us uh be able to feed that

insights into actual improvement for the

agent AI agents.

So I want to quickly talk about what's

next for us. Uh in our journey of

building customer support AI agent at

lift uh we

have a we we we have we have some some

level of ability to run our offline u

simulator but in fact I think you know

it's not repeatable. These are currently

stored as uh scatter script across

different notebooks and different

analysis repo. Uh I think one thing that

we're looking really really looking into

investing is a systematic eval harness.

Uh then having a

uh a harness system that can help run

our offline evaluation in in a

systematic and standardized manner and

allowed different people to contributes

to uh our evaluation suite with a very

we predefined uh primitive and config

based workflow.

And another thing that we've been

talking thinking a lot about is uh post

training. uh as we mentioned earlier you

know identify model failure mode has

really helped us tune the agent context

and as far as the agent harness uh but

over the years we have gathered a lot of

real user signals on our agent

performance as well uh so we were really

starting to think about how do we

fine-tune uh model that does different

tasks for our customer support AI agent

as well as framing a reward modeling

problem to to enable reinforcement

learning.

I want to quickly share a little bit

about the work that we're doing with

eval hardness and how we think about

building uh building building evaluacing

agentic applications. Uh so again, EVAL

is really that scaffolding that you that

we need to be able to run EVA

efficiently uh in a standardized format

across uh all the different agents and

sub agents that we have for customer

support AI agents and and as you can see

from my offline simulator slide earlier

uh our eval harness is config driven and

these are typically stored as YAML file

that's easily editable uh by different

contributor and not just by engineers.

Uh analysts and data scientists can can

contribute to this uh evaluation suite

as well.

And you know with thousands if not tens

of thousands of examples uh in our

evaluation suite we need you know

parallelisms and throughput to be able

to run uh our offline evaluation in a

reasonable amount of times and to enable

uh user to be uh different user to be a

to be able to contribute to our eval

evaluation suite. We also define you

know primitive around our eval harness.

Uh these are high level things like

task, data sets, personas, LM adapter

and an evaluator.

And after all you know the benefit of

having a eval hardness is that you can

define the config once and run these

eval indefinitely many times across

different at different touch points or

different gates of your agent

development process. This can be you

know locally when you're developing this

agent

uh at any point when you tune a prompt

you can run the evaluation suite and get

a immediate in uh immediate feedback on

how your agent is doing compared to the

previous versions. We can run these at

pre-commit hook uh to make sure that our

performance doesn't degrade before we

push a change uh to uh our agent

service. Uh another area that we are

looking at is also at CI/CD and how we

can use our eval harness to uh build our

regression test suite accept acceptant

test suite as well.

So that wraps up our presentation today.

uh we've gone through a lot of we've

gone through a lot of uh a lot of

different topics uh for evaluations

uh really I think this is an endto-end

journey uh for building a evaluation

pipeline that works for customer support

AI agent or any userfacing agentic

applications

uh ashen and I we are very interested to

hear about you know what you all have

been working on and share share any

learnings that you have for eval. So

feel free to contact us uh if you have

any questions or you you just want to

share about brainstorm about how to

improve your evaluations.

All right, thank you and I hope you'll

enjoy the AI engineer warfare.

>> Thank you.

>> Thanks all.