The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

[music]

>> All right. Um, thank you for joining my

session.

>> [applause]

>> Thank you, man.

Uh, I'm Sandy. Uh, I'm a technical lead

uh, for data and AI at Databricks. Um,

prior to working in Databricks, I worked

in Amazon Web Services uh, for 5 years

as a principal architect for data and

AI.

Uh,

in the past few years, I worked

extensively uh, building and scaling

data and AI platforms using distributed

systems and technology.

And in the past couple of years,

specifically, I've been working with

customers trying to figure out what we

do with this new AI technology.

Uh, when I say new AI, AI has been here

for a long time, but we all started

experimenting quite exponentially uh, in

in the past couple of years, right? And

I have learned a great deal of lessons

on from building demos and how to take

those demos to production working with

different customers in

uh, B2B software industries and then uh,

regulating industries like financial uh,

services.

So, in this session,

I want to share

a playbook, a framework that I put

together from lessons that I have

learned working in the trenches

uh, that you can take and apply on when

you think about how to put your AI

systems into production. And I think

this session is nicely placed in the

afternoon because what you can do now is

in this framework, you can fit the

different

um, knowledge, the knowledge that you've

gathered attending these different

sessions throughout the day and see

where they fit in each of these

you know, elements in the framework.

So, when I started 2 years ago, this is

the pattern I noticed in every customer

conversation, right? So, everyone wanted

to do something with AI. Uh, there was

immense pressure from the top to do

something, to build a demo.

And every conversation started with

let's choose the model, right? And it

was nobody's fault because the market

was like that. We were talking about

models, the models were new technology

for us, right? And every conversation

started, shall we use GPT? Shall we use

Claude?

You know, there was huge debate with

with within organizations. Then, you

would choose a model, you'll build some

features, offsets of over what features

to build for that application.

Uh, you would build that in a controlled

environment, so predictable data sets,

you know, um, limited scenarios, and

then it looked great as a demo, and then

leadership would get happy, they would

sign it off, and they'll put it into an

environment in a production environment.

Then,

after a few weeks, people would start

asking questions that what the hell is

AI doing?

Right? Why is it not answering the

questions the way we expected it to

answer when we were doing the demos?

Uh, it would result in not only

less you know, no realization in return

on investment, but also loss of money

and effort in building these demos that

can never scale to production.

Throughout these uh, meetings, I

gathered three insights that connect to

everything that you we are talking about

when thinking about taking uh, AI to

production. The first one is the

observability gap, right? When we use AI

and put it into production, if we can't

see what it is actually doing, if we

can't trace every decision that it's

making, it's no use in production.

Second is the evaluation gap gap. A lot

of these conversations that we were

doing, we were not actually thinking

about what is what is that one thing

that we are measuring. Yes, we talk

about accuracy, we talk about latency,

we talk about groundedness, but we were

not defining what is that exact thing

like that matters to the business, and

how can we build a system that can

continuously measure that, whether it's

improving, whether it's not improving,

like what what is that system that we

need to build. And that was that

evaluation gap that I noticed. And the

third is the governance gap. Like we

were not actually thinking what happens

when AI fails in production. Who's

accountable? Who do I go to when

something happens at 3:00 a.m. in the

morning, right? Who needs to own the

data assets that feed some AI responses?

What happens if AI

you know, uh, talks um, uh, um,

nonsense to a customer, right? What what

happens, right? So, there is no

accountability, no governance around it.

And these three insights led me to

build a framework

on how

I think AI should be taken to

production, and this has been

implemented across multiple customer

organizations, and I think this is

something that you can pick up from

here.

These are the five pillars, and these

are absolutely what you need to think

about even before starting a project,

right? Then you start build them

gradually,

preferably in sequence, but in real

life, I know that this sequence don't

work, but these are the pillars that you

have to know about and you have to think

about when start building. First one is

evaluation.

Before touching any code, before

discussing about any models, any

features, you have to think about when

we build this system, how do we measure?

What does success look like, and what is

that system that will help us

continuously measure what success looks

like for us?

Second is how do we trace each and every

decision that AI makes. It's not only

important for the performance of the AI

system, it is also important for the

regulators. In Europe or in a lot of

companies, especially in regulated

industry,

you cannot even onboard AI into

production without having tracing and

observability in place.

So, this is a must-have.

The third is the data foundation,

right? Uh, I I I I think of data

foundation in in two ways. One is the

question data, so that is basically

the data needed for the AI to answer

questions that users ask to it. So, it

could be your pre-training data, post

post-training data, data that you use

APIs to hook onto and and get to the

answer that the user needs. The other

one is the tracking data, related to the

tracing data in observability, but when

you think from the data foundation and

and uh, data strategy perspective, this

needs to be handled in this pillar

because you need a whole data strategy

now with tracing data, especially when

you run hundreds of agents in your

organization.

Fourth is orchestration. One agent would

work pretty well. You don't need to

think about orchestration. But when you

onboard five agents, the the complexity

increases exponentially, right? You will

have multiple coordination patterns

between these agents, they will need to

talk to each other in multiple different

ways, they will need to each wait for

each other's responses, there's a lot of

complexity that comes in. And that's

where orchestration patterns and

thinking about how you will orchestrate

your agents in a particular system

becomes really important.

Fifth is governance. This is where

you think about what happens when

something fails. Who's accountable? How

do we govern data? How do we secure it?

How do we secure our systems? How what

do we make sure how do we make sure that

no one injects into our agent and leads

uh, to you know, misbehavior, right? Or

loss of reputation.

So, in the rest of the session, I will

dive a bit deeper into each of these

pillars and tell you how how you can

think about when you start working with

them, right? The first one is

evaluation.

Evaluation is basically specification

for your AI system.

You define success. As I mentioned, it's

not like

talking about accuracy. You have to

define it with numbers, like what

accuracy is is is good for your business

use case,

right?

Define it in numbers.

Uh, define what kind of you know, false

positives you can handle. What should be

the deflection? So, this is this is an

example from a

a retail chatbot, right? A banking

chatbot where

when you implement a chatbot with an AI

agent, one of the main goals is to

deflect simple queries

um, to the agent so that a human agent

don't need to uh, deal with them, right?

And so, you need to uh,

you need to track those queries and

track those numbers and put that system

in place. Second is building those test

test cases, like the evaluation data

set. You've heard about golden data sets

in evaluation. Talk with the domain

experts and find what is actually

happening in real life on the ground.

Like what answer would a support human

support agent

um, give to a customer on a particular

question. Collect those information.

What happens in gray areas, in edge

cases, like what happens when a human

sees a customer asking a confusing

question, right? Collect those into a

data set, and then automate your AI

testing, right? So, you put a question

to AI, it answers, take that answer,

compare against the test set, and

automate this whole pipeline so that

when you put AI in production, that

pipeline can actually take live

responses and evaluate against the test

data set that you're building, and then

give you the result in terms of how AI

is performing against those numbers and

the goals that you've defined.

When we talk about evaluation,

there are three main layers that I see

appear across organization, and this is

an architectural decision that you need

to make when you build these evaluation

systems, right? The first layer is

deterministic. These are the easy stuff,

like you know, checking formats, you

know, checking email formats, phone

formats, the regular expression things

that we have already been doing with our

coding systems, right? The uh, the the

other other is like you know, you could

use a classic ML models for name entity

recognition to for intent

classification,

for understanding what is first name,

last name, PII detection, etc.

So, the these these are easy stuff,

cheap stuff, you should get them out of

the way. We have already been doing this

for years. The second layer is the

non-deterministic semantic stuff, all

right? This is where groundedness comes

in. This is where we implement

technologies like LLMs judges. We all

know what LLMs judges are, right?

Everyone? Okay, I see a lot of nods. So,

um

Again, this This is a pretty simple

version of how a uh how a prompt would

look for an LLM as a judge. Um

So, with LLM as a judge, you you you use

a separate LLM from the primary LLM to

judge the response of the primary

model.

And when you do that, you tell the

secondary, the judge model,

on how it should uh judge the primary

model's output. So, it could be around

safety, groundedness, you know,

relevance to the answer, etc. etc.,

right? Again, that can feed from a lot

of uh the evaluation data set that you

have created, right? To look at what are

the expected answers, and then it can

check against that. This is a sample

prompt on how these things work, but I'm

sure you've attended some of these

sessions where you've seen vendors doing

this automatically at scale. Uh for

example, in Databricks we in MLflow

you'll find automatic LLM as judge, uh

where you can create these custom LLM as

judges that run automatically on traces.

That's your second layer. The third

layer is behavioral, right? This is

where uh you think about a tool calls,

like is our agents calling the right

tool? Are they getting into loops? So,

for example, um you know, the first

layer you you can have a user ask a

question, "What is my account balance?"

And

you could go and check that, okay, this

there is no deterministic problem with

it. The seman- the agent answered right,

that, "Okay, your account balance is

this many dollars."

And that was right, and you can see this

is this is right, but when you go into

the behavioral checks, you will see that

the agent uh actually made three calls

to the database to find that answer.

Right? And that is because it was doing

making duplicate calls for whatever

reason.

Calls failed,

you know, calls did not work, it went

and retried and stuff like that. Now,

three API calls in demo environment is

fine, but in production, when you get

thousands of queries from users every

day, and there's like duplication in API

calls, that's an expensive operation.

And that's where you need to think about

behavioral evaluation. And this layer is

very, very important. I see a lot of

organizations, a lot of teams miss them

when when talking about this.

The second layer is observability,

right? Uh and in this pillar, what we're

talking about tracing, right? So, you

collect [snorts] all the decisions that

an agent is making. So, I want to

explain this with a scenario here,

right? And this is a scenario from an

actual project I worked on with a

banking uh retail retail banking

chatbot. Now, obviously, if you've seen

tracing data, it's not as beautiful as

this slide, right? So, I've simplified

it and made it beautiful for this slide.

But what this slide says is basically, a

user comes in and says, uh "You know, I

have been charged an overdraft fee, can

you waive it for me?" Because the user

thinks that the customer thinks that

that is not legitimate.

So, the agent does an intent

classification,

and you all you know about this because

you've enabled observability, you're

capturing traces, and you're actually

seeing what the agent is doing, right?

What AI is doing.

Intent classification, it is done, this

it took this many seconds, this was

confidence score. Then it goes and

connects to the customer's account,

maybe in a database, a customer

database, call calls an API, connects to

the customer database, gets the account

details.

It retrieves policy documents. It checks

from a rag vector database,

um what is uh what is the policy around

overdraft, right? Is what the customer

claiming is legitimate? So, it checks

for policy documents. Then it goes and

does a reasoning on what should be

uh

you know, responded to the customer, and

then it does some final guardrail

checks, and responds to the customer.

Now, if you did not set up a system that

helps you look visualize all of these

traces,

when the customer comes to you and

raises a dispute,

you have no way to check what the AI

did.

Right? You have nowhere to go, and you

end up saying that I don't have have any

idea. Let's Let's give the customer a

discount or something, and then make

them happy.

So, this is why you need this, and this

is why regulators are are are basically

mandating, because otherwise there's no

production system if you cannot do this

kind of stuff.

So, this is where um

you know, you you you detect this the

example that I gave around duplicate API

calls. This is where you start detecting

this stuff. So, when you when you enable

these traces, you can actually go and

see duplicate calls, and then take

relevant actions based on that. Not only

that, you can actually do that in online

monitoring. So, when it's happening in

production, in on you can set up online

monitoring, and at that point, if it is

doing duplicate calls, you can apply

fallback strategies. Or even if it is

doing a call that is failing, you can

actually go and apply a strategy where

it will say, "Okay, go and retry for

three times, not more than three times.

If it if it is more than three times,

then report somewhere, or pass it to a

human to take some action."

The third pillar is the most important

pillar, in my opinion, is the data data

data foundation, right? Uh

in my typical project projects, I spend

60% of my time, uh and I see I see a lot

of organizations spending a lot of time

here, because

no one expected agents to come suddenly

in the market and start querying data.

Data was always built for humans, and

humans are always forgiving.

You find the wrong data in a report, you

just go and ask someone to correct it.

Agents don't forgive you, right? Agents

will go, find it wrong, they'll give you

the wrong answer confidently.

Right? And you wouldn't know what's

happening.

And this is why data quality,

setting the right data strategy, has

become so important for enterprises now.

I divide it into two sections. One is

the question data, as I was explaining,

like data needed for actually serving

the AI's uh outcome.

And the other one is the tracking data.

This is the observability data, the

tracing data I was talking about

earlier.

You need a proper plan on how you

collect this tracing data, and how you

serve it to auditors, to regulators, to

do online monitoring, to run LLM as

judges on the tracing, and everything

else, right? So, there it needs a proper

strategy on how you structure the schema

and everything on the tracing data.

Um

On Databricks,

um we

create a robust data foundation for our

customers

using uh some of the technologies that

we provide. If you don't know

Databricks, Databricks has been built on

some open-source technologies like

Apache Spark, MLflow, and Delta Lake.

Uh we provide a bunch of capabilities on

top of it. So, the blue layer at the

bottom is basically your cloud storage.

Databricks works on the three major

clouds, Google, AWS, Azure.

Okay.

I thought it was for me.

So, so once you once you store raw data

on your cloud storage,

uh the data is then um

um we we we bring in a a layer called

the Delta Lake layer, which uh which

basically brings in database-like

properties on top of your raw data. So,

you have got images, text files, video

files, or whatever. We we help you

create this um

you know, uh table-like structure on top

of it using manifest files, right? And

and we help you to uh incrementally load

data, do all of those um data management

tasks in a structured way.

On top of that, we bring in Unity

Catalog, which is a data catalog. Uh

with Unity Catalog, you can centrally

apply permissions on top of the data.

You can

um you can uh share the data using uh

Delta Sharing, but also uh what happens

with Unity Catalog is uh you you can

enable discovery and um you know, um

ownership, metadata tagging capabilities

at the catalog level. What that means

is, when you apply table a description,

column description,

uh tag columns uh like PII columns with

metadata, it becomes really easy for AI

to then get that context when it queries

these tables on top of Unity Catalog.

So, everything is governed at one layer

through Unity Catalog, and on on top of

that we bring in different uh

applications. So, whether it's AI

through Mosaic AI, so to build LLM, tune

LLM, or even build AI applications, we

bring in uh data warehousing

capabilities, BI capabilities, and um

uh and some of the other text-to-SQL

capabilities. We have got Genie that uh

helps you write natural language to do

SQL querying, etc.

And one application of that in the

observability and tracking tracking

data, as I was showing, is is this. So,

basically, think about when I was

talking about the tracking data

strategy.

Organizations, especially enterprises,

will not be running AI in just one

framework. They'll be using different

frameworks,

CrewAI, LangChain, etc. etc. They'll be

using different cloud platforms.

And once they do that, you need a

centralized layer of collecting that

tracing data, so that you can serve sev-

several use cases on the right hand

side. So, whether it's for operational

dashboarding, for first line support, uh

a lot of these uh first line

um first line of defense teams need

health monitoring uh sort of dashboards,

right? These teams can also write SQL

using Databricks Genie to do

text-to-SQL. But they can also build

Databricks apps using coding agents uh

to create common workspaces or custom uh

UIs that customers might need for

different uh different use cases.

And then we've got Agent Bricks and

MLflow that serves you uh LLM out of the

box LLM as judges,

and uh proactively monitor a

The idea is,

no no no matter where your AI runs, you

can create this kind of strategy

bringing in data in one common place and

serving uh different teams from one

shared location.

The fourth pillar is multi-agent

orchestration patterns. As I said, one

agent is good, multiple agents increases

complexity. That's where you start

thinking about, okay, what pattern is

good for my use case. The first one I

describe here is the orchestrator worker

pattern. Where you have one orchestrator

which orchestrates all the work, which

controls all the work from a centralized

plane, and then distributes this work to

different agents based on their

specialized skills.

And then every request goes through the

orchestrator, so you have got central

control. If something goes wrong, you

can go to the orchestrator logs and look

into them and see what has happened.

Right? So, that's the orchestration data

uh pattern.

There is this choreography pattern where

each agent is independent, they're

autonomous, they don't depend on an

orchestrator. All of them talk to a

message bus

and they listen to the events that they

are interested in.

Right? So, think about agents that are

independent of each other, right? They

can run parallelly. So, they are not

sequential, like one agent is not

dependent on another. So, they run

parallelly, they listen to the message

bus for the

for the events that they are interested

in. Maybe it's a trigger for, let's say,

a mortgage application, and it says, uh

you know, uh the mortgage application

agent

uh one of the agent uh looks customer

details, right? The other agent looks at

approval details and everything else,

right? They can work in parallel, and

the advantage it brings you is the

latency is reduced because they are not

dependent on an orchestrator and sending

messages back and forth. Right? So, this

is the choreography pattern. And the

third one is human in the loop, which is

where when an agent crosses a threshold

or serves below threshold a confidence

threshold, then a human is called in the

workflow to look into the pattern uh so,

look into the looking into what the

agent has done and then take action

based on that.

I have

done a deep dive video on multi-agent

orchestration pattern

uh for the online track of this

conference. Uh it's already on YouTube,

so you can look into it. I talk about

the real implications of when you think

about multi-agent patterns.

One is uh state management, the other is

fault tolerance, like what happens when

things fail, like how do you manage

them? I talk about different patterns.

And then talk about how how you think

about scaling them in large scale on

enterprises.

Pillar five is governance, right? Now,

here I'm not talking about data

governance at all. That's given, we need

that, right? From AI perspective, what

what what are we thinking about?

Regulatory, right? Audit trails, have we

got the trail of every action, every

user connection, every request,

everything that happens in the system?

Are we capturing everything?

Are we doing pre-validation of personal

information? Are we using name entity

recognition? The the easy stuff, the

rejects and all of those things, right?

In our example, the work that I was

doing with the customer that I

mentioned, we already detected 47 PII

breaches during the testing phase by

applying this layer. So, that's that's

really important.

Fourth is um prompt versioning. You have

to treat prompt versioning as change

management in enterprise grade solution.

It cannot be just change to a prompt and

commit to get. It has to be is it has to

go through proper change management

processes as you do with code. So,

basically treating prompt as code.

Third is model change management. So, as

models change, the model providers

upgrade these models, you have to have a

system to understand whether that

upgraded model will be good for your use

case, for your data. Right? Model

providers you will put evaluation

benchmarks on three uh benchmark uh

boards,

but those are not really useful when you

put them in your context, in your

enterprise. So, that's where these

evaluation data sets come in handy,

where you try these different models on

this evaluation data set and try to

understand which one performs better.

And that management needs to be done

because

from a risk perspective, you cannot

really rely on one single model. You

have to have the flexibility to switch

to different models and also test them

on your own data.

That management needs to be done.

Uh in Databricks, uh we have taken all

of these these points, these pillars

that I've been talking about into Agent

Bricks. We are building Agent Bricks to

make uh all of these operations out of

the box for you,

uh so that it's easy to implement

production grade AI applications on uh

on in in your enterprises.

So, I wanted to quickly touch upon a

case study, just to give you a uh a

flavor of how these things go, right?

So, when I was working with this client

um

they were a retail banking they were

building a retail banking chatbot you

know, one and a half 18 months ago.

Uh

their their problem the the problem they

wanted to solve is they had got around

20,000 odd calls per month from

customers on their chatbot. They wanted

to deflect they they they saw that there

were like 60% of them were simple

queries, what is my account balance, you

know, what do I do with my overdraft and

all of those stuff, like that can be

answered simply. So, they wanted to the

reliance on human agents for those

answers. So,

they identified those queries and they

wanted to automate them. Right?

They spent around 85K in 6 months doing

a POC which did not succeed. When we got

involved, we found those insights that I

was talking like no one knew why things

were failing when it was in production

when when when when it was

in production. No one could actually

measure

why

why it's not succeeding and no one could

actually understand who is accountable

for what when things go wrong. Right?

So, the goal we set for them is AI agent

handles 60% of user queries, right?

Which were simple user queries and then

a way to identify and track them. The

key difference in this project that we

when we did is that we selected the

model in week seven, like in a eight

weeks POC.

Right? And this is how it turned out.

For the week one and two, we built the

evaluation layer. We collected 200

cases on their actual human agents

answering to their customers on simple

queries and understand how they are

responding to them.

We created that database.

Then we defined the success metrics.

What does success look like to you? So,

out of let's say 100 queries, you need

60 queries or the 60% of the queries

that are simple queries

to be

uh to be handled by the agent, right?

They needed some sort of accuracy. So,

85% it was around 85% accuracy target.

They needed latency, all of the

operational targets that you need. They

were there.

Then we created this automated

evaluation pipeline for them. And what

what I mean by that is an automated

system where you can capture a user's uh

a user's question and the AI agent's

response. You take that, compare that

against your evaluation data set.

You rate that, and if the rating is

below certain threshold, you get it

checked by a human. And you if if

something goes wrong, you make sure that

you find the solution. So, it could be a

change to the prompt, it could be change

to a tool calling system or something

else. Once you have done that, you add

that test case in the test data set. So,

that when it happens next time, the test

cases cases catch them. So, the the the

the summary of that story is that your

evaluation data set is a living system.

You start with 200, maybe there is no

correct number here, but once you start,

as you start building in production,

this is a living system. This will keep

growing. And the

and the bigger it grows, the better your

system will be.

In the second week, we talked we thought

about the foundational layer, right? So,

the the question data, we thought we

thought that, okay,

if you have to call the database, have

you got the API connections right? Have

you got a system in place that can trace

the API connections? Are those secure,

right? We were not talking about MCP at

that time.

Right? It was just direct API calls to

database to run queries.

Have you got the distributed storage?

Have you got the Have you Are you

collecting traces?

And this is where when when we started

testing after building these systems, we

could catch those duplicate API calls,

right? We could catch why customer

satisfaction was dropping and stuff like

that.

And then comes in week seven to eight,

we started talking about models. Now

that we had the evaluation data set, we

could run different models on that data

set to see the responses, compare them

against the expected responses, and

calculate a number on on on the

accuracy, right? That helped us to

decide which model to use.

Now, that decision didn't take long,

right? We in as I explained in the

introduction, like we spent weeks

debating on which model to use,

but when you took the other approach,

you can actually

uh do that in a very quick way.

So,

once once that's done, we we stitched

everything that I was talking about

around observability, evaluation, the

layers of evaluation. Once we had that

system that can make AI visible,

measurable, and accountable, that's when

we started launching it to production.

And that's when uh so, this is this is

the result uh six weeks post launch,

uh we we calculated the operational

metrics, of course, you know, the

accuracy, the deflection rate, the

response time, uh the customer CSAT. But

what's important here is

in few weeks time, when uh there was a

problem with uh so, one of the one of

the things that happened was that the

bank changed some uh

interest rate related policies. So, when

that when they changed the policy, they

actually sent emails to customers or

notifications in the application in the

in the in the mobile banking app about

the policy change.

But when the customers came and queried

on the chatbot for further questions,

they couldn't get the right answers and

they were like

putting thumbs down on the answers. So

they were getting this feedback, right?

So feedback decreased.

The problem with this kind of system, if

you did not have this measurement

system, is that you couldn't actually

know what's happening.

But because we had the measurement

system in place,

these

the drop in C set was detected, right?

Because we were getting negative

feedback from customers.

We could actually look into the tracing

decisions and see that the agent was

looking at a policy policy document that

was outdated. So the the new policy

document was not updated in the vector

database. The embeddings did not come

through.

Because it did not come through, it was

giving it stale answers. And that's when

we went and fixed that. But it it was

all possible because we built that those

systems

that

that led us

to to detect this.

Before you go, I I generally in these

sessions I share different

artifacts that you can take away. I have

a QR code at the end for you to download

and you will find multiple artifacts.

One of the important artifacts that I

want to talk about is the production

incident playbook. This is something

that a lot of us tend to miss when we

work in AI projects. And this playbook

is basically a definition of what needs

to happen when things fail in

production.

First, you detect using your eval

dashboard.

Then you diagnose using your tracing as

I explained.

Then you contain. So basically, you you

are versioning your prompts. Is there a

Is there is a If there is a problem with

the prompt, you you take that prompt

out, right? And start the changes,

deflect it to a human. Or in my

multi-agent orchestration video, I've

talked about multiple fault tolerance

failure recovery patterns

around saga pattern, compensation

pattern,

and circuit breaker pattern that you can

look into the video. I've explained them

in details on how you can handle them.

And then you use the test case library

to fix. So you look into LLM's judge

reports, you look into your evaluation

data set reports.

Then you fix your problem. Once you fix

your problem, you put those test cases

in your data set, right? And and create

that eval suite that is a living system

that will keep growing.

And

and and you you keep improving your AI

system based on that, right? But this

playbook needs to be in place. When it

runs in production, you will need to

integrate it with your ITSM system so

that it alerts the right person at the

right time. You know, a lot of these

organizations would have existing

ITSM systems, right? So which

which is used for alerting and you know,

making sure that the downstream systems

don't get affected, etc.

So

once once you have this in place, you

can go and stitch it together to other

systems.

So what can you do tomorrow, right?

Start with If If you have a project in

mind, start with defining success.

Success not from the technical sense,

from the business sense. What it means

means for the business, right? Come up

with a few examples of what good answers

look like.

And and create a data data set of that.

And then build that pipeline using

simple Python code. See if you can

automate that so that when you run AI

and get some response, it can become it

can go and compare You can go and

compare the answer against that data set

and then that can be delivered

to to the to the customer.

Now, these are three lessons that I have

learned while doing these things with my

you you know, my easily miss.

The test case library, as I explained,

is a growing system. It will grow over

time.

And because it grows over time, you need

some sort of governance around it. You

need a owner, right? You need to You

need to figure out which test cases

relate to what kind of problem. So that

whenever you go back to it, you can

you can relate your answers to those

sort of problems, right? If it is a

security If it is login, so you can say

that the agent did not ask for login

credentials when the customer asked the

answer. And all those kind of issues can

be put under a security category within

that data set. So to categorize the rows

in your data set so that you can pick up

what changed and compare it with them.

The second is prompt versioning. Now,

when you start versioning prompts using

Git, you know, we all know when you put

Git message commit messages

tend to be

simple commit messages. But you have to

put governance around what kind of

commit messages you are putting in when

you're changing these prompts because

you need to understand when a prompt was

changed, for exact what reason it was

changed, right? What was the failure

that caused this prompt to be changed?

What kind of failure would it address

and what would it correct, right? In the

next version. That needs to be

documented.

Otherwise, it becomes difficult because

when you go back and look into prompt

versioning and look at different

versions and you cannot trace why why

those changes were made, then it becomes

difficult to track what's happening.

The third, the layer three evals, right?

So the behavioral evals that I was

talking about around tool calls and

stuff like that, they can be really

expensive as you grow your eval data set

as well. So when you have a wrong tool

call, for example, and you want to

correct that system, when you correct it

and run it against the eval data set,

you have to basically run it against

let's say if you have got 300, 400, 500

rows in the data set, you have to run it

against them. And you do all the testing

again and again and again and again and

again,

that can cost you a lot of money. So you

have to put some governance around that.

So for example, when in your continuous

integration pipeline, when you do the

prompt

change,

you can actually put

some checks around

just just selecting a small subset of

the eval data set to do the testing. And

you only do the full test when you merge

to the main branch.

So you can put these kind of decisions

in place so that you can reduce cost

around

around

you know, expensive eval decision.

If you scan this QR code, it'll take you

to a Google Drive link where I have put

some examples on

some of these how these templates look

like, what evaluation checklist should

look like.

I've given you some guide on

set setting up tracing

with open source technologies

so that you can quickly set up some

tracing and start testing in the test

environment before you

decide on what kind of tools you want to

use.

Thank you very much for listening to me.

This is This QR code will take you to my

LinkedIn profile. So I share

I have a newsletter where I share this

kind of topics every week. So if you're

interested, you can join. It's free.

I basically share what I learn in in the

field working with customers, right? So

it might be useful for you.

Thank you very much.

>> [applause]