AI Writes the Code. So What's Left for Engineers? - Laurie Voss, Arize AI

[music]

>> My name is Laurie. I'm head of developer

relations at Rise AI.

Uh in a former life I co-founded NPM

Inc. So some of may you some of you may

remember me from JavaScript days.

Uh but these days I think about AI

agents and uh in this case what they are

doing to our jobs.

Uh so a little bit over 3 years ago um

uh Swix wrote an essay called the rise

of the AI engineer and his argument was

that when it comes to shipping AI

products you want engineers not

researchers. I think that is a message

that resonated with everybody especially

who the people who come to this

conference. Uh and so he created the job

title of the AI engineer and companies

since then have been hiring for it. Um

and part of what I'm here to do today to

do is to convince you

uh that he did not in fact go far

enough. Uh

there's not just an AI engineer

revolution happening. There is a

revolution happening to every kind of

engineer.

Um all engineering jobs are being

replaced by a new type of job and not

everybody is going to be happy to get

that news.

Um

the underlying change that kicked all of

this off is that agents have made

writing code basically free. Like you

pay for the tokens but relative to the

cost of the time it used to take you uh

it is basically free. Um

everything that comes after writing the

code did not become free at the same

time. So the the bottleneck moved from

writing code is really slow and all of

this other stuff happens at roughly the

same speed to writing code is super fast

uh and now we get stuck behind new

things. So the first the thing it's

stuck behind right this second like this

month is code review.

Um

uh but my thesis is that it's not going

to stop there. We are going to figure

out how to automate code review. We are

going to make uh code review move at the

speed of agents, and then we are going

to run into another bottleneck, and

another bottleneck, and another

bottleneck, and at some point we're

going to stop. We're going to hit one

that we can't get past. Uh and that is

what is going to be left for the humans

to do. Whatever the bottleneck is that

we can't get agents past. So, I'm making

a bet that I

can think of a thing that agents really

are not going to be able to do. Uh

and then once we've come to that

bottleneck, we're going to have to

figure out how do we train the next

generation of engineers to be able to do

that job because we're not training them

to do that job right now.

So, a quick warning {slash} apology. I

have 18 minutes because that is how long

Swiss gives for talks.

Uh

and I have a really big pile of data. I

did a ton of research for this. Uh and I

don't have time to give it all to you. I

don't have time to like quote my sources

and do all that stuff. So, I'm just

going to throw numbers at you with a

bunch of citations on the slides, and

I'm just going to have to sort of ask

you to trust me, bro.

Uh

>> [laughter]

>> Uh so, developers who switch on

autonomous coding agents write 741%

more code, eight times as much. Uh and

but they ship only 30% faster. Like, 30%

faster is not nothing. 30% faster is

amazing. We are shipping a lot more code

than we used to, but given that we're

writing eight times more code, only a

third more actual shipping is kind of

bad, right? It means that we're stuck

somewhere.

Uh and where we're stuck is code review.

Um and the reason that we're stuck is

because humans can't just review faster.

Uh there is an enormous study done by

Cisco 20 years ago about how fast you

can do code review, and how accurately

you can do code review. Um

and it turns out that humans' ability to

like parse what's going on in code falls

off radically after about 450 lines of

code per hour. So, if you've produced a

10,000 line diff with your agent, it

would take you like a week to reasonably

review that code. And the answer is that

nobody's actually doing that, right? Cuz

we're all throwing 10,000 diffs around

willy-nilly these days.

Um so, I did a whole talk about this

phenomenon at a previous AIE, and that

talk is called the death of the code

review. And if you want all of the

information about the death of the code

review, you can watch that talk.

Um but the problem facing us right now,

uh which is not the same as the problem

I'm trying to solve with this overall

with this talk, is that the cost of

producing plausible code has collapsed

while the cost of knowing whether to

trust that code has stayed exactly the

same.

Um so, the obvious review move right now

is to make the machines do the review,

like I said. And that is already

happening. Um GitHub's Copilot uh does

more than one in five of the code

reviews on GitHub as of last month. Uh

you can, if you want to, opt in to let

uh

Copilot be the final reviewer before

something goes to

uh production. You can make it have the

approval power, if you want to, which is

crazy. I don't know who's turning on

that option, but you can do it if you

want to.

Um OpenAI says that at at OpenAI, humans

may review their pull requests, but

aren't required to.

Um

but the problem is that the machine

reviewers aren't very good yet. Uh the

data says that there's

uh the data that I found

put four AI reviewers through their

paces, and they all found a bunch of

bugs, but none of the same bugs. They

did not agree on any individual bug.

Uh

they all found actual issues in the

code. They all found different issues in

the code.

Um so, they're worth running, but if you

are serious about this, you're never

going to run just one of them. Uh

because they are not reliable enough

yet.

Um

And there are some people who

went all the way. They were like, "I'm

just not going to review the code

anymore. I'm just going to assume my

agent has one shot at this perfectly."

Um Dexter Horty, who is here, I don't

know if he's in the audience, probably

not. Um

uh he spent 6 months telling people not

to read the code, and then after that 6

months, he got on stage and said, "Nope,

I was wrong. You can't actually

uh skip code review entirely. We tried

that. It was a disaster. We had to rip

replace whole systems." Uh

that's not an option. The agents are not

good enough. They're not one shotting

the code.

Um Amazon had a run of outages uh that

they did postmortems on and decided that

it was because their coding agent had

produced a bunch of slop and they just

released it into the wild without

thinking.

Uh and so now they don't let them do

that anymore. Uh senior engineers must

sign off on code reviews at Amazon. Um

and Anthropic published a postmortem

uh on bugs that had got past every

reviewer that they had, both human and

machine. So, the problem is that the

automations that we've built so far to

check whether our code is good enough

are not themselves good enough. Um

Claude Fable 5 passes the tests

uh on 80% of a standard coding

benchmark. Um and co- So, Cognition, the

makers of Devin, um they built a

benchmark that asks a different

question, which is not does it pass the

tests? The question is would you as the

reviewer, as the maintainer of this

repo, merge this PR? And if you ask that

question instead, the pass rate falls

from 87% uh to 54%.

Um

because there's a lot of extra

information that goes into deciding

whether you would merge it. There's code

style, there's side effects, there's

external systems, there's history that

is not encoded into the code base. It is

encoded into the brain of the

maintainer.

Um

So, what the maintainer is adding, what

maintainer is demonstrably adding, is

something that we haven't given the

agents yet. There is information they

don't have, there is a skill they don't

have. Um and it is the sense of what

good means. That is what we are That is

what the agents don't have yet. Uh and

that is a critical insight that I'm

going to come back to.

Um

Eliyahu Goldratt, if I'm pronouncing

that correctly, was a physicist who

became a management guru and in 1984 he

wrote a a novel about running a factory.

Does that sound like an interesting

novel to you? Um it is called The Goal.

Um and his big idea was the theory of

constraints. Uh every system has one

bottleneck and speeding things up does

not work unless the thing that you were

speeding up is the bottleneck.

Uh

that is our problem with code review. Uh

we've sped up the code and we have not

sped up the review.

Um his second idea is the one that

matters today, which is that when you

fix the bottleneck, it doesn't go away,

it just moves and finds another

bottleneck to pile up against. Um so

let's assume, despite everything that

I've said about how bad the code

reviewers are right now, that we figured

that out, that we get automated code

review to a higher standard and we move

that bottleneck. What is the next

bottleneck uh that it is going to run

into?

Um

if we figure out review, the next

bottleneck is probably going to be

operations. How do I deploy this thing?

How do I scale it up? Uh and then you're

probably going to run into security. How

do I make this thing safe at scale?

Uh how do I make sure that this thing is

private? How do I make sure that it's

not radically going off the rails? Uh

but I'm going to make and this is the

big leap that I'm making in this talk.

I'm going to assume we figured those

things out as well because those things

are very heavily solved by humans.

There's an enormous amount of data out

there about them. If we can figure out

how to do operations, how to do scaling,

how to do security using progressively

more advanced agents, what is it that we

are left to do? Like, what is the role

of a human in that situation?

Um

it's also it's a big question

[clears throat] and it's also a

long-term question. I'm not telling you

what is going to happen in the next 12

months. In the next 12 months, I think

we'll probably make some real progress

on code reviews. I do not think 12

months from now we'll have sorted out

operations and scaling and security. I

think that is going to take somewhere

between 5 and 10 years to get done.

Uh despite people who are like, "AGI

2027." No, we're going to slowly iterate

on this stuff.

Um

But the other assumption that you need

to make for my for the my thesis to be

correct is that the amount of software

that we need is effectively infinite.

And I feel that one is a much easier

case to make. Think of every website

you've ever run into

uh in the world in production, and it's

probably pretty bad. Uh and the reason

it's bad is not because we don't know

how to make really great websites. The

problem is that there's not enough

people to make all of the websites good.

Every time you run into a terrible

website at your hairdresser or the

government or like, you know, your bank,

like it's not because there's not

somebody who knows how to make that

website good. It's because A, it's very

expensive to make it good, and B, the

people who are doing that are busy doing

other things.

Uh you know, probably shipping AI

agents. Um

so assume that we can make as much

software as we want basically for free,

and that the appetite for making

software is essentially infinite. That

is a wonderful situation in which to be

in. We've got infinite demand and also

suddenly infinite supply.

Uh where does the bottleneck end up?

What do we think the agents are not

going to eat? Um to try and figure that

out, I split up the job of software

development into as many pieces as I

could think of, and I sorted them by how

soon the agents uh get to them.

Uh writing code is already collapsed,

reviewing and maintaining code is going

soon.

Um

shipping to production and scaling are

next on the chopping block, although

there isn't a lot of data there yet. Uh

and then there's the stuff that I think

is probably safe, which is deciding what

to build,

uh

defining what good means, and finally

making it delightful.

What's left is finding out what people

actually want. Every piece of software

is a formalization of something that a

human wanted to get done.

Uh

So, if a customer tells you, "I need to

to keep track of my orders," there are

10,000 pieces of software that

keep track of your orders. That's what

they do.

Uh but only one of those, if you're

lucky, is good if the thing that you're

doing is you're running a bakery. If

you're running like an auto parts store,

that's going to be different. If you are

running

uh you know, a

giant manufacturing concern, it's going

to be completely different. The That

requirement is not sufficient. The whole

job is figuring out what exactly do you

mean when you say that you want to keep

track of your orders. What does that

mean for your business? What does it

have to do? How does that work?

Uh and that knowledge isn't in the

training data. It isn't in the weights.

It is in the mind of the person who runs

that business. And the only way to get

it out is by flapping your meat parts at

them and having them flap their meat

parts back. Like, you have to talk to

them. That is the only way to get the

information out.

Um

And that work has a very important

property, which is that it doesn't

transfer. Uh if you think about what

good means for like a calendar app and

what good means for a scheduling app,

those are solving almost the same

problem, right? A calendar app and a

scheduling [clears throat] app, but

they're completely different pieces of

software. They have totally different

requirements and a totally different

definition of good, even though

[clears throat] those two use cases are

so close together, and we have a zillion

use cases.

Um

Anytime that you've used software that

has a zillion configuration options and

it still doesn't [snorts] do what you

need. I'm looking at you, Jira. Uh

like

that is the problem you're running into

of software that is trying to do

everything and still failing at it.

Um

so the bottleneck moves to describing

the problem and defining what it means

to have solved the problem and my big

bet for this talk and for the next 10

years is that is where it stays. Uh Fred

Brooks, who ran IBM System 360 project

and wrote uh The Mythical Man-Month, uh

he wrote about this in 1986. He split

the difficulty of software into two

kinds. Accidental complexity is the

difficulty of the tools or the

languages, the typing, the syntax, and

then there's the essential complexity,

which is the prob- difficulty of the

problem itself.

Uh and he predicted that no single tool

would make us 10 times more productive

because the essential part is the hard

part. Uh

and Brooks's prediction is looking

pretty solid because we made code eight

times faster and we still didn't get

very much faster at shipping because the

hard part is the essential part.

Um Shawn Grove from OpenAI made the same

point uh 2 years ago. Uh no, sorry, a

year ago. Uh what year is it? Um

he said that code is a lossy projection

of the specification and whoever writes

the spec, if it's a PM or a lawmaker or

an engineer or a marketer, they become

the programmer. Uh

and so writing [clears throat] down what

good means is where the value is now. Uh

I'm beginning to run out of time because

I

haven't timed this well. Um

anything that you can verify cheaply,

you can train against until you beat it

if you are model maker. Uh so uh

Cognition tried to write down because of

their benchmark about what what it

counts to merge something, does it, you

know, is is this code mergeable? They

were like, let's see if we can figure

out a benchmark that defines what

mergeable really means. Let's see if we

can get all of that other information

into the benchmark, and then we can

train a model on it. Um

uh

Cursor's reviewer BugBot turns every

thumbs down that you that your team

gives it into a rule, and they have like

44,000 rules now. Uh

so the moment the maintainer takes

knowledge out of their brain and puts it

somewhere written down that a model can

train on it, [laughter]

uh the bottleneck moves again.

That is why I think security can go.

That is why I think operations can go. I

think we can write those things down

eventually, even though we haven't done

it yet. Uh

Andrew Ng, the co-founder of Google

Brain and Coursera Coursera, says that

one of his team proposed having twice as

many product managers as engineers, uh

and he said that that was a first in his

life, uh but his answer is since then

hasn't been hire more PMs. It's to blur

the jobs together. In his words, when an

engineer understands users and can make

decisions on what to build and build it

directly, uh they can execute incredibly

quickly, and that is where I think we

are beginning to hit on the answer.

Uh in the 1960s, there there was a job

called the systems analyst. Their job

was not to write code. Their job was to

talk to the business and find out what

the problem was.

Uh

then that analyst turned into something

that we call a product manager today,

and we've had two professions uh where

there used to be one and a half. I think

what is happening is they are collapsing

back into one. We are going back to the

'60s. So that is not an AI engineer.

They're not just building AI. They're

not just building with AI. They are

building everything. They are building

all software.

Uh and I've decided to call this job the

product engineer. This is somebody who

finds out what people need, defines what

good looks like, and ships it uh with

agents doing the typing

um

Gergo Orosz whose name I have probably

butchered he writes the pragmatic

engineer newsletter he described the

product-minded engineer back in 2019 as

a nice trait for an engineer to have and

what I'm saying now is it has gone from

nice to essential. It has gone from the

from

a skill that senior engineers used to

have to a skill that every engineer has

to have if they're going to be an

engineer.

um

And you can see the market already

paying for this. Forward deployed

engineers who look very much like

product engineers

job postings for them rose 800% in 9

months of 2025. They are everywhere

everybody is trying to hire them. Who in

this room is trying to hire a forward

deployed engineer show of hands.

Okay, well more I was expecting more

hands oh well. uh

So

uh The average comp for forward deployed

forward deployed engineer by the way is

$240,000

which is to say that it pays like a

senior engineer because it is a senior

engineer. It's a very useful kind of

engineer. um

But nobody agrees on what to call it

yet. There are like a thousand different

job titles for this thing that I am

describing and have decided arbitrarily

to call the product engineer.

um

And probably at this point some of you

are thinking that isn't programming and

you are correct. It is not programming.

The programming is being done by an

agent. It is however recognizably still

software development and we are going to

have to wrap our minds around the idea

that being a software developer and

being a programmer have become divorced

in 2026.

um

And

when I say stuff like well I think the

hard part is deciding what good is I

realize that I'm falling into a trap

that lots of people in AI land fall

into, which is all of the parts

everything about everything can be

automated by AI except the thing that

I'm good at because that's too hard. I

heard a VC say that the other day and I

was like, "Buddy,

uh

like, no. No, someone is going to

automate your job any day now. Um

uh so

you have to take this with a grain of

salt. Am I Am I just thinking that the

thing that I enjoy about software

development, which is figuring out what

the problem is, is the hard part of

software? You can tell me if I'm wrong.

Um

but say you believe me so far.

Programming goes to zero, product

engineering goes to infinity. That

brings us to a new and important

problem, which is junior developers

because the way we used to train a

junior developer is we used to give them

very well-specified tickets that said,

"Write this software this way in

particular and then give it to me so

that I can tell you all the ways that

you were wrong."

Uh

and that is gone because that's what we

do with the That's what we do with the

coding agents now. We say, "Write this

thing and then I'll tell you how you're

wrong." And then we have a conversation

with them. Uh and there's That is not a

job that we need a human to do anymore.

But we still need an enormous number of

forward-deployed engineers or uh product

engineers and we have to get them from

somewhere. We can't just abandon the

current generation of new developers

who've showed up and said, "How do I do

this stuff?" We're like, "I don't know

because I don't know what your job is

anymore." Uh

>> [snorts]

>> this is not a theoretical future. This

is an actual graph of labor statistics

since January 2023. It is sorted by age.

The red line is engineers aged 22 to 25.

Their employment is collapsing. We are

not hiring junior engineers. This is not

a thing that I think is going to happen.

This is a thing that's been happening

for 2 years. Job market sucks for them.

Meanwhile, the job for older engineers

41 to 49 has gone way up because senior

engineers are still super valuable, more

valuable than before, but junior

engineers are being screwed and we need

to fix that as a profession.

Uh so if we're going to reap the

benefits of a world where we 10x code

output and figured out the rest of the

software supply chain and figured out

what humans need to do, uh

then we are going to need a lot of

people to do that. I said that the

demand for the software is going to

infinity. We need infinity people. Where

are we going to get them?

Uh we've been creating them accidentally

so far and we need to figure out a way

to do them on purpose. Google has a

product uh has a program called the

associate product manager program which

turns out people that look very much

like product engineers. They turn out 50

of them per year. We need 10,000 of them

per year. Even if you look at all of the

other pro all of the other companies

that have something a program like

Google's APM, it's being extensively

copied. They're only turning out about a

thousand people a year and we need 10

times that number at minimum uh to keep

up with demand. Uh

>> [snorts]

>> universities teach data structures, boot

camps teach React, nobody teaches go sit

with a baker for a week and come back

with a spec and that is the person that

we need.

Um so if you lead an engineering team,

you're going to have to grow them on

purpose.

Uh IBM is already redesigning its

entry-level role around talking to

customers and writing specs instead of

programming. Um you're going to have to

figure out how to do that for yourself.

You're going to have to figure out how

do you train product people? Uh and

you're going to have to do it very

quickly because the time we should have

started was two years ago and there's a

whole cohort of very sad unemployed

juniors right now who have this problem

and we are not hiring them and we need

to do something about that or we are

going to be screwed and not just them.

Uh

so if all of that happens, there's one

big thing that I don't want to just

gloss over or hand wave away, which is a

lot of people got into programming

because they love the craft. They love a

clean abstraction, the feeling of a hard

bug finally giving up, making a machine

do exactly what you told it to do. They

did not sign up to interview bakers

about what they want in their lives.

Like that is not the job that they

signed up for. Um

and they are going to hate this. They

already hate this. They are complaining

about it very loudly. I have no

consolation prize for them. Um

but uh I do have an observation, which

is that for a lot of people who think

they love the typing, maybe what you

loved was the moment just before you

started typing, when you figured out the

problem.

That That part doesn't go away. That

part is still there. That creative spark

when you're like, I know how to make

this into a repeatable system that glows

in my brain. That's not going away.

Uh

and the craft of programming will

survive uh in the same way that like

woodworking survived the furniture

factory. Like programmers program the

way that singers sing, right? They do it

in an empty room with nobody watching

just because they love it and that won't

go away. You can still do that. Um

it just won't be the thing that people

pay you hundreds of thousands of dollars

every year to do.

Um so here is my prediction. The next 10

years are going to be ugly. The last 2

years have been ugly. Um

the load of this stuff is arriving

before the tools are ready. The junior

ladder is gone before we have figured

out how to train more juniors

uh and a lot of people who love the

craft are going to have to decide if

they love the job that is replacing it.

Um but this has happened before.

Compilers and then frameworks and then

open source each made a generations

worth of typing before them

uh unnecessary and each time we came out

of it with more software developers. We

just changed what software development

meant.

Uh and I think that's going to happen

again.

Uh, in 10 years time we will get a

profession that is bigger than today's,

pays about as well, and is mostly shaped

like product engineering. That is the

case that I'm making for you today.

But that is a whole lot of high-level

predictions, and I hate those. What does

this mean for real? What should you

actually do with this today?

Uh,

first, be prepared for a world in which

humans are barely involved in the

production of code. So, write down your

high blast radius code, and make that be

the only thing that you are reviewing

anymore, because the rest of it your

automatic reviewers are going to get

done.

Uh,

then delegate that everything that's

cheap to verify. Spend a lot of time

thinking about how would I build an

automatic way of telling whether this

thing, whatever that thing is, is good

or not, and then feed that to your agent

instead of trying to figure out the

review process.

Uh,

and finally, put your best people's

judgment into specs and rubrics and

evals. You know, I the arise guy was

going to say evals at some point. Uh,

get stuff that exists only in people's

heads into code. That is your job as a

product product engineer. It is turning

the sense of the problem into a spec of

the problem, so that the agents can

build it for us.

Uh,

and measure the right thing. 741% more

code was never the point. Measure how

much stuff you are actually shipping.

Uh, measure how often your reviewers

comments get acted on. Measure the gap

between the tests [clears throat] passed

and a maintainer would merge this.

Uh,

and remember that the last reviewer is

production. Watching what it does there

with another model grading it, that is

an eval. I'm not going to pitch arise,

that's just what we do, and that's why

we're so popular right now. Uh,

and I'll leave it at that. Um,

3 years ago, Swyx wrote about the rise

of the AI engineer, which is engineers

learning to build with models. And the

rise of the product engineer is about

what's left when the models do the

building.

Uh,

finding out what people need, writing

down what good means, and making it

something people like, that is the job

now. It is going to be the job for

everyone in this room, whether we like

it or not. We are all product engineers

now.

Thank you so much for your time and

attention.

>> [applause]