Focus Dario Amodei (Google Brain)

um so as uh as Bill mentioned I'm a

researcher on the uh Google brain team

and I'm also an adviser to the uh open

philanthropy project which uh which

thinks about uh social impacts about uh

AI so I you know I I mostly in the past

have worked on uh issues in uh uh you

know applying uh deep neural networks to

speech and vision but recently I've

become very interested in the issue of

safety and reliability of AI systems and

so I got together with a bunch of

collaborators some from Google some from

Stanford Berkeley open AI in the last

few months um and we recently released

this paper called concrete problems in

AI safety um which tries to bring some

kind of categorize and bring some

Precision to the uh to the uh to the to

the AI safety discussion so that's

that's what I'm be talking about here um

and so you know pretty much me and all

all the collaborators on this project

ultimately got interested in this

because of our experience working with

deep neural networks which we found over

the last um few years to you know to

have become you know very very powerful

and capable but you know also they can

be very opaque and can behave in very

unpredictable ways um and all all of us

kind of had the same thought that you

know as these systems continue to become

more powerful and be applied in more

kind of safety safety critical states

that you know this this is going to

become an increasingly important area so

we kind of Define in the paper the term

accidents as as what it is that we're um

worried about which I think is you know

the same as what everyone was talking

about today unintended and harm behavior

in machine Learning System um and you

know I I and my co-authors you know kind

of we mentioned in the paper that that

there are some trends that you know that

that we think are really going to raise

the importance of accidents and they've

been mentioned by some some previous

speakers today but first of all

reinforcement learning which allows

increasing intertwinement between the

agent and its environment all the

progress on Atari and on go uh complex

agents and environments which just

increase the range of strategy the

agents can have and the way the

environments can be disrupted increasing

autonomy which can take the humans out

of out of a loop and end to endend

systems which are often maybe a system

might be made up of many deep learning

components that talk to each other and

it makes it more difficult for a human

to understand what's going on and to

intervene in the system in some way in

some way that's helpful um so you know

we we really with these Trends we really

wanted to take kind of a forward-looking

approach to you know how to think about

the safety of AI systems and you know

there has been some some public

discussion of forward-looking approaches

but I think a lot of it has has focused

somewhat unhelpfully on these kind of

very extreme scenarios like AGI or super

intelligence and so um we wanted to take

an approach that was yes forward-looking

but also concrete empirical could be

worked on now could relate to systems

that you know that uh that we deal with

today um so one of the one of kind of

the main things we do we do in the paper

is try and provide a categorization of

the ways that accidents could happen and

actually I think you could have many

categorizations but you know we're we're

just kind of trying to provide one

framework among many for for thinking

about things um so what what happens

when when a machine Learning System does

does something you didn't want it to do

um one way to describe it is that the

designer had in mind a certain informal

notion of the object objective or task

that it was supposed to accomplish um

and the actual system failed at this

catastrophically so throughout the paper

we use the example of like a cleaning

robot whose task is to clean a room so

you know my my informal objective is it

should clean the room in a way that when

I look at the room at the end I should

you know I should be happy with what

it's done um and if the robot you know

like damages property or injures someone

or something like that then you know

that that was not what I had in mind in

my in my informal objective so we can

kind of classify where did where did

things go wrong so it could be that you

wrote down uh the the incorrect

objective function so you had an

informal objective function in your mind

you turned that into a formal objective

function and and no matter no matter how

perfect or omniscient your system was at

learning and inferring things it's just

the wrong objective function and it will

always do the wrong thing um a second

case is where the ex the objective

function is expensive we might know the

true objective function for instance we

could always consult humans and like ask

them to do a detailed evaluation of

whether every single action taken by the

agent is is a correct one but but you

know it's not practical to do that and

so we end up using the objective

function only you know only occasionally

and you know as a result we can end up

doing the wrong thing and the third

class is you have the right objective

function but something goes wrong during

the learning inference process um

there's kind of a fourth category which

I don't talk about but which I think is

is very important it's just kind of

different different topic from what I'm

talking about which is non-ml problems

that can arise as part of larger ml or

AI systems software implementations

verification issues security issues that

that that sort of stuff um so I'm kind

of going to go through the the the three

types and because this is a very very

brief talk I'll probably probably go

through my my slides pretty pretty

quickly but just want to give a flavor

of uh of of what it is that that we're

talking about out um most of this is is

posed mostly but not completely in terms

of reinforcement learning agents because

we see that as kind of a good Paradigm

but you know a lot of this stuff we we

imagine might might apply more broadly

um so uh under under the heading of

wrong objective functions um we're we we

think about kind of two ways two broad

ways that you can like easily get the

wrong objective function when you meant

to write down the right objective

function um one of those ways is what we

call side effects um which is that

systems that narrowly focus on one

aspect of an environment that could be

very large in the real world may end up

disrupting other parts of the

environment by default um and so you

know I'll exp I'll explain a bit a bit

that that that a bit more later but we

kind of have this running example of the

cleaning robot and the idea is that we

incentivize the clean robot to move a

box and if there's a vas vase in its

path and we haven't thought carefully

about the vase it'll just knock over the

vase while it's moving the box and we

probably didn't want that uh reward

hacking has been mentioned in some of

the previous talks uh this occurs when

an objective function can be gained

where where there's a cheap way to

achieve the formal objective function

without achieving the the ultimate

intent of the informal objective

function so example is if we choose to

reward our cleaning robot for not seeing

dirt it might close its eyes instead of

cleaning because it just sees no evil

hears no evil there's no dirt um so uh

little little bit more on on on side

effects to think a little bit more

formally about it you know we might have

an environment where we're trying to do

something like you know incentivize this

to move the box so we might naively

write down an objective function that

maximizes something that refers to only

this variable in the environment um and

when we do that we're implicitly saying

that we don't care about this other

variable in the environment namely

namely the vase um what we really wanted

was some complex objective function

which I'm calling fa aware which says

like move the Box while respecting some

like Common Sense constraints about like

the other parts of the environment that

are like really hard for me to write

down um and you know in this particular

case well I can just take take into

account the vase but you know what what

if there are 10,000 of these variables

and in particular what if the variables

I don't care about have more commonality

across goals and agents than than the

variables that I do care about that

suggests that maybe I can take a kind of

uh in integrated approach to this um so

here are just quickly a few of the ideas

one is what we call learning a side

effect regularizer we can think of this

as maybe a transfer learning uh uh

problem where um you know I learn in one

environment and one thing I learn is how

to accomplish the goal and the other

thing I learn is how not to disrupt the

environment and maybe the information

about how not to disrupt the environment

is more trans transferable could be a

separate you know neural net or

something else module that that can be

applied to other environments without

much or any change even when I have to

entirely relearn the goal just just just

as we do for like in modelbased

reinforcement learning when we when we

when we uh repurpose the state state

transition model but but maybe not the

uh the value function um there are

information theoretic measures um a

little bit a little bit obscure but

starting to be to be used more in RL uh

that measure kind of intrinsic measures

of reward and we've wondered whether

those measures like empowerment which

which measure the the agent's abil uh

the agent's theoretical ability to

control to uh to change its environment

could be relevant here um we could take

a kind of basian approach where we think

of the reward function as something that

has un certainty in it and we attempt to

learn it um we could use basian methods

and then kind of this thing about side

effect is might be something that

relates to the prior um another thing is

multi-agent approaches uh which I think

there's been a fair amount of literature

on it has been has been discussed here

uh Cooperative re inverse reinforcement

learning was mentioned once where an an

an agent and a human work together to

optimize the humans values uh mechanism

design approaches where instead of

thinking of side effects we think of

other agents that care about the

environment and committing negative

externalities against them um uh so uh

reward hacking here are here are a few

things for uh for for reward hacking

just a few ways that you can kind of get

the wrong objective function or an

objective function that's gameable um

there's something called good heart's

law um which I just found out when I was

reading about this which says when a

metric becomes a Target it ceases to

become a good metric um so for example

in the example of our cleaning robot if

you notice that we could you know that

how good a job it's doing of cleaning is

proportional to how much detergent it

uses and this correlation holds almost

perfectly you could imagine oring it

that way and then the agent May respond

by dumping out all of its detergent and

not cleaning anything um and this this

only happens when you optimize on that

goal um partial observation is a big one

um goals and rewards in the real world

aren't experienced directly which

they're assumed to be experienced

directly in kind of the formalism of

reinforcement learning and so there's

all kinds of aliasing aliasing things

that can happen again example of the

robot closing its eyes so that it

doesn't see any dirt and if its reward

function isn't constructed carefully

then it will think that its job is done

um wireheading has been discussed by by

Stuart Russell and uh and some people in

the isia group um where a reward signal

is implemented in the environment and so

you can physically tamper with it and

replace its own reward function so

there's this kind of like self self-

modification aspect to it feedback loops

where there's a piece of the reward

function that can amplify itself and

drown out all other pieces of the reward

function so so uh Leon BAU gives an

example of font size and ad placement um

where uh ads that are more successfully

display in larger fonts because you

think people want to see them more um

but this is a this is a self-fulfilling

feedback loop and so you have these

giant ads that may not have been popular

at all um and simply complicated systems

um any any system that's complicated

just like with software it's

increasingly likely to break under

optimization pressure um some someone

showed that with tool assisted speedruns

you can uh execute arbitrary code by

just pressing the control buttons in uh

in Super Mario um I don't think an RL

agent has ever done that but but you

know the environment is such that they

could um so so some approaches to it

something that's been been mentioned a

lot by by Eric and a couple other people

um we shouldn't underestimate just

verification model uh a monitoring

isolation unit testing of Parts um uh

there's this idea of adversarial reward

functions where maybe the reason this

hacking is happening is that the

objective function is this kind of

static thing whereas the agent is this

active thing that's trying to like um

you know just trying to outsmart the the

reward function in some sense so is

there some way we didn't get super

speciic specific about this that you

could use generative adversarial

networks or other unsupervised

techniques to try and make the the the

objective function whatever it is more

more more resistant um making your

reward dependent on your model of the

environment instead of something that's

given to you is something that can help

deal with uh uh partial observation and

isolating yourself algorithmically from

parts of the environment that that

relate to the to to the reward um okay

uh uh expensive objective functions um

so so here we only have one problem and

it's basically the problem that I

described how do we ensure safe behavior

when the true objective function is too

expensive to evaluate frequently so

human could tell you what the right

thing to do is but you defeat the

purpose of automation if you always have

to ask a human so an example is a clean

robot might often find uh uh human

possessions it can't ask every time it

finds something whether it belongs to a

human um it might end up throwing out

the wrong one so it has it has to learn

when to ask and when it's when when you

know when it should know based on what

it's uh what it's asking so I won't go

into into this in too much detail um but

one kind of toy system we start we've

started playing with and actually have

started doing uh experiments with is is

this setup of a semi-supervised or or

active RL um which is the idea that you

know there's there's a reward it's just

like RL except there's a there's a

reward in the environment which we can

think of as you know maybe coming from

from Human supervision where the agent

actually has to actively request the

true objective and can only do so every

once in a while um and so can it do the

best it can to kind of like to to you

know learn to do the right thing based

on that very limited information and so

one one other thing we could we could

study and I'm skipping over a lot of

stuff is you know will the agent over

time as it requests the true objective

begin to learn learnable proxies say

instead of a human giving a detailed

evaluation it gives the human a button

that it can say yes or no to something

and get get a much quicker evaluation so

will the agent learn to develop

learnable proxies that are reliable and

that are ultimately Tethered to to the

to the true objective so all these kind

of variations on reinforcement learning

where where the supervision is weakened

in some way or altered in some way seem

like they have uh seem like they have a

fair amount of uh promise um correct

correct objective function problem in

learning um okay um this is a case where

um uh you know the the agent if it kind

of had had infinite time and correct

beliefs would do the right thing but

something goes wrong in the meantime um

so safe exploration is how do we avoid

catastrophic exploratory moves um a

robot we want a robot to experiment with

cleaning strategies we don't want it to

stick it a wet mop in electrical socket

we don't want helicopters to crash as

one of the previous speakers discussed

um distributional shift which I think

has has also been been discussed uh uh

previously how do we detect and respond

gracefully to differences between the

train and test

distribution uh oh yeah sorry that that

example is wrong the example I meant to

give is like the robots trained on

Factory floor it like uses a bunch of

harsh chemical and like then it works in

the office and that that that doesn't

make sense so generally these have

gotten a lot of attention and so those

those sections of our paper are more

kind of reviews but also with pointers

to how this can be applied to kind of

emerging uh emerging deep learning

applications um so go just going a

little bit a little bit quickly through

it there's substantial prior work in

this area I think there's a lot on

adapting risk sensitive performance

metrics to deep learning simulated

exploration especially with the Advent

of model-based reinforcement learning in

deepl setups and uh setups that use

bounded exploration to to ensure uh uh

air to ensure ergodicity um ml systems

can fail silently and unpredictably when

the test distribution differs from the

train distribution this is a Roomba that

like ate someone's hair

um uh uh with with classifiers I mean in

some sense things can't be very bad but

um for for people who remember this you

know a classifier can do something

incredibly offensive if you if if it's

not train in the right way and you give

it the wrong inputs uh RL agents can act

on really bad uh perceptual info we can

add ad hoc restrictions to make sure

that we kind of detect this but I think

ultimately we need and should be able to

have a machine learning solution because

you know if you're outside the training

distribution you have enough information

to detect that your your inputs are

different from from where they were

before and there's been a lot of work on

this um I'm running short on time so

I'll just summarize it very quickly but

kind of covariant shift assumptions that

make assumptions about the probability

of the output given the input being the

same and only the inputs changes and the

converse of that um there's I think

something very interesting is like

broadening the train distribution and

understanding if if if I train on a few

different types of data how much of that

does it take to generalize how do I

determine when I've generalized well um

and also how to respond in agents which

I think has been a very very

understudied uh uh topic there's been a

lot of study of like how to detect being

out of distribution how to respond to it

in classifiers but agents can do a whole

whole lot of things right if I if I see

a chemical that I've never seen before

um you know then then I I as a human can

experiment with that chemical in a

controlled environment instead of you

know even even if that's more expensive

instead of doing something else or I can

try and access more training data so I

think I think there's a whole world of

of kind of Agents remedying their own

uncertainty and knowing knowing what

they don't know um this is um some some

stuff we've kind of started proposing

experiments which I won't kind of go

through in in great detail but the

thrust of a lot of these is kind of

looking for large Suites of simulated

environments where we can we can have

kind of very broad setups where we can

test agents in a way that you know that

you know they don't overfit and they can

actually be shown to address these tasks

and just as simulated environments in

video games have been very important for

you know go and Atari and reinforcement

learning in general My Hope Is they can

be very important for safety and in fact

within a couple days of publishing the

paper some folks at openai implemented a

couple of the ideas we had in open AI

gym which is one of their tool open

source tools for RL uh just just one um

uh and and so finally since there seems

to have been kind of a tradition of

everyone mentioning a little bit the the

the long-term concerns I did want to

give my perspective which is I I do

think that kind of long-term concerns

about AGI and and and super intelligence

can be seen as kind of extreme versions

of accident risk um not Terminator

robots taking over but kind of the more

academic writers like like Nick Bostrom

who talk about you know if you if you

make a make a machine that you know

makes paper clips for you maybe maybe

it'll make so many paper clips that it

it takes over the world so this is kind

of a extreme and more fanciful version

of uh uh accident risks I I I think it's

too early to work on this directly I I

don't know what such systems would look

like and and what to do with them but

what I do feel strongly is that if we

take a principled and general approach

to the safety of today's systems that is

the correct response to concerns about

the far future so Andrew in who was my

my former boss had a quote worrying

about super intelligence today is like

worrying about overpopulation on Mars

but i' I'd like to propose a bit of a

modification to that even though I think

I agree with a lot of which is it may

not be worth worrying about

overpopulation on Mars today but we can

and should study overpopulation on earth

that is a serious issue we can think

about and if we do that right then a lot

of what we learn May someday apply to

Mars if and when we do go to

[Applause]

Mars but