Random Sampling in Statistics: Expected Value and Variance of the Sample Mean

welcome back okay so we're talking about

the theory of random sampling to say

something about a large but unknown

population in terms of a random smaller

sample of that population this is useful

all over in statistics and this is kind

of an entry point to more advanced uh

topics so we showed last time that you

can have this population um which is

kind of a large population its PDF may

or may not even be known but it a mean

and and a variance and then um the

sample statistics if I take a subsample

a little n subsample of that big n

population those samples become random

variables and the average of those

random variables

xbar um is hopefully an estimate of the

population mean mu so the kind of uh

sample mean should be a good estimate of

the population mean under some circum

ances and we can also compute things

like the sample variance and so on and

so forth so this opens up a ton of

questions um all kinds of questions come

up

so question one um does the expectation

of xar does

xar equal mu we we've kind of hinted

that this sample mean should converge uh

to me in the as as little n gets bigger

and bigger as my sample size gets bigger

but can I actually show that the

expectation of this random variable is

in fact the population mean mu that

would be very useful we're going to do

that today another question um what is

the variance of xbar meaning we we have

a pretty good gut feeling that this

sample mean should be gausian

distributed for reasonably large n we

know that from the central limit theorem

this should be kind of normally

distributed hopefully with a center

around the true mean mu but what's the

variance of that distribution is it a

fat distribution is it skinny we want

the variance of xar to be really really

small because that means xar is a really

really tight estimate of mu so this has

implications um about the convergence of

these values with n how fast is xar

converged to me how efficient is it

things like that there's questions you

know are is there a bias is it is the

expectation of xar mu plus some constant

Offset you know is there any bias in my

estimates those are all powerful

statistics questions we're going to ask

and answer those for this very very

simple case of simple random sampling to

iner things about a population but these

questions hold much more generally in

statistics in data analysis and even in

machine learning okay does my you know

if I sample data do I converge to a good

model of a much bigger complex process

okay good so we're going to jump in uh

and we're going to start with the

expectation value because that's always

easier to work within the variance

because the formula is simpler than the

formula for variance so we want to show

that um the expectation value of

xar equals mu meaning that xar is an

unbiased estimate of mu so we say that

xbar is an

unbiased

estimate

of mu this is statistics language for

the expectation value of xar equals mu

and there's no constant error so the x

xar is unbiased meaning it it converges

its expectation value is exactly mu okay

so we're going to prove this now this is

pretty easy to prove uh maybe I will do

this in green so the expectation of xar

um is literally I'm just going to plug

this in to the expectation this is equal

to the

expectation of 1 / n sum I = 1 to n of

each of my random variables

XI now we know that we can pop this

constant out and this the expectation of

a sum is the sum of expectations so this

equals 1 / n sum I = 1 to

n um

expectation of each of these

XIs here's a really important fact that

I need you to to believe and to know and

I'm just going to write it down here the

E for any uh for any

individual X for any

individual uh

indiv individual

sample

XI the expected value of

XI is equal to

Mu uh and the

variance of

x i is equal to Sigma squ where mu is

the true population mean and sigma squar

is the true population variance this is

super important any one of these

individual samples it's expected value

is mean mu and its expected variance is

Sigma squar where those are the

population values you can actually

convince yourself of this pretty easily

you can write down this expectation

value um this I'll just do it for for

the expected value and you can convince

yourself also for the variance um this

is the sum over every single possible n

over all of the big n uh J equal 1 of

all of the little values X J times the

probability that my random variable x i

equals little

XJ that's just the definition of

expected value of this random variable

it's the sum over all the possible

things it could be times the probability

that it is actually that

thing and there are each of these um the

chance that I drew any one of these for

x i is just 1 over n that's the

probability so this equals the sum over

uh big n of little X J * a probability

of 1 over big n this is the definition

of

my population mean it's 1 / n times the

sum of all of those little XIs so you

can convince yourself anyway that each

of these random variables each of these

XIs their expected value is Mu and their

expected variance their variance is

Sigma squar you can think about it

because each of these X's is pulled from

this population so you can kind of say

that uh x i is

distributed according to whatever the

distribution of my

population was okay whatever my

population distribution is each of these

XIs is randomly sampled from that

population distribution so anyway this

let's go back to to what we're trying to

show we're trying to show that the

expectation of xar equals mu so we take

our sample mean xar we plug it into this

expectation and it's the sum of all of

these little the the these random

variables x i * 1 over little n the

constant pops out the sum of an expect

the expectation of a sum is the sum of

the expectations and now each of these

expectation

values is Mu so I have um essentially

this

equals uh 1/ n time the sum of IAL 1 to

little n of mu each of these is equal to

Mu this is n * mu * 1/ n this whole

thing just equals mu the expected value

of xar is equal to Mu very very cool

this means that xar the sample mean is

an unbiased estimate uh of the

population mean mu and hopefully as n

gets bigger and bigger this expected

value um sorry the this distribution of

xar gets Tighter and Tighter and Tighter

around this expected value you good um

maybe I'll just draw a little picture so

um probably I have some population

distribution and I'm actually going to

draw it to be kind of gnarly um but

let's say it has some mean value some

mu the sample mean

xar by the central limit theorem we'll

prove this later but by the central

limit theorem xar is going to be a

normally distributed variable about its

expected value of

mu so

xar should be normally distributed with

its expectation value centered around mu

and we want xar to get Tighter and

Tighter and Tighter we want the spread

of possible X bars to be really really

small around this value of mu as n gets

larger that spread of course is related

to the variance of this uh of this xar

quantity so now let's talk about what's

the

variance of xar okay variance of xar

tells me how good this estimate is for

um increasing sample size little n okay

good um this result makes intuitive

sense now let's talk about the variance

uh of

xar so I'm going to actually

prove a slight approximation what I'm

going to write down is not the exact

variance of xar it's an approximation to

the variance of xar making an assumption

that each of these X's is

independent now remember we sampled

without replacement so every time I drew

a sample my population got a little

smaller that technically builds in a

small amount of dependence between these

variables but for really really big and

for really really big populations you

can kind of assume that these are are

independent and that's what I'm going to

write down here and then I'm going to

write down the correction for finite n

for finite population size so this is an

approximation uh this equals again I'm

going to plug in this expression into

xar this equals the variance of the sum

VAR of 1 / n * X1 plus dot dot dot plus

X little

n and I'm just going to again remind you

this is uh

um this is if if these are independent

samples then I can say this this is

um actually sorry if they're independent

samples then I can split these into the

sum of a bunch of variances so I'll wait

I'll I'll I'll wait to write down my

Independence assumption in a minute um

so my 1/n pops out as a 1 over n^ 2

that's how variance of a constant times

a variable you can pop that constant

squared out so this equals 1 over n^

squar times the variance of this sum and

that is the sum of the individual

variances that is VAR

X1 plus dot dot dot plus VAR

xn now I've used this assumption this is

true if my X eyes are independent

and that's true for very very large

population size and much much greater

than one like n a million or 100,000 or

10,000 this is going to be a very good

approximation technically there is joint

co-variance between these variables and

so this step is actually not exactly

true it's really kind of this is

approximately equal to this for very

large population size so be on the watch

for me making those kinds of approx IM

again we're trying to compute the

variance of our sample mean we want that

variance to be small it's equal now

approximately to 1 n^ 2times the sum of

the variances of all of those individual

elements and the sum of those variances

each of those variances are the

population variance Sigma squar so I can

write this

now as you know each of these this is

just um let's

say

uh this is n * Sigma

s and so this whole thing is

approximately equal to n / n^ 2 * Sigma

squ that's Sigma squar over n and

actually this is the result from the

central limit theorem so I want you to

go back and and check out that Central

limit theorem uh video this is the

result from the central limit

theorem um that that if you have the sum

of a bunch of independent random

variables each with their own variance

Sigma squar then the sum of those

variables would have um this uh variance

okay so this is actually all coming from

the central limit theorem this is um I

guess law of large numbers this is

Central limit theorem

good now I'll show this in the next

video I'll actually go through the Gory

details of deriving this in the next

video but remember this is only true for

very very large n very large population

so for

finite uh population

size Big N technically this VAR X bar

there is a correction and again I'm

going to derive this in the next lecture

there's a correction it's Sigma 2ar over

n * 1 - little n minus1 over big n minus

one okay and again

and this is approximately equal to Sigma

2 over little

n when uh little n is much less than big

n when I have a really big

population um and my sample size is

small compared to that really big

population then I recover this this very

very good approximation to the variance

of xar so for small populations and

small samples you need this this finite

size correction most of the time we're

going to end up using this result from

the Central limit theorem we're going to

assume that our sample mean xar is a

normally distributed random variable

with mean mu and variance Sigma squ Over

N where Sigma squar and mu are the

variance and mean of our overall

population so this xbar tells us a lot

about this unknown

population um so measuring this xar

measuring all of this sample taking this

random sample and Computing this xar the

sample mean tells me a ton about the

population and as n gets bigger and

bigger and bigger this variance gets

smaller and smaller and smaller meaning

we

converge uh to the true population mean

with a relatively small sample n okay

super cool stuff in the next lecture

this is going to be a technical lecture

I'm actually going to derive this finite

n correction uh to the variance of xar

it's pretty technical you can probably

skip it if you like but if you want to

know where it comes from um all write

this out in terms of the co-variances um

for the shrinking without replacement

population okay thank you