Data Analytics

Statistics and Probability

Every concept from the module, in order. Each one says why it exists, what it is in plain words, and the code that produces it — run for real against the employees table, so every number below is a genuine output.

What statistics is for

Descriptive vs inferential

Why it exists

The split that organises the whole subject. Every technique you learn sits on one side or the other, and which side you are on decides how confident you are allowed to sound.

In short

DESCRIPTIVE: you measured everybody, so you are only summarising and cannot be wrong. INFERENTIAL: you measured a few and want to talk about everybody, so there is always a risk attached.

Code

print("rows in the file:", len(df))
print()
print("DESCRIPTIVE - a claim about these 201 people only")
print("   average salary here:", round(sal.mean()))
print()
print("INFERENTIAL - a claim about everyone, built from a few")
print("   30 people say:", round(sal.sample(30, random_state=1).mean()))
print("   and we use that to talk about people we never measured")

Output

rows in the file: 201 DESCRIPTIVE - a claim about these 201 people only average salary here: 95648 INFERENTIAL - a claim about everyone, built from a few 30 people say: 70032 and we use that to talk about people we never measured

What kind of column is it

The four scales

Why it exists

The scale of a column decides which calculations are legal on it. This is the most common beginner mistake and the computer will not stop you making it.

In short

NOMINAL: names, no order. ORDINAL: there is an order but the gaps are not equal. INTERVAL: equal gaps but no true zero. RATIO: equal gaps and a real zero, so ratios make sense.

Code

print("NOMINAL   department :", df['department'].unique())
print("ORDINAL   education  :", df['education'].unique())
print("INTERVAL  join_date  :", df['join_date'].head(2).values)
print("RATIO     salary     :", df['salary'].head(2).values)
print()
print("the test for RATIO: does zero mean none of it?")
print("   salary 0  = no salary at all   -> ratio")
print("   year 0    = not 'no time'      -> interval")

Output

NOMINAL department : ['HR' 'Marketing' 'Tech' 'Sales'] ORDINAL education : ['Masters' 'Diploma' 'Bachelors'] INTERVAL join_date : ['2018-01-03' '2024-07-02'] RATIO salary : [47764. 86052.] the test for RATIO: does zero mean none of it? salary 0 = no salary at all -> ratio year 0 = not 'no time' -> interval

The scale decides what you may compute

Why it exists

Python will happily average a column of ID numbers and hand you something meaningless. This is the habit that stops you putting that number in a report.

In short

dept_id is NOMINAL - a name that happens to be written with digits. Its mean runs and means nothing. A rating is ORDINAL, and pandas refuses the median outright rather than giving you a wrong answer.

Code

print("dept_id values:", df['dept_id'].unique())
print("their mean    :", round(df['dept_id'].mean(), 2))
print("   -> valid arithmetic. there is no department 2.32.")
print()
print("now the same idea on a rating:")
try:
    print(df['rating'].median())
except TypeError as e:
    print("   TypeError -", str(e)[:44])

Output

dept_id values: [3 4 2 1] their mean : 2.32 -> valid arithmetic. there is no department 2.32. now the same idea on a rating: TypeError - Cannot convert ['Poor' 'Poor' 'Good' 'Excell

Discrete vs continuous

Why it exists

It decides whether counting an exact value makes sense, and it is the reason probability changes shape when you reach the normal curve.

In short

DISCRETE: whole steps you can count, with nothing in between. CONTINUOUS: any value in a range - you only stopped at one decimal place because the measuring tape did.

Code

print("DISCRETE   experience :", df['experience'].unique()[:6], "... whole years")
print("CONTINUOUS height_cm  :", df['height_cm'].head(3).values, "... any value")
print()
print("people exactly 170.0 cm tall:", (df['height_cm'] == 170.0).sum())
print("this is why continuous probability needs a RANGE, never a single point.")

Output

DISCRETE experience : [ 4 13 1 11 9 3] ... whole years CONTINUOUS height_cm : [169.5 169.5 166.5] ... any value people exactly 170.0 cm tall: 0 this is why continuous probability needs a RANGE, never a single point.

Where the middle is

Day 4

Mean (the average)

Why it exists

The single number everyone asks for first. Budgets, targets and salary bands all start here.

In short

Add everything up, divide by how many there are. Every value affects it, which is its strength and its weakness.

Code

print("mean salary:", round(sal.mean()))

Output

mean salary: 95648

Median (the middle value)

Why it exists

Use it whenever the data might contain extreme values. Property prices and salaries almost always need it.

In short

Line everything up in order and take the middle one. Half are below, half above. Extreme values cannot drag it.

Code

print("median salary:", round(sal.median()))

Output

median salary: 68647

Mode (the most common)

Why it exists

The only centre that works on words. Which city, which rating, which product sells most.

In short

The value that appears most often. A column can have more than one mode, or none worth reporting.

Code

print("most common department:", df['department'].mode()[0])
print("most common rating    :", df['rating'].mode()[0])

Output

most common department: Sales most common rating : Poor

When the mean lies

Why it exists

Knowing when NOT to trust the average. This is the single most useful habit in this whole list.

In short

One extreme value drags the mean and leaves the median untouched. A big gap between them is the warning sign.

Code

print("mean  :", round(sal.mean()))
print("median:", round(sal.median()))
print("gap   :", round(sal.mean() - sal.median()))
print("highest salary:", round(sal.max()))

Output

mean : 95648 median: 68647 gap : 27001 highest salary: 5200000

How spread out it is

Days 4 and 6

Range

Why it exists

A one-second check on how wide the data is. Mostly used to spot that something is wrong.

In short

Largest minus smallest. Uses only two values, so a single bad row ruins it completely.

Code

print("range:", round(sal.max() - sal.min()))

Output

range: 5166915

Quartiles and IQR

Why it exists

The spread measure that ignores extremes. It is what box plots are built from.

In short

Q1 has 25% below it, Q3 has 75% below it. IQR is the distance between them - the middle half of the data.

Code

print("Q1 :", round(sal.quantile(0.25)))
print("Q3 :", round(sal.quantile(0.75)))
print("IQR:", round(sal.quantile(0.75) - sal.quantile(0.25)))

Output

Q1 : 59004 Q3 : 80600 IQR: 21596

Five-number summary

Why it exists

The fastest complete picture of one column. Report this before anything else.

In short

Minimum, Q1, median, Q3, maximum. Five numbers that describe the whole shape.

Code

print("min   :", round(sal.min()))
print("Q1    :", round(sal.quantile(0.25)))
print("median:", round(sal.median()))
print("Q3    :", round(sal.quantile(0.75)))
print("max   :", round(sal.max()))

Output

min : 33085 Q1 : 59004 median: 68647 Q3 : 80600 max : 5200000

Variance

Why it exists

A stepping stone to standard deviation. You will rarely report it on its own.

In short

The average of the squared distances from the mean. Squaring stops the pluses and minuses cancelling out, but it also leaves the answer in squared units.

Code

print("variance:", round(sal.var()))
print("its units are RUPEES SQUARED, which is why nobody reports it")

Output

variance: 132519227501 its units are RUPEES SQUARED, which is why nobody reports it

Standard deviation

Why it exists

The standard way to say how spread out people are. Needed for the empirical rule, z-scores and everything in inference.

In short

The square root of the variance, which puts the answer back into normal units. Roughly: how far a typical value sits from the mean.

Code

print("standard deviation:", int(sal.std()))

Output

standard deviation: 364031

n minus 1, and ddof

Why it exists

Stops you silently under-reporting spread. Also a very common interview question.

In short

A sample is always a bit less spread out than the population it came from, so we divide by n-1 to compensate. Pandas does this by default; NumPy does not.

Code

import numpy as np
print("pandas .std()      :", int(sal.std()))            # divides by n-1
print("numpy  .std()      :", int(np.std(sal)))          # divides by n
print("numpy with ddof=1  :", int(np.std(sal, ddof=1)))

Output

pandas .std() : 364031 numpy .std() : 363116 numpy with ddof=1 : 364031

The 68-95-99.7 rule

Why it exists

Lets you judge in your head whether a value is ordinary or unusual, with no calculation.

In short

In a bell-shaped column, about 68% of values sit within 1 standard deviation of the mean, 95% within 2, and 99.7% within 3.

Code

m, s = h.mean(), h.std()
for k in (1, 2, 3):
    inside = ((h >= m - k*s) & (h <= m + k*s)).mean() * 100
    print(k, "sd:", round(inside, 1), "%")

Output

1 sd: 69.7 % 2 sd: 95.0 % 3 sd: 100.0 %

Skewness

Why it exists

Tells you in one number whether the mean can be trusted.

In short

How lopsided the shape is. 0 is perfectly symmetric. Positive means a long tail to the right. Above about 1 means the mean is being dragged.

Code

print("salary skew:", round(sal.skew(), 2), "  <- badly lopsided")
print("height skew:", round(h.skew(), 2), "  <- nearly symmetric")

Output

salary skew: 14.06 <- badly lopsided height skew: -0.19 <- nearly symmetric

Finding outliers

Day 5

The 1.5 x IQR rule

Why it exists

The standard, defensible way to flag suspicious values. Use it before reporting any average.

In short

Anything below Q1 - 1.5*IQR or above Q3 + 1.5*IQR is flagged. It is a rule of thumb, not proof that a value is wrong.

Code

low  = q1 - 1.5 * iqr
high = q3 + 1.5 * iqr
print("fences:", round(low), "to", round(high))
flagged = sal[(sal < low) | (sal > high)]
print("flagged:", len(flagged), "->", flagged.values)

Output

fences: 26610 to 112994 flagged: 2 -> [5200000. 116894.]

What removing outliers does

Why it exists

Shows you exactly how much damage two rows can do, which is the argument for always checking.

In short

Removing flagged values barely moves the median but can transform the mean and the standard deviation.

Code

kept = sal[(sal >= q1 - 1.5*iqr) & (sal <= q3 + 1.5*iqr)]
print("before: mean", int(sal.mean()),  "median", int(sal.median()),  "std", int(sal.std()))
print("after : mean", int(kept.mean()), "median", int(kept.median()), "std", int(kept.std()))

Output

before: mean 95648 median 68647 std 364031 after : mean 69630 median 68342 std 16066

An outlier is not always an error

Why it exists

Stops you deleting your most important row. A fraud, a bestseller and a typo all look identical to the rule.

In short

The rule finds unusual values. Whether to remove one is a judgement you make by looking at the row, never automatic.

Code

big = df[df['salary'] > 1000000]
print("rows above 10 lakh:", len(big))
print("   department:", big['department'].values)
print("   salary    :", big['salary'].values)
print()
print("is this a typo, a CEO, or fraud? the rule cannot tell you. go and look.")

Output

rows above 10 lakh: 1 department: ['Tech'] salary : [5200000.] is this a typo, a CEO, or fraud? the rule cannot tell you. go and look.

Two columns together

Day 7

Covariance

Why it exists

A stepping stone to correlation. You will almost never report it.

In short

Tells you whether two columns move together, but the answer comes out in a unit nobody can picture - here, years times rupees.

Code

d = df.dropna(subset=['salary','experience'])
print("covariance:", round(d['experience'].cov(d['salary'])))
print("units: YEARS x RUPEES. meaningless on its own.")

Output

covariance: 128281 units: YEARS x RUPEES. meaningless on its own.

Pearson correlation

Why it exists

The standard measure of how strongly two numeric columns move together.

In short

Covariance scaled to sit between -1 and +1. Near 0 means no straight-line relationship. It only measures STRAIGHT lines, and outliers wreck it.

Code

d = df.dropna(subset=['salary','experience'])
print("with outliers   :", round(d['experience'].corr(d['salary']), 3))
c = d[(d['salary'] >= q1-1.5*iqr) & (d['salary'] <= q3+1.5*iqr)]
print("without outliers:", round(c['experience'].corr(c['salary']), 3))

Output

with outliers : 0.079 without outliers: 0.607

Spearman correlation

Why it exists

The version to trust when the data is messy or you have not cleaned it yet.

In short

Works on ranks instead of values. The biggest salary is just 'rank 1', so how extreme it is cannot distort the answer.

Code

d = df.dropna(subset=['salary','experience'])
print("pearson :", round(d['experience'].corr(d['salary']), 3))
print("spearman:", round(d['experience'].corr(d['salary'], method='spearman'), 3))
print()
print("same data. pearson was wrecked by 2 rows. spearman was not.")

Output

pearson : 0.079 spearman: 0.612 same data. pearson was wrecked by 2 rows. spearman was not.

Correlation is not causation

Why it exists

The sentence that stops you making an expensive wrong decision.

In short

Two columns moving together has three possible explanations: A causes B, B causes A, or something else causes both. Correlation cannot tell you which.

Code

print("experience and salary correlate at 0.607.")
print()
print("  1  experience raises pay")
print("  2  higher paid people stay longer")
print("  3  department drives both - Tech pays more AND keeps people")
print()
print("if you cannot rule out 2 and 3, say 'associated', not 'causes'.")

Output

experience and salary correlate at 0.607. 1 experience raises pay 2 higher paid people stay longer 3 department drives both - Tech pays more AND keeps people if you cannot rule out 2 and 3, say 'associated', not 'causes'.

Samples and populations

Day 3

Parameter vs statistic

Why it exists

Two words that get used as if they mean the same thing. Interviewers ask this directly, because it shows whether you know what you are actually computing.

In short

A PARAMETER describes the population and you almost never know it. A STATISTIC describes your sample and is the only thing you can compute. Inferential statistics is using the second to guess the first.

Code

print("PARAMETER  mu     - the population average :", round(sal.mean()))
print("STATISTIC  x-bar  - this sample average    :", round(sal.sample(30, random_state=1).mean()))
print()
print("at work you only ever see the second line.")
print("we can see both here only because this file IS our population.")

Output

PARAMETER mu - the population average : 95648 STATISTIC x-bar - this sample average : 70032 at work you only ever see the second line. we can see both here only because this file IS our population.

More data does not fix bias

Why it exists

The most expensive mistake in sampling, and the reason a huge survey can still be wrong. Remember it as a story rather than a formula.

In short

In 1936 the Literary Digest asked ten million people, got 2.4 million replies, and still called the election wrong because of WHO they asked. A 50,000-person honest sample beat them. Size fixes noise; only method fixes bias.

Code

print("an honest sample of 30:")
print("   ", round(sal.sample(30, random_state=1).mean()))
print()
print("a BIASED sample - every Tech employee, 49 people:")
print("   ", round(df[df['department'] == 'Tech']['salary'].mean()))
print()
print("the truth:", round(sal.mean()))
print("the biased one used more people and landed further away.")

Output

an honest sample of 30: 70032 a BIASED sample - every Tech employee, 49 people: 189754 the truth: 95648 the biased one used more people and landed further away.

Population vs sample

Why it exists

You will never measure everybody. Every number you report at work comes from a sample.

In short

The POPULATION is everyone you care about. A SAMPLE is the few you actually measured. Greek letters for the population, Roman for the sample.

Code

print("population mean  mu     :", round(pop.mean()))
print("sample mean      x-bar  :", round(pop.sample(30, random_state=1).mean()))
print()
print("we can only ever compute the second one. the first is what we want.")

Output

population mean mu : 69630 sample mean x-bar : 70006 we can only ever compute the second one. the first is what we want.

How to build a sample

Why it exists

A biased sample gives a confident wrong answer, and more data does not fix it.

In short

SIMPLE RANDOM: everyone equally likely. STRATIFIED: split into groups, sample each in proportion. SYSTEMATIC: every nth one. CLUSTER: pick whole groups. The first two are the ones you will use.

Code

print("SIMPLE RANDOM - everyone equally likely")
print("   ", round(pop.sample(30, random_state=1).mean()))
print()
print("STRATIFIED - take some from every group on purpose")
for name, g in df.groupby('department'):
    print("   ", name, "->", round(g['salary'].sample(8, random_state=1).mean()))
print()
print("SYSTEMATIC - every 10th row.   CLUSTER - pick whole offices, survey all of them.")
print("CONVENIENCE - whoever is nearby. fastest, and the one that goes wrong.")

Output

SIMPLE RANDOM - everyone equally likely 70006 STRATIFIED - take some from every group on purpose HR -> 56387 Marketing -> 66750 Sales -> 75178 Tech -> 82573 SYSTEMATIC - every 10th row. CLUSTER - pick whole offices, survey all of them. CONVENIENCE - whoever is nearby. fastest, and the one that goes wrong.

Bias vs noise

Why it exists

They look the same on a screen and need completely different fixes.

In short

NOISE is random scatter - it shrinks with more data. BIAS is being consistently off in one direction - more data just makes you confidently wrong. Only a better sampling method fixes bias.

Code

print("noise: 5 honest samples, all different, all near the truth")
for s in (1,2,3):
    print("   ", round(pop.sample(30, random_state=s).mean()))
print()
print("bias: only surveying Tech")
print("   ", round(df[df['department']=='Tech']['salary'].mean()),
      " <- consistently too high, and 10x the data will not help")

Output

noise: 5 honest samples, all different, all near the truth 70006 67849 67775 bias: only surveying Tech 189754 <- consistently too high, and 10x the data will not help

Probability basics

Day 8

The two ways to get a probability

Why it exists

Knowing which one you are in. Students think probability means dice; at work it is almost always counting.

In short

THEORETICAL: reason from the structure of the thing - a coin has 2 sides, so heads is 1/2. EMPIRICAL: count what actually happened - 49 of 201 work in Tech, so 0.244.

Code

print("theoretical: a coin has 2 sides      -> P(heads) = 0.5")
print("empirical  : count the actual rows   -> P(Tech)  =",
      round((df['department'] == 'Tech').mean(), 3))
print()
print("note it is 0.244, NOT 0.25 - the four departments are different sizes")

Output

theoretical: a coin has 2 sides -> P(heads) = 0.5 empirical : count the actual rows -> P(Tech) = 0.244 note it is 0.244, NOT 0.25 - the four departments are different sizes

The scale, and the complement rule

Why it exists

The complement rule turns horrible calculations into one line. Worth knowing purely for that.

In short

Every probability sits between 0 and 1, and all the pieces sum to 1. So the chance of something NOT happening is 1 minus the chance it does.

Code

p = df['department'].value_counts(normalize=True)
print(p.round(3))
print("total:", p.sum())
print()
print("P(not Tech) = 1 -", round(p['Tech'],3), "=", round(1 - p['Tech'], 3))

Output

department Sales 0.323 Tech 0.244 HR 0.224 Marketing 0.209 Name: proportion, dtype: float64 total: 1.0 P(not Tech) = 1 - 0.244 = 0.756

The OR rule

Why it exists

Stops you double-counting, which is the most common arithmetic slip in this topic.

In short

P(A or B) = P(A) + P(B) - P(A and B). You subtract the overlap because anyone in both groups got counted twice.

Code

t = df['department'] == 'Tech'
e = df['rating'] == 'Excellent'
print("P(Tech)         :", round(t.mean(), 3))
print("P(Excellent)    :", round(e.mean(), 3))
print("just added      :", round(t.mean() + e.mean(), 3), " <- too big")
print("actually        :", round((t | e).mean(), 3))
print("because", (t & e).sum(), "people are in BOTH and got counted twice")

Output

P(Tech) : 0.244 P(Excellent) : 0.244 just added : 0.488 <- too big actually : 0.423 because 13 people are in BOTH and got counted twice

The AND rule, and its condition

Why it exists

The condition is the part everyone forgets, and forgetting it gives wrong answers.

In short

P(A and B) = P(A) x P(B), but ONLY if the two do not affect each other. Always check by multiplying and comparing against the real count.

Code

t  = df['department'] == 'Tech'
e  = df['rating'] == 'Excellent'
hi = df['salary'] > df['salary'].median()

print("Tech & Excellent : actual", round((t & e).mean(), 3),
      " predicted", round(t.mean()*e.mean(), 3), " -> works")
print("Tech & well paid : actual", round((t & hi).mean(), 3),
      " predicted", round(t.mean()*hi.mean(), 3), " -> FAILS")

Output

Tech & Excellent : actual 0.065 predicted 0.059 -> works Tech & well paid : actual 0.189 predicted 0.12 -> FAILS

Mutually exclusive vs independent

Why it exists

These two get merged in every exam. They are close to opposites.

In short

MUTUALLY EXCLUSIVE: both cannot happen - you cannot be in HR and Tech. INDEPENDENT: one tells you nothing about the other. If two things are mutually exclusive, knowing one tells you the other is impossible, which is total information - the opposite of independent.

Code

print("P(HR and Tech) =", ((df['department']=='HR') & (df['department']=='Tech')).sum(),
      " -> mutually exclusive")
print()
print("tell me someone is in HR. do you now know about Tech?")
print("YES - certainly not in Tech. that is a LOT of information.")
print("independent means ZERO information. so they are opposites.")

Output

P(HR and Tech) = 0 -> mutually exclusive tell me someone is in HR. do you now know about Tech? YES - certainly not in Tech. that is a LOT of information. independent means ZERO information. so they are opposites.

Conditional probability

Why it exists

The shape almost every real question comes in - not 'will a customer buy' but 'will a customer WHO CLICKED THE AD buy'.

In short

P(A | B) is the chance of A given B is already true. The bar means 'given', not divide. To do it: throw away every row where B is false, then count normally. It is a filter.

Code

print("P(Excellent)        :", round((df['rating']=='Excellent').mean(), 3))
tech = df[df['department']=='Tech']
print("P(Excellent | Tech) :", round((tech['rating']=='Excellent').mean(), 3))
print("  barely moved -> department tells you nothing about rating")
print()
hi = df[df['salary'] > df['salary'].median()]
print("P(Tech)             :", round((df['department']=='Tech').mean(), 3))
print("P(Tech | well paid) :", round((hi['department']=='Tech').mean(), 3))
print("  jumped -> salary DOES tell you about department")

Output

P(Excellent) : 0.244 P(Excellent | Tech) : 0.265 barely moved -> department tells you nothing about rating P(Tech) : 0.244 P(Tech | well paid) : 0.384 jumped -> salary DOES tell you about department

P(A|B) is not P(B|A)

Why it exists

Getting this backwards is how confident, expensive, wrong decisions get made.

In short

The two directions are different questions with different answers. Same people on top of both fractions - only the denominator changes.

Code

med  = df['salary'].median()
tech = df[df['department']=='Tech']
hi   = df[df['salary'] > med]
both = ((df['department']=='Tech') & (df['salary'] > med)).sum()

print("P(Tech | well paid) =", both, "/", len(hi), "=", round(both/len(hi), 3))
print("P(well paid | Tech) =", both, "/", len(tech), "=", round(both/len(tech), 3))
print()
print("same", both, "people on top. only the bottom changed. answer doubled.")
print()
print("'most food poisoning victims ate out'  - true")
print("'most people who eat out get poisoned' - nonsense")

Output

P(Tech | well paid) = 38 / 99 = 0.384 P(well paid | Tech) = 38 / 49 = 0.776 same 38 people on top. only the bottom changed. answer doubled. 'most food poisoning victims ate out' - true 'most people who eat out get poisoned' - nonsense

The normal distribution

Day 9

What a normal distribution is

Why it exists

It is the shape that makes confidence intervals and every test possible. Everything after this depends on it.

In short

A symmetric bell shape. Described COMPLETELY by two numbers - the mean and the standard deviation. Mean, median and mode all sit in the same place.

Code

print("height column: mean", round(h.mean(),1), " std", round(h.std(),1),
      " skew", round(h.skew(),2))
print("mean", round(h.mean(),1), " median", round(h.median(),1),
      " -> almost identical, so it is symmetric")

Output

height column: mean 168.5 std 7.7 skew -0.19 mean 168.5 median 168.2 -> almost identical, so it is symmetric

Probability is AREA, not height

Why it exists

Explains why the chance of any exact measurement is zero, which sounds absurd until you see why.

In short

For something measured rather than counted, no single exact value has any share of the total. You can only ask about a RANGE, and the probability is the area under the curve over that range.

Code

print("P(height is exactly 170.000...) = 0")
print("between 169 and 170 there are endless possible values")
print()
print("so ask about a RANGE instead:")
print("P(165 to 175) =", round(((h>=165)&(h<=175)).mean(), 3))

Output

P(height is exactly 170.000...) = 0 between 169 and 170 there are endless possible values so ask about a RANGE instead: P(165 to 175) = 0.527

Z-score

Why it exists

Makes two things comparable when the raw numbers are not - 78 in maths against 55 in physics.

In short

How many standard deviations a value sits from the mean. z = (value - mean) / standard deviation. Positive is above average, negative below.

Code

m, s = h.mean(), h.std()
for cm in (160, 170, 175, 185):
    print(cm, "cm -> z =", round((cm-m)/s, 2))
print()
print("marks: A got 78 (mean 70, sd 8) -> z =", round((78-70)/8, 2))
print("       B got 55 (mean 45, sd 4) -> z =", round((55-45)/4, 2))
print("       B did far better, despite the lower mark")

Output

160 cm -> z = -1.1 170 cm -> z = 0.2 175 cm -> z = 0.84 185 cm -> z = 2.14 marks: A got 78 (mean 70, sd 8) -> z = 1.0 B got 55 (mean 45, sd 4) -> z = 2.5 B did far better, despite the lower mark

PDF, CDF and PPF

Why it exists

Three questions about the same curve. CDF and PPF are the two you will actually call.

In short

PDF (Probability Density Function) is the shape itself. CDF (Cumulative Distribution Function) takes a value and returns the percentage below it. PPF (Percent Point Function) takes a percentage and returns the value. CDF and PPF are inverses.

Code

m, s = h.mean(), h.std()
print("CDF: 175 cm  ->", round(stats.norm.cdf(175, m, s)*100, 1), "% are shorter")
print("PPF: 80%     ->", round(stats.norm.ppf(0.80, m, s), 1), "cm")
print()
print("they undo each other. 'inverse CDF' and 'quantile function' both mean PPF.")

Output

CDF: 175 cm -> 80.0 % are shorter PPF: 80% -> 175.0 cm they undo each other. 'inverse CDF' and 'quantile function' both mean PPF.

Standardisation vs normalisation

Why it exists

Two words you will meet and confuse. Thirty seconds each is enough.

In short

STANDARDISATION: subtract the mean, divide by the std - that is the z-score, centred on 0 with no limits. NORMALISATION: squeeze everything into 0 to 1 using (value-min)/(max-min).

Code

z    = (h - h.mean()) / h.std()
norm = (h - h.min()) / (h.max() - h.min())

print("first 3 heights :", h.head(3).values)
print("standardised (z):", z.round(2).head(3).values)
print("normalised 0-1  :", norm.round(2).head(3).values)
print()
print("note: round(series) works. round(array) does NOT - use .round(2) instead.")

Output

first 3 heights : [169.5 169.5 166.5] standardised (z): [ 0.13 0.13 -0.26] normalised 0-1 : [0.53 0.53 0.46] note: round(series) works. round(array) does NOT - use .round(2) instead.

The Central Limit Theorem

Day 10

The problem it solves

Why it exists

This is the hinge. Everything after it - intervals, tests, A/B - is this applied.

In short

One sample can be badly wrong, and you only ever get one. So how can any number be trusted? The answer is that the PATTERN of sample averages is completely predictable, even when the data is not.

Code

print("five samples of 30 from the SAME column:")
for s in (1,2,3,4,5):
    print("   ", round(sal.sample(30, random_state=s).mean()))
print("the truth:", round(sal.mean()))
print()
print("if I take only ONE of these, which am I holding? no way to know.")

Output

five samples of 30 from the SAME column: 70032 244006 239817 65149 235447 the truth: 95648 if I take only ONE of these, which am I holding? no way to know.

What the theorem says

Why it exists

It lets you say how wrong your one sample might be, without ever taking a second one.

In short

Take the average of a sample and repeat. Those averages form their own shape. The BIGGER each sample, the CLOSER that shape gets to a normal bell curve, centred on the true mean - whatever shape the original data had.

Code

means = []
for i in range(1000):
    means.append(pop.sample(30, replace=True, random_state=i).mean())
means = pd.Series(means)
print("average of the 1000 averages:", round(means.mean()), " (truth 69630)")
print("spread of those averages    :", round(means.std()))
print("skew of those averages      :", round(means.skew(), 2), " -> near 0 = normal")

Output

average of the 1000 averages: 69680 (truth 69630) spread of those averages : 3026 skew of those averages : 0.09 -> near 0 = normal

Bigger samples, not more samples

Why it exists

The most common wrong takeaway. Repeating a small sample does not help.

In short

Normal means skew 0, so 'becomes normal' means 'skew heads to zero'. It is the SIZE of each sample that drives it down - the number of repeats only makes the picture smoother.

Code

def skew_of(n, reps):
    means = []
    for i in range(reps):
        means.append(sal.sample(n, replace=True, random_state=i).mean())
    return round(pd.Series(means).skew(), 2)

print("raw column skew:", round(sal.skew(), 2))
print()
print("BIGGER samples fix it:")
for n in (30, 100, 500): print("   ", n, "->", skew_of(n, 2000))
print()
print("MORE samples do not:")
for r in (200, 5000, 20000): print("   ", r, "samples of 30 ->", skew_of(30, r))

Output

raw column skew: 14.06 BIGGER samples fix it: 30 -> 2.52 100 -> 1.41 500 -> 0.77 MORE samples do not: 200 samples of 30 -> 2.21 5000 samples of 30 -> 2.52 20000 samples of 30 -> 2.51

Standard error

Why it exists

The number that tells you how much to trust your own average. Nothing else does this.

In short

The standard deviation OF AVERAGES, rather than of people. It shrinks as the sample grows: sigma / sqrt(n). A standard deviation does not shrink with more data; a standard error does.

Code

sigma = pop.std()
print("standard deviation of PEOPLE :", round(sigma), " (never changes)")
print()
for n in (5, 30, 100, 400):
    print("  n =", n, " standard error =", round(sigma/math.sqrt(n)))
print()
print("4x the data for 2x the precision. it never gets cheaper.")

Output

standard deviation of PEOPLE : 16066 (never changes) n = 5 standard error = 7185 n = 30 standard error = 2933 n = 100 standard error = 1607 n = 400 standard error = 803 4x the data for 2x the precision. it never gets cheaper.

Confidence intervals

Day 11

What a confidence interval is

Why it exists

It turns 'the average is X' - which you cannot support - into something you can defend.

In short

Your sample's average, plus and minus a margin. margin = 1.96 x standard error. The result is reported as 'X, give or take Y'.

Code

sample = pop.sample(100, replace=True, random_state=54)
m  = sample.mean()
se = sample.std() / math.sqrt(100)
print("sample mean    :", round(m))
print("standard error :", round(se))
print("95% interval   :", round(m - 1.96*se), "to", round(m + 1.96*se))
print("the truth      :", round(pop.mean()), " <- inside")

Output

sample mean : 72149 standard error : 1631 95% interval : 68952 to 75346 the truth : 69630 <- inside

What the 95% actually means

Why it exists

The single most common error in applied statistics. People with degrees get this wrong.

In short

WRONG: '95% chance the true mean is in my interval.' The true mean is FIXED - it is in or out, already decided. RIGHT: '95% of the intervals this METHOD produces contain the truth.' The randomness is in the interval, not in the truth.

Code

caught, truth = 0, pop.mean()
for i in range(100):
    x = pop.sample(100, replace=True, random_state=i)
    e = 1.96 * x.std() / math.sqrt(100)
    if x.mean() - e <= truth <= x.mean() + e:
        caught += 1
print(caught, "of 100 intervals contained the truth")
print()
print("the truth never moved. the INTERVAL moved.")
print("the 4 that missed did nothing wrong - that is what 95% means.")

Output

96 of 100 intervals contained the truth the truth never moved. the INTERVAL moved. the 4 that missed did nothing wrong - that is what 95% means.

It is about the AVERAGE, not the data

Why it exists

A very easy mistake that makes the interval look ten times too narrow.

In short

The interval describes where the MEAN probably sits. It does not describe where 95% of the individual values sit - that range is far wider.

Code

sample = pop.sample(100, replace=True, random_state=54)
m  = sample.mean(); se = sample.std()/math.sqrt(100)
lo, hi = m-1.96*se, m+1.96*se
print("interval for the MEAN :", round(lo), "to", round(hi))
print("share of SALARIES in that range:", round(((pop>=lo)&(pop<=hi)).mean()*100, 1), "%")
print()
print("where 95% of PEOPLE actually sit:",
      round(pop.quantile(0.025)), "to", round(pop.quantile(0.975)))

Output

interval for the MEAN : 68952 to 75346 share of SALARIES in that range: 13.2 % where 95% of PEOPLE actually sit: 39821 to 103162

The two levers

Why it exists

Both cost something. Knowing what lets you defend the sample size you chose.

In short

More CONFIDENCE makes the interval WIDER - certainty is bought with vagueness. More DATA makes it narrower, but at four times the data for twice the precision.

Code

sample = pop.sample(100, replace=True, random_state=54)
se = sample.std()/math.sqrt(100); m = sample.mean()
for lvl, z in [(90, 1.645), (95, 1.96), (99, 2.576)]:
    print(lvl, "% -> width", round(2*z*se))
print()
for n in (100, 400, 1600):
    print("n =", n, "-> width", round(2*1.96*pop.std()/math.sqrt(n)))

Output

90 % -> width 5367 95 % -> width 6395 99 % -> width 8404 n = 100 -> width 6298 n = 400 -> width 3149 n = 1600 -> width 1574

How many people do you need

Why it exists

Answers 'why do polls always ask about a thousand people' - and lets you size your own survey.

In short

Turn the margin formula inside out to give n. Use p = 0.5 because p(1-p) is largest there, so you are safe whatever the real answer is. The population size is NOT in the formula.

Code

from math import ceil
for E_ in (0.05, 0.03, 0.01):
    n = (1.96**2 * 0.5 * 0.5) / E_**2
    print("margin +/-", int(E_*100), "% -> need", ceil(n), "people")
print()
print("population size is not in the formula.")
print("1,000 people tells you as much about India as about Iceland.")

Output

margin +/- 5 % -> need 385 people margin +/- 3 % -> need 1068 people margin +/- 1 % -> need 9604 people population size is not in the formula. 1,000 people tells you as much about India as about Iceland.

Hypothesis testing

Day 12

The one question every test asks

Why it exists

Stops you reporting a difference that was only luck. Two averages are never exactly equal.

In short

Could I see a difference this big by pure chance, even if there were no real difference at all? If yes, you have no finding. If noise would almost never produce it, you do.

Code

tech = pop[df['department']=='Tech']
hr   = pop[df['department']=='HR']
print("Tech average:", round(tech.mean()))
print("HR average  :", round(hr.mean()))
print("difference  :", round(tech.mean() - hr.mean()))
print()
print("real, or just these 90 people? that is what a test decides.")

Output

Tech average: 82419 HR average : 56705 difference : 25714 real, or just these 90 people? that is what a test decides.

Null and alternative hypothesis

Why it exists

The structure of every test. It is a criminal trial, and the match is exact.

In short

NULL (H0) is the boring position: no difference, nothing going on. Assumed true until the evidence forces you off it - like 'innocent'. ALTERNATIVE (H1) is what you would have to prove.

Code

print("H0  the null         : Tech and HR are paid the same")
print("H1  the alternative  : they are not")
print()
print("court     statistics")
print("innocent  H0 - assumed until disproved")
print("guilty    H1 - what the prosecution must prove")

Output

H0 the null : Tech and HR are paid the same H1 the alternative : they are not court statistics innocent H0 - assumed until disproved guilty H1 - what the prosecution must prove

The p-value

Why it exists

The number every test reports. Misreading it is the most common error in applied statistics.

In short

The probability of seeing data at least this extreme IF the null were true. It is a statement about the DATA, computed in a world where nothing is going on - not a probability about your hypothesis.

Code

sample = pop.sample(100, replace=True, random_state=54)
t, p = stats.ttest_1samp(sample, 65000)
print("sample average:", round(sample.mean()), " claim: 65000")
print("p-value       :", round(p, 4))
print()
print("reads as: IF the true average really were 65,000,")
print("a sample like ours would almost never happen. so REJECT 65,000.")
print("(0.0000 is rounding - the p-value is tiny, never exactly zero.)")

Output

sample average: 72149 claim: 65000 p-value : 0.0 reads as: IF the true average really were 65,000, a sample like ours would almost never happen. so REJECT 65,000. (0.0000 is rounding - the p-value is tiny, never exactly zero.)

What a p-value is NOT

Why it exists

These two sentences get written in real reports and they are both wrong.

In short

p = 0.03 does NOT mean '3% chance the null is true', and NOT '97% chance my finding is real'. Same trap as confidence intervals: the probability belongs to the random thing (the data), never the fixed thing (the hypothesis).

Code

print("p = 0.03 means:")
print("  IF nothing were going on, data this extreme happens 3% of the time")
print()
print("it does NOT mean:")
print("  '3% chance the null is true'")
print("  '97% chance my finding is real'")

Output

p = 0.03 means: IF nothing were going on, data this extreme happens 3% of the time it does NOT mean: '3% chance the null is true' '97% chance my finding is real'

The 0.05 threshold

Why it exists

Knowing it is a convention, not a law, is part of using it honestly.

In short

Somebody has to draw a line for 'surprising enough'. 0.05 is where most fields settled, suggested about a hundred years ago. Particle physics uses about 1 in 3.5 million. 0.049 and 0.051 are the same result.

Code

print("0.05 is a CONVENTION, not a law of nature.")
print()
print("  most fields      0.05")
print("  some medicine    0.01")
print("  particle physics about 0.0000003")
print()
print("so p = 0.049 and p = 0.051 mean the same thing. do not pretend otherwise.")

Output

0.05 is a CONVENTION, not a law of nature. most fields 0.05 some medicine 0.01 particle physics about 0.0000003 so p = 0.049 and p = 0.051 mean the same thing. do not pretend otherwise.

Fail to reject is not proved

Why it exists

Never say 'accept the null'. Once a student writes it they keep writing it.

In short

A high p-value means the evidence was not strong enough - not that there is nothing there. A tiny sample fails to reject almost anything. 'Not guilty' is a verdict about the evidence, not a certificate of innocence.

Code

sample = pop.sample(100, replace=True, random_state=54)
t, p = stats.ttest_1samp(sample, 72000)
print("sample average:", round(sample.mean()), " claim: 72000")
print("p-value       :", round(p, 4), " -> fail to reject")
print()
print("we did NOT prove the average is 72,000.")
print("we found no evidence against it. completely different.")

Output

sample average: 72149 claim: 72000 p-value : 0.9274 -> fail to reject we did NOT prove the average is 72,000. we found no evidence against it. completely different.

Type I and Type II errors

Why it exists

Explains why running lots of tests is dangerous, and why small samples miss real effects.

In short

TYPE I: you say there is a difference and there is not - crying wolf. Happens 5% of the time BY DESIGN at p<0.05. TYPE II: a real difference exists and you miss it, almost always because the sample was too small.

Code

print("Type I  - convict the innocent. 5% of the time, by design.")
print("Type II - let the guilty go. usually too small a sample.")
print()
print("a real 6,000 difference, three sample sizes:")
import numpy as np
rng = np.random.default_rng(0)
for n in (10, 30, 200):
    a = rng.normal(70000, 16000, n); b = rng.normal(76000, 16000, n)
    p = stats.ttest_ind(a, b)[1]
    print("   n =", n, " p =", round(p,4), " ->", "found it" if p<0.05 else "MISSED IT")

Output

Type I - convict the innocent. 5% of the time, by design. Type II - let the guilty go. usually too small a sample. a real 6,000 difference, three sample sizes: n = 10 p = 0.6779 -> MISSED IT n = 30 p = 0.5749 -> MISSED IT n = 200 p = 0.0001 -> found it

Which test to use

Day 13

The decision table

Why it exists

The only skill worth having here. You will never compute one by hand.

In short

Comparing NUMBERS goes to a t-test. Comparing COUNTS in categories goes to chi-square. Get that split right and you pick correctly almost every time.

Code

print("one number vs a claim ....... stats.ttest_1samp(data, claim)")
print("two groups, a number ........ stats.ttest_ind(a, b, equal_var=False)")
print("two categories, counts ...... stats.chi2_contingency(crosstab)")
print("3+ groups, a number ......... ANOVA")

Output

one number vs a claim ....... stats.ttest_1samp(data, claim) two groups, a number ........ stats.ttest_ind(a, b, equal_var=False) two categories, counts ...... stats.chi2_contingency(crosstab) 3+ groups, a number ......... ANOVA

Two-sample t-test

Why it exists

The everyday test: do these two groups really differ?

In short

Compares the averages of two groups. Use equal_var=False as your default - it does not assume the groups have the same spread, and costs almost nothing.

Code

tech = clean[clean['department'] == 'Tech']['salary']
hr   = clean[clean['department'] == 'HR']['salary']
t, p = stats.ttest_ind(tech, hr, equal_var=False)
print("Tech", round(tech.mean()), "vs HR", round(hr.mean()), " p =", p)
print("   -> real difference")
print()
sales = clean[clean['department'] == 'Sales']['salary']
mk    = clean[clean['department'] == 'Marketing']['salary']
t, p = stats.ttest_ind(sales, mk, equal_var=False)
print("Sales", round(sales.mean()), "vs Marketing", round(mk.mean()), " p =", round(p,4))
print("   -> also real, but you could not have guessed by eye")

Output

Tech 82419 vs HR 56705 p = 1.1560215947861482e-15 -> real difference Sales 66535 vs Marketing 73954 p = 0.0086 -> also real, but you could not have guessed by eye

Chi-square, and a result that finds nothing

Why it exists

Most courses only show tests that succeed, which teaches students that a test confirms what they hoped. Usually it does not.

In short

Compares a table of counts against what you would expect if the two categories were unrelated. A high p-value means no evidence of a link - NOT proof there is none.

Code

table = pd.crosstab(clean['department'], clean['rating'])
chi2, p, dof, exp = stats.chi2_contingency(table)
print(table)
print()
print("chi2 =", round(chi2,2), " p =", round(p,4))
print("   -> NO evidence that rating depends on department")
print("   (say 'no evidence of a link', not 'proved they are unrelated')")

Output

rating Average Excellent Good Poor department HR 8 13 11 12 Marketing 11 9 8 14 Sales 15 14 19 17 Tech 8 12 12 14 chi2 = 3.59 p = 0.9364 -> NO evidence that rating depends on department (say 'no evidence of a link', not 'proved they are unrelated')

A/B testing

Why it exists

The one application you will meet in a job interview and in real work.

In short

Show version A to one group, B to another, count conversions, and test the difference. It is a chi-square on a 2x2 table - there is no new statistics in it at all.

Code

import numpy as np
big   = np.array([[1104, 96], [1062, 138]])     # 1200 each, 8.0% vs 11.5%
small = np.array([[110, 10], [106, 14]])        # 120 each, same rates
for label, tab in (("1200 per arm", big), ("120 per arm", small)):
    chi2, p, dof, exp = stats.chi2_contingency(tab, correction=False)
    print(label, " p =", round(p,4), " ->",
          "ship B" if p < 0.05 else "cannot see it")
print()
print("SAME lift both times. only the amount of data changed.")
print("so size the test BEFORE you launch, and do not stop early.")

Output

1200 per arm p = 0.0039 -> ship B 120 per arm p = 0.3894 -> cannot see it SAME lift both times. only the amount of data changed. so size the test BEFORE you launch, and do not stop early.

p-hacking

Why it exists

The habit that separates an analyst people trust from one they do not.

In short

At a 5% threshold, one test in twenty comes back significant when nothing is there - by design. Test everything against everything and you WILL find something. It will not be real.

Code

import numpy as np
rng = np.random.default_rng(13)
hits = []
for i in range(20):
    a = rng.normal(0, 1, 50); b = rng.normal(0, 1, 50)   # identical sources
    p = stats.ttest_ind(a, b)[1]
    if p < 0.05: hits.append((i+1, round(p, 4)))
print("20 tests on data with NO real difference")
print("came back 'significant':", hits)
print()
print("nothing malfunctioned. 1-in-20 is what 5% MEANS.")
print("decide what you are testing BEFORE you look.")

Output

20 tests on data with NO real difference came back 'significant': [(18, np.float64(0.0057))] nothing malfunctioned. 1-in-20 is what 5% MEANS. decide what you are testing BEFORE you look.