TAP Academy TAP Academy Data Analytics

Quick reference

Python, NumPy
and Pandas

Every function from the course, grouped by the class it was taught in. Each one has a short explanation and a working example, and the output shown is the real result of running that code.

260FUNCTIONS
85PYTHON
50NUMPY
77PANDAS
48SQL

Python

Sessions D02 to D25. The language itself — types, strings, collections, loops, functions, classes and files.

85 functions

Introduction — D02 3

print()

Shows something on the screen.

print("Hello")
print("Total:", 25)
print("a", "b", sep="-")
Output
Hello
Total: 25
a-b

input()

Takes typed text from the user. Always returns a string.

# name = input("Enter name: ")
# Whatever the user types comes back as TEXT
name = "Rahul"          # pretend this came from input()
print(name, type(name))
Output
Rahul <class 'str'>

int() on input

Converts the text from input into a number so you can do maths.

age_text = "25"                 # input() always gives text
age = int(age_text)
print(age + 5)
Output
30

Data types — D03 7

type()

Tells you what kind of value something is.

print(type(10))
print(type(3.5))
print(type("hi"))
print(type(True))
Output
<class 'int'>
<class 'float'>
<class 'str'>
<class 'bool'>

int()

Converts a value into a whole number. Cuts off the decimal.

print(int("42"))
print(int(9.87))
Output
42
9

float()

Converts a value into a decimal number.

print(float("3.5"))
print(float(10))
Output
3.5
10.0

str()

Converts any value into text.

print(str(99) + " marks")
Output
99 marks

bool()

Converts a value into True or False. Empty things are False.

print(bool(1), bool(0))
print(bool("hi"), bool(""))
print(bool([]), bool([1]))
Output
True False
True False
False True

round()

Rounds a decimal number.

print(round(3.7))
print(round(3.14159, 2))
Output
4
3.14

abs()

Gives the value without the minus sign.

print(abs(-15))
print(abs(15))
Output
15
15

Operators — D04 to D05 3

Arithmetic

Basic maths. // gives whole division, % gives remainder, ** is power.

a, b = 17, 5
print(a + b, a - b, a * b)
print(a / b)      # normal division -> decimal
print(a // b)     # floor division  -> whole number
print(a % b)      # remainder
print(a ** 2)     # power
Output
22 12 85
3.4
3
2
289

Comparison

Compares two values. Always gives True or False.

print(5 > 3, 5 == 5, 5 != 3)
print(5 <= 4)
Output
True True True
False

and / or / not

Joins conditions together.

age = 22
print(age > 18 and age < 30)
print(age < 18 or age > 20)
print(not (age == 22))
Output
True
True
False

Strings — D11 14

len()

Counts how many characters are in the text.

s = "Data Analytics"
print(len(s))
Output
14

upper() / lower()

Changes the case of the text.

s = "Data"
print(s.upper())
print(s.lower())
Output
DATA
data

title() / capitalize()

title() capitalises every word. capitalize() only the first.

s = "rahul kumar sharma"
print(s.title())
print(s.capitalize())
Output
Rahul Kumar Sharma
Rahul kumar sharma

strip()

Removes spaces from the start and end.

s = "   Rahul   "
print("[" + s.strip() + "]")
print("[" + s.lstrip() + "]")
print("[" + s.rstrip() + "]")
Output
[Rahul]
[Rahul   ]
[   Rahul]

split()

Breaks text into a list. Splits on spaces by default.

s = "Rahul,Priya,Amit"
print(s.split(","))
print("a b c".split())
Output
['Rahul', 'Priya', 'Amit']
['a', 'b', 'c']

join()

Joins a list of text into one string.

names = ["Rahul", "Priya", "Amit"]
print(", ".join(names))
print("-".join("abc"))
Output
Rahul, Priya, Amit
a-b-c

replace()

Swaps one piece of text for another.

s = "I like tea"
print(s.replace("tea", "coffee"))
Output
I like coffee

find() / index()

Finds where a piece of text starts. find() gives -1 if missing, index() gives an error.

s = "Data Analytics"
print(s.find("Analy"))
print(s.find("Python"))
Output
5
-1

count()

Counts how many times something appears.

s = "banana"
print(s.count("a"))
Output
3

startswith() / endswith()

Checks how the text begins or ends.

f = "report.csv"
print(f.endswith(".csv"))
print(f.startswith("rep"))
Output
True
True

isdigit() / isalpha()

Checks what the text is made of.

print("12345".isdigit())
print("abc".isalpha())
print("abc123".isalpha())
Output
True
True
False

Slicing

Takes a part of the text using [start:stop:step].

s = "Data Analytics"
print(s[0])
print(s[0:4])
print(s[5:])
print(s[-1])
print(s[::-1])   # reversed
Output
D
Data
Analytics
s
scitylanA ataD

f-strings

Puts variables inside text. The easiest way to build a message.

name, marks = "Rahul", 87
print(f"{name} scored {marks} marks")
print(f"Average: {87/3:.2f}")
Output
Rahul scored 87 marks
Average: 29.00

format()

The older way of putting variables inside text.

print("{} scored {}".format("Rahul", 87))
Output
Rahul scored 87

Lists — D12 to D13 14

Creating a list

A list holds many values in order. It can be changed.

nums = [10, 20, 30]
mixed = [1, "two", 3.0, True]
empty = []
print(nums, mixed, empty)
Output
[10, 20, 30] [1, 'two', 3.0, True] []

Indexing and slicing

Get one item, or a part of the list.

nums = [10, 20, 30, 40, 50]
print(nums[0])
print(nums[-1])
print(nums[1:4])
print(nums[::2])
Output
10
50
[20, 30, 40]
[10, 30, 50]

append()

Adds ONE item to the end.

nums = [1, 2]
nums.append(3)
print(nums)
nums.append([4, 5])      # adds the whole list as one item
print(nums)
Output
[1, 2, 3]
[1, 2, 3, [4, 5]]

extend()

Adds MANY items to the end, one by one.

nums = [1, 2]
nums.extend([3, 4])
print(nums)
Output
[1, 2, 3, 4]

insert()

Adds an item at a chosen position.

nums = [10, 30]
nums.insert(1, 20)
print(nums)
Output
[10, 20, 30]

remove()

Removes the first matching value.

nums = [10, 20, 30, 20]
nums.remove(20)
print(nums)
Output
[10, 30, 20]

pop()

Removes by position and gives the item back. Last item if you give nothing.

nums = [10, 20, 30]
last = nums.pop()
first = nums.pop(0)
print(last, first, nums)
Output
30 10 [20]

sort()

Arranges the list in order. Changes the original list.

nums = [30, 10, 20]
nums.sort()
print(nums)
nums.sort(reverse=True)
print(nums)
Output
[10, 20, 30]
[30, 20, 10]

sorted()

Same as sort but gives a NEW list. The original stays as it was.

nums = [30, 10, 20]
new = sorted(nums)
print(new)
print(nums)      # unchanged
Output
[10, 20, 30]
[30, 10, 20]

reverse()

Flips the order of the list.

nums = [1, 2, 3]
nums.reverse()
print(nums)
Output
[3, 2, 1]

count() / index()

Counts a value, or finds its position.

nums = [10, 20, 10, 30]
print(nums.count(10))
print(nums.index(20))
Output
2
1

len() / sum() / min() / max()

Quick summaries of a list of numbers.

nums = [10, 20, 30, 40]
print(len(nums), sum(nums))
print(min(nums), max(nums))
print(sum(nums) / len(nums))     # average
Output
4 100
10 40
25.0

copy()

Makes a separate copy. Without it, both names point to the same list.

a = [1, 2, 3]
b = a              # SAME list
c = a.copy()       # NEW list
a.append(4)
print(b)
print(c)
Output
[1, 2, 3, 4]
[1, 2, 3]

List comprehension

Builds a new list in one line. Very common in data work.

nums = [1, 2, 3, 4, 5]
squares = [n * n for n in nums]
evens   = [n for n in nums if n % 2 == 0]
print(squares)
print(evens)
Output
[1, 4, 9, 16, 25]
[2, 4]

Tuples — D13 3

Creating a tuple

Like a list, but it CANNOT be changed after you make it.

t = (10, 20, 30)
print(t, len(t), t[0])
Output
(10, 20, 30) 3 10

Why tuples

Trying to change one gives an error. That safety is the point.

t = (1, 2, 3)
try:
    t[0] = 99
except TypeError as e:
    print("Error:", e)
Output
Error: 'tuple' object does not support item assignment

Unpacking

Puts each item into its own variable.

point = (10, 20)
x, y = point
print(x, y)
Output
10 20

Dictionaries — D16 7

Creating a dictionary

Stores key–value pairs. You look things up by name, not by position.

student = {"name": "Rahul", "marks": 87, "city": "Bangalore"}
print(student)
print(student["name"])
Output
{'name': 'Rahul', 'marks': 87, 'city': 'Bangalore'}
Rahul

Adding and changing

Assign to a key to add it or update it.

d = {"a": 1}
d["b"] = 2          # add
d["a"] = 99         # change
print(d)
Output
{'a': 99, 'b': 2}

get()

Safely reads a key. Gives None (or your default) instead of an error.

d = {"a": 1}
print(d.get("a"))
print(d.get("z"))
print(d.get("z", 0))
Output
1
None
0

keys() / values() / items()

Gets the keys, the values, or both together.

d = {"a": 1, "b": 2}
print(list(d.keys()))
print(list(d.values()))
print(list(d.items()))
Output
['a', 'b']
[1, 2]
[('a', 1), ('b', 2)]

Looping through a dictionary

items() gives you the key and value together.

marks = {"Rahul": 87, "Priya": 92}
for name, m in marks.items():
    print(name, "scored", m)
Output
Rahul scored 87
Priya scored 92

pop() / del

Removes a key.

d = {"a": 1, "b": 2}
d.pop("a")
print(d)
Output
{'b': 2}

update()

Merges another dictionary in.

d = {"a": 1}
d.update({"b": 2, "c": 3})
print(d)
Output
{'a': 1, 'b': 2, 'c': 3}

Sets — D14 4

Creating a set

Holds only UNIQUE values. Order is not kept.

s = {1, 2, 2, 3, 3, 3}
print(s)
print(set([10, 20, 10]))
Output
{1, 2, 3}
{10, 20}

add() / remove()

Adds or removes one value.

s = {1, 2}
s.add(3)
s.remove(1)
print(s)
Output
{2, 3}

union / intersection / difference

Compares two sets.

a = {1, 2, 3}
b = {3, 4, 5}
print(a | b)      # union - everything
print(a & b)      # intersection - common
print(a - b)      # difference - only in a
Output
{1, 2, 3, 4, 5}
{3}
{1, 2}

Removing duplicates

The most common real use of a set.

names = ["Rahul", "Priya", "Rahul", "Amit"]
print(list(set(names)))
Output
['Amit', 'Priya', 'Rahul']

Conditional statements — D08 2

if / elif / else

Runs different code depending on the answer.

marks = 78
if marks >= 90:
    print("A grade")
elif marks >= 75:
    print("B grade")
else:
    print("C grade")
Output
B grade

in

Checks whether a value is present.

names = ["Rahul", "Priya"]
print("Rahul" in names)
print("Amit" in names)
print("ta" in "Data")
Output
True
False
True

Loops — D09 to D10 6

for loop

Repeats once for every item.

for name in ["Rahul", "Priya"]:
    print("Hello", name)
Output
Hello Rahul
Hello Priya

range()

Makes a sequence of numbers to loop over.

print(list(range(5)))
print(list(range(2, 6)))
print(list(range(0, 10, 2)))
for i in range(3):
    print(i)
Output
[0, 1, 2, 3, 4]
[2, 3, 4, 5]
[0, 2, 4, 6, 8]
0
1
2

while loop

Keeps repeating while the condition stays true.

count = 1
while count <= 3:
    print(count)
    count = count + 1
Output
1
2
3

break / continue

break stops the loop. continue skips to the next round.

for i in range(6):
    if i == 4:
        break
    if i % 2 == 0:
        continue
    print(i)
Output
1
3

enumerate()

Gives the position AND the item together.

names = ["Rahul", "Priya"]
for i, name in enumerate(names):
    print(i, name)
Output
0 Rahul
1 Priya

zip()

Loops through two lists side by side.

names = ["Rahul", "Priya"]
marks = [87, 92]
for n, m in zip(names, marks):
    print(n, m)
Output
Rahul 87
Priya 92

Functions — D06 to D07 8

def

Creates your own reusable block of code.

def greet(name):
    return "Hello " + name

print(greet("Rahul"))
Output
Hello Rahul

return

Sends a value back. Without it you get None.

def add(a, b):
    return a + b

def add_no_return(a, b):
    total = a + b        # nothing sent back

print(add(2, 3))
print(add_no_return(2, 3))
Output
5
None

Default values

Gives a parameter a fallback if nothing is passed.

def greet(name, greeting="Hello"):
    return greeting + " " + name

print(greet("Rahul"))
print(greet("Priya", "Good morning"))
Output
Hello Rahul
Good morning Priya

*args

Accepts any number of values.

def total(*nums):
    return sum(nums)

print(total(1, 2))
print(total(1, 2, 3, 4, 5))
Output
3
15

**kwargs

Accepts any number of named values.

def show(**info):
    for k, v in info.items():
        print(k, "=", v)

show(name="Rahul", city="Bangalore")
Output
name = Rahul
city = Bangalore

lambda

A tiny one-line function with no name.

square = lambda x: x * x
print(square(5))

pairs = [(1, "b"), (3, "a"), (2, "c")]
print(sorted(pairs, key=lambda p: p[1]))
Output
25
[(3, 'a'), (1, 'b'), (2, 'c')]

map()

Applies a function to every item.

nums = [1, 2, 3]
print(list(map(lambda x: x * 10, nums)))
print(list(map(str, nums)))
Output
[10, 20, 30]
['1', '2', '3']

filter()

Keeps only the items that pass a test.

nums = [1, 2, 3, 4, 5, 6]
print(list(filter(lambda x: x % 2 == 0, nums)))
Output
[2, 4, 6]

File handling — D23 to D25 4

open() with write

Creates a file and writes into it.

with open("demo.txt", "w") as f:
    f.write("Line one\n")
    f.write("Line two\n")
print("file written")
Output
file written

open() with read

Reads the whole file as one string.

with open("demo.txt", "w") as f:
    f.write("Line one\nLine two\n")

with open("demo.txt", "r") as f:
    print(f.read())
Output
Line one
Line two

readlines()

Reads the file into a list of lines.

with open("demo.txt", "w") as f:
    f.write("a\nb\nc\n")

with open("demo.txt") as f:
    lines = f.readlines()
print(lines)
print([l.strip() for l in lines])
Output
['a\n', 'b\n', 'c\n']
['a', 'b', 'c']

Append mode

"a" adds to the end instead of erasing the file.

with open("demo.txt", "w") as f:
    f.write("first\n")
with open("demo.txt", "a") as f:
    f.write("second\n")
with open("demo.txt") as f:
    print(f.read())
Output
first
second

Error handling — D24 3

try / except

Catches an error so the program does not stop.

try:
    x = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero")
Output
Cannot divide by zero

except with the message

Shows what actually went wrong.

try:
    n = int("abc")
except ValueError as e:
    print("Error:", e)
Output
Error: invalid literal for int() with base 10: 'abc'

finally

Runs whether there was an error or not.

try:
    print("trying")
except:
    print("failed")
finally:
    print("always runs")
Output
trying
always runs

Object orientation — D17 to D22 3

class and __init__

A blueprint for making objects. __init__ runs when you create one.

class Student:
    def __init__(self, name, marks):
        self.name = name
        self.marks = marks

s = Student("Rahul", 87)
print(s.name, s.marks)
Output
Rahul 87

Methods

A function that belongs to the class.

class Student:
    def __init__(self, name, marks):
        self.name = name
        self.marks = marks
    def grade(self):
        return "Pass" if self.marks >= 40 else "Fail"

s = Student("Rahul", 87)
print(s.grade())
Output
Pass

Inheritance

A new class takes everything from an existing one.

class Person:
    def __init__(self, name):
        self.name = name
    def greet(self):
        return "Hi, I am " + self.name

class Student(Person):
    def study(self):
        return self.name + " is studying"

s = Student("Rahul")
print(s.greet())
print(s.study())
Output
Hi, I am Rahul
Rahul is studying

Modules and libraries — D25 4

import

Brings in code someone else wrote.

import math
print(math.sqrt(16))
print(math.pi)
Output
4.0
3.141592653589793

from ... import

Brings in just one piece.

from math import sqrt, floor
print(sqrt(25), floor(3.9))
Output
5.0 3

import as

Gives the module a short name. This is why pandas is pd.

import math as m
print(m.sqrt(9))
Output
3.0

random

Makes random numbers.

import random
random.seed(1)               # seed makes it repeatable
print(random.randint(1, 10))
print(random.choice(["a", "b", "c"]))
Output
3
c

NumPy

Sessions D25 to D30. Arrays, indexing, aggregation, boolean slicing and reshaping.

50 functions

NumPy introduction — D25 to D26 11

np.array()

Turns a list into a NumPy array.

a = np.array([10, 20, 30])
print(a)
print(type(a))
Output
[10 20 30]
<class 'numpy.ndarray'>

2D array

A list of lists becomes a table with rows and columns.

b = np.array([[1, 2, 3],
              [4, 5, 6]])
print(b)
print(b.shape)     # (rows, columns)
Output
[[1 2 3]
 [4 5 6]]
(2, 3)

np.zeros()

Makes an array filled with 0.

print(np.zeros(4))
print(np.zeros((2, 3)))
Output
[0. 0. 0. 0.]
[[0. 0. 0.]
 [0. 0. 0.]]

np.ones()

Makes an array filled with 1.

print(np.ones(3))
print(np.ones((2, 2)))
Output
[1. 1. 1.]
[[1. 1.]
 [1. 1.]]

np.full()

Makes an array filled with any value you choose.

print(np.full(4, 7))
print(np.full((2, 2), 9))
Output
[7 7 7 7]
[[9 9]
 [9 9]]

np.eye() / np.identity()

Makes an identity matrix - 1 on the diagonal, 0 everywhere else.

print(np.eye(3))
print(np.identity(2))
Output
[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
[[1. 0.]
 [0. 1.]]

np.arange()

Makes numbers in a range. Like range(), but gives an array.

print(np.arange(5))
print(np.arange(2, 10, 2))
Output
[0 1 2 3 4]
[2 4 6 8]

np.linspace()

Makes a fixed COUNT of evenly spaced numbers between two values.

print(np.linspace(0, 10, 5))
print(np.linspace(0, 1, 3))
Output
[ 0.   2.5  5.   7.5 10. ]
[0.  0.5 1. ]

np.random.rand()

Random decimals between 0 and 1.

np.random.seed(1)
print(np.random.rand(3))
print(np.random.rand(2, 2))
Output
[4.17022005e-01 7.20324493e-01 1.14374817e-04]
[[0.30233257 0.14675589]
 [0.09233859 0.18626021]]

np.random.randint()

Random whole numbers in a range.

np.random.seed(1)
print(np.random.randint(1, 100, 5))
Output
[38 13 73 10 76]

np.random.seed()

Fixes the randomness so you get the same numbers every time.

np.random.seed(42)
print(np.random.randint(1, 10, 4))
np.random.seed(42)
print(np.random.randint(1, 10, 4))     # same again
Output
[7 4 8 5]
[7 4 8 5]

Array attributes — D26 to D27 5

shape

Rows and columns.

a = np.array([[1, 2, 3], [4, 5, 6]])
print(a.shape)
print(np.array([1,2,3,4]).shape)
Output
(2, 3)
(4,)

ndim

How many dimensions - 1 for a line, 2 for a table.

print(np.array([1, 2, 3]).ndim)
print(np.array([[1, 2], [3, 4]]).ndim)
Output
1
2

size

Total number of items.

a = np.array([[1, 2, 3], [4, 5, 6]])
print(a.size)
Output
6

dtype

What kind of numbers are inside.

print(np.array([1, 2, 3]).dtype)
print(np.array([1.5, 2.5]).dtype)
print(np.array(["a", "b"]).dtype)
Output
int64
float64
<U1

astype()

Changes the data type.

a = np.array([1.7, 2.9, 3.1])
print(a.astype(int))
print(np.array([1, 0, 1]).astype(bool))
Output
[1 2 3]
[ True False  True]

Indexing and slicing — D28 7

Indexing 1D

Get one value by its position. Counting starts at 0.

a = np.array([10, 20, 30, 40])
print(a[0])
print(a[-1])
Output
10
40

Slicing 1D

Take a part using [start:stop].

a = np.array([10, 20, 30, 40, 50])
print(a[1:4])
print(a[:3])
print(a[::2])
Output
[20 30 40]
[10 20 30]
[10 30 50]

Indexing 2D

Use [row, column].

b = np.array([[1, 2, 3],
              [4, 5, 6]])
print(b[0, 2])
print(b[1, 0])
Output
3
4

Slicing 2D

A colon means "everything" in that direction.

b = np.array([[1, 2, 3],
              [4, 5, 6]])
print(b[0, :])      # first row
print(b[:, 1])      # second column
print(b[:, 0:2])    # first two columns
Output
[1 2 3]
[2 5]
[[1 2]
 [4 5]]

Boolean mask

Keeps only the values where the condition is True.

a = np.array([10, 25, 3, 40, 7])
print(a > 20)
print(a[a > 20])
Output
[False  True False  True False]
[25 40]

Multiple conditions

Use & for and, | for or. Each condition needs brackets.

a = np.array([10, 25, 3, 40, 7])
print(a[(a > 5) & (a < 30)])
print(a[(a < 5) | (a > 35)])
Output
[10 25  7]
[ 3 40]

Fancy indexing

Pick several positions at once using a list.

a = np.array([10, 20, 30, 40, 50])
print(a[[0, 2, 4]])
Output
[10 30 50]

Reshaping — D30 4

reshape()

Rearranges the same values into a new shape.

a = np.arange(6)
print(a)
print(a.reshape(2, 3))
print(a.reshape(3, 2))
Output
[0 1 2 3 4 5]
[[0 1 2]
 [3 4 5]]
[[0 1]
 [2 3]
 [4 5]]

reshape(-1, 1)

-1 means "work it out for me".

a = np.arange(6)
print(a.reshape(-1, 2))
print(a.reshape(2, -1))
Output
[[0 1]
 [2 3]
 [4 5]]
[[0 1 2]
 [3 4 5]]

ravel() / flatten()

Turns a table back into a single line.

b = np.array([[1, 2], [3, 4]])
print(b.ravel())
print(b.flatten())
Output
[1 2 3 4]
[1 2 3 4]

T (transpose)

Flips rows into columns.

b = np.array([[1, 2, 3], [4, 5, 6]])
print(b)
print(b.T)
Output
[[1 2 3]
 [4 5 6]]
[[1 4]
 [2 5]
 [3 6]]

Stacking arrays — D30 4

np.concatenate()

Joins arrays together.

a = np.array([1, 2])
b = np.array([3, 4])
print(np.concatenate([a, b]))
Output
[1 2 3 4]

np.vstack()

Stacks arrays on top of each other (vertically).

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.vstack([a, b]))
Output
[[1 2 3]
 [4 5 6]]

np.hstack()

Puts arrays side by side (horizontally).

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.hstack([a, b]))
Output
[1 2 3 4 5 6]

np.split()

Cuts one array into several pieces.

a = np.arange(6)
print(np.split(a, 3))
Output
[array([0, 1]), array([2, 3]), array([4, 5])]

Operations and broadcasting — D27 5

Arithmetic

Maths happens on every value at once. No loop needed.

a = np.array([10, 20, 30])
print(a + 5)
print(a * 2)
print(a / 10)
print(a ** 2)
Output
[15 25 35]
[20 40 60]
[1. 2. 3.]
[100 400 900]

Array with array

Two arrays of the same shape work position by position.

a = np.array([1, 2, 3])
b = np.array([10, 20, 30])
print(a + b)
print(a * b)
Output
[11 22 33]
[10 40 90]

np.sqrt() / np.exp() / np.log()

Common maths functions applied to every value.

a = np.array([1, 4, 9, 16])
print(np.sqrt(a))
print(np.round(np.log(np.array([1.0, 2.718])), 2))
Output
[1. 2. 3. 4.]
[0. 1.]

np.round() / floor / ceil

Rounding. floor goes down, ceil goes up.

a = np.array([1.2, 2.7, 3.5])
print(np.round(a))
print(np.floor(a))
print(np.ceil(a))
Output
[1. 3. 4.]
[1. 2. 3.]
[2. 3. 4.]

Broadcasting

A small array stretches automatically to fit a bigger one.

b = np.array([[1, 2, 3],
              [4, 5, 6]])
print(b + np.array([10, 20, 30]))
print(b * 2)
Output
[[11 22 33]
 [14 25 36]]
[[ 2  4  6]
 [ 8 10 12]]

Aggregate functions — D28 8

sum()

Adds everything up.

a = np.array([10, 20, 30])
print(a.sum())
print(np.sum(a))
Output
60
60

mean() / median()

The average and the middle value.

a = np.array([10, 20, 30, 100])
print(a.mean())
print(np.median(a))
Output
40.0
25.0

std() / var()

How spread out the numbers are.

a = np.array([10, 20, 30, 40])
print(round(a.std(), 2))
print(a.var())
Output
11.18
125.0

min() / max()

The smallest and largest value.

a = np.array([10, 5, 30])
print(a.min(), a.max())
Output
5 30

argmin() / argmax()

The POSITION of the smallest or largest value.

a = np.array([10, 5, 30])
print(a.argmin())
print(a.argmax())
Output
1
2

axis=0 and axis=1

axis=0 goes down the columns. axis=1 goes across the rows.

b = np.array([[1, 2, 3],
              [4, 5, 6]])
print(b.sum())          # everything
print(b.sum(axis=0))    # down each column
print(b.sum(axis=1))    # across each row
Output
21
[5 7 9]
[ 6 15]

cumsum()

A running total.

a = np.array([10, 20, 30])
print(a.cumsum())
Output
[10 30 60]

prod()

Multiplies everything together.

a = np.array([1, 2, 3, 4])
print(a.prod())
Output
24

Boolean slicing and sorting — D29 6

np.where()

Picks one value or another depending on a condition.

a = np.array([10, 25, 3, 40])
print(np.where(a > 20, "high", "low"))
print(np.where(a > 20, a, 0))
Output
['low' 'high' 'low' 'high']
[ 0 25  0 40]

np.sort()

Arranges values in order.

a = np.array([30, 10, 20])
print(np.sort(a))
Output
[10 20 30]

np.argsort()

Gives the POSITIONS that would sort the array.

a = np.array([30, 10, 20])
print(np.argsort(a))
Output
[1 2 0]

np.unique()

Gives the distinct values, in order.

a = np.array([3, 1, 2, 3, 1])
print(np.unique(a))
print(np.unique(a, return_counts=True))
Output
[1 2 3]
(array([1, 2, 3]), array([2, 1, 2]))

any() / all()

any = at least one True. all = every one True.

a = np.array([10, 25, 3])
print((a > 20).any())
print((a > 20).all())
Output
True
False

np.isnan()

Finds missing values.

a = np.array([1.0, np.nan, 3.0])
print(np.isnan(a))
print(a[~np.isnan(a)])      # ~ means NOT
Output
[False  True False]
[1. 3.]

Pandas

Sessions D31 to D37. Series and DataFrames, filtering, sorting, missing values, duplicates and merging.

77 functions

Series and DataFrame — D31 to D32 4

pd.Series()

A Series is ONE column of data with an index.

s = pd.Series([10, 20, 30])
print(s)
print(s[0], s.sum())
Output
0    10
1    20
2    30
dtype: int64
10 60

Series with an index

You can give your own labels instead of 0,1,2.

s = pd.Series([87, 92], index=['Rahul', 'Priya'])
print(s)
print(s['Rahul'])
Output
Rahul    87
Priya    92
dtype: int64
87

pd.DataFrame() from a dictionary

A DataFrame is a full table. Each key becomes a column.

data = {'name': ['Rahul', 'Priya'], 'marks': [87, 92]}
d = pd.DataFrame(data)
print(d)
Output
    name  marks
0  Rahul     87
1  Priya     92

pd.DataFrame() from a list of lists

Give the column names separately.

rows = [['Rahul', 87], ['Priya', 92]]
d = pd.DataFrame(rows, columns=['name', 'marks'])
print(d)
Output
    name  marks
0  Rahul     87
1  Priya     92

Reading files — D31 3

pd.read_csv()

Loads a CSV file into a DataFrame. The most used function in pandas.

df.to_csv('demo.csv', index=False)      # make a file first
d = pd.read_csv('demo.csv')
print(d.head(3))
Output
    name   dept       city   age  exp  salary     rating
0  Rahul   Tech  Bangalore  28.0    4   72000       Good
1  Priya  Sales    Chennai  34.0   10   95000  Excellent
2   Amit   Tech  Bangalore  25.0    2   58000       Poor

read_csv() useful options

Common settings you will need.

df.to_csv('demo.csv', index=False)
print(pd.read_csv('demo.csv', nrows=2))
print(pd.read_csv('demo.csv', usecols=['name','salary']).head(2))
Output
    name   dept       city   age  exp  salary     rating
0  Rahul   Tech  Bangalore  28.0    4   72000       Good
1  Priya  Sales    Chennai  34.0   10   95000  Excellent
    name  salary
0  Rahul   72000
1  Priya   95000

to_csv()

Saves a DataFrame to a CSV file. index=False stops the extra column.

df.head(2).to_csv('out.csv', index=False)
print(open('out.csv').read())
Output
name,dept,city,age,exp,salary,rating
Rahul,Tech,Bangalore,28.0,4,72000,Good
Priya,Sales,Chennai,34.0,10,95000,Excellent

EDA and inspection — D32 to D33 7

head() / tail()

Shows the first or last few rows. Always start here.

print(df.head(3))
print(df.tail(2))
Output
    name   dept       city   age  exp  salary     rating
0  Rahul   Tech  Bangalore  28.0    4   72000       Good
1  Priya  Sales    Chennai  34.0   10   95000  Excellent
2   Amit   Tech  Bangalore  25.0    2   58000       Poor
     name   dept     city   age  exp  salary   rating
4  Vikram  Sales  Chennai  30.0    6   68000  Average
5   Divya   Tech     Pune   NaN    8   88000     Good

shape

How many rows and columns.

print(df.shape)
print("rows:", df.shape[0], "columns:", df.shape[1])
Output
(6, 7)
rows: 6 columns: 7

info()

A summary: column names, how many values are filled, and the data types.

df.info()
Output
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6 entries, 0 to 5
Data columns (total 7 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   name    6 non-null      object 
 1   dept    6 non-null      object 
 2   city    6 non-null      object 
 3   age     5 non-null      float64
 4   exp     6 non-null      int64  
 5   salary  6 non-null      int64  
 6   rating  6 non-null      object 
dtypes: float64(1), int64(2), object(4)
memory usage: 464.0+ bytes

describe()

Summary statistics for the number columns.

print(df.describe())
Output
             age        exp         salary
count   5.000000   6.000000       6.000000
mean   31.600000   7.500000   83500.000000
std     6.188699   4.636809   22394.195677
min    25.000000   2.000000   58000.000000
25%    28.000000   4.500000   69000.000000
50%    30.000000   7.000000   80000.000000
75%    34.000000   9.500000   93250.000000
max    41.000000  15.000000  120000.000000

columns / index / dtypes

The column names, the row labels, and the data types.

print(list(df.columns))
print(df.dtypes)
Output
['name', 'dept', 'city', 'age', 'exp', 'salary', 'rating']
name       object
dept       object
city       object
age       float64
exp         int64
salary      int64
rating     object
dtype: object

values

The data as a plain NumPy array, without labels.

print(df[['exp','salary']].values)
Output
[[     4  72000]
 [    10  95000]
 [     2  58000]
 [    15 120000]
 [     6  68000]
 [     8  88000]]

sample()

Shows random rows instead of the first few.

print(df.sample(2, random_state=1))
Output
    name   dept       city   age  exp  salary     rating
2   Amit   Tech  Bangalore  25.0    2   58000       Poor
1  Priya  Sales    Chennai  34.0   10   95000  Excellent

Selecting rows and columns — D32 to D33 6

Selecting one column

One pair of brackets gives a Series.

print(df['name'])
Output
0     Rahul
1     Priya
2      Amit
3     Sneha
4    Vikram
5     Divya
Name: name, dtype: object

Selecting several columns

Two pairs of brackets give a DataFrame.

print(df[['name', 'salary']])
Output
     name  salary
0   Rahul   72000
1   Priya   95000
2    Amit   58000
3   Sneha  120000
4  Vikram   68000
5   Divya   88000

loc[]

Selects by LABEL. Row labels and column names.

print(df.loc[0])
print(df.loc[0:2, ['name','salary']])
Output
name          Rahul
dept           Tech
city      Bangalore
age            28.0
exp               4
salary        72000
rating         Good
Name: 0, dtype: object
    name  salary
0  Rahul   72000
1  Priya   95000
2   Amit   58000

iloc[]

Selects by POSITION. Numbers only.

print(df.iloc[0])
print(df.iloc[0:3, 0:2])
print(df.iloc[-1])
Output
name          Rahul
dept           Tech
city      Bangalore
age            28.0
exp               4
salary        72000
rating         Good
Name: 0, dtype: object
    name   dept
0  Rahul   Tech
1  Priya  Sales
2   Amit   Tech
name      Divya
dept       Tech
city       Pune
age         NaN
exp           8
salary    88000
rating     Good
Name: 5, dtype: object

loc vs iloc

loc uses names and INCLUDES the end. iloc uses numbers and EXCLUDES it.

print(df.loc[0:2, 'name'])     # 3 rows - end included
print(df.iloc[0:2, 0])         # 2 rows - end excluded
Output
0    Rahul
1    Priya
2     Amit
Name: name, dtype: object
0    Rahul
1    Priya
Name: name, dtype: object

at[] / iat[]

Gets one single value. Faster than loc for one cell.

print(df.at[0, 'name'])
print(df.iat[0, 0])
Output
Rahul
Rahul

Filtering — D33 to D34 7

Boolean filter

Keeps only rows where the condition is True.

print(df[df['salary'] > 80000])
Output
    name   dept     city   age  exp  salary     rating
1  Priya  Sales  Chennai  34.0   10   95000  Excellent
3  Sneha     HR     Pune  41.0   15  120000  Excellent
5  Divya   Tech     Pune   NaN    8   88000       Good

Two conditions

& means and, | means or. Each condition needs brackets.

print(df[(df['dept'] == 'Tech') & (df['salary'] > 60000)])
print(df[(df['dept'] == 'HR') | (df['exp'] > 9)])
Output
    name  dept       city   age  exp  salary rating
0  Rahul  Tech  Bangalore  28.0    4   72000   Good
5  Divya  Tech       Pune   NaN    8   88000   Good
    name   dept     city   age  exp  salary     rating
1  Priya  Sales  Chennai  34.0   10   95000  Excellent
3  Sneha     HR     Pune  41.0   15  120000  Excellent

isin()

Keeps rows where the value is in your list.

print(df[df['city'].isin(['Chennai', 'Pune'])])
Output
     name   dept     city   age  exp  salary     rating
1   Priya  Sales  Chennai  34.0   10   95000  Excellent
3   Sneha     HR     Pune  41.0   15  120000  Excellent
4  Vikram  Sales  Chennai  30.0    6   68000    Average
5   Divya   Tech     Pune   NaN    8   88000       Good

between()

Keeps rows where a number falls in a range.

print(df[df['salary'].between(60000, 90000)])
Output
     name   dept       city   age  exp  salary   rating
0   Rahul   Tech  Bangalore  28.0    4   72000     Good
4  Vikram  Sales    Chennai  30.0    6   68000  Average
5   Divya   Tech       Pune   NaN    8   88000     Good

str.contains()

Keeps rows where text contains something.

print(df[df['name'].str.contains('a')])
Output
     name   dept       city   age  exp  salary     rating
0   Rahul   Tech  Bangalore  28.0    4   72000       Good
1   Priya  Sales    Chennai  34.0   10   95000  Excellent
3   Sneha     HR       Pune  41.0   15  120000  Excellent
4  Vikram  Sales    Chennai  30.0    6   68000    Average
5   Divya   Tech       Pune   NaN    8   88000       Good

query()

Filters using a readable sentence.

print(df.query('salary > 80000 and dept == "Sales"'))
Output
    name   dept     city   age  exp  salary     rating
1  Priya  Sales  Chennai  34.0   10   95000  Excellent

~ (not)

Flips the condition around.

print(df[~(df['dept'] == 'Tech')])
Output
     name   dept     city   age  exp  salary     rating
1   Priya  Sales  Chennai  34.0   10   95000  Excellent
3   Sneha     HR     Pune  41.0   15  120000  Excellent
4  Vikram  Sales  Chennai  30.0    6   68000    Average

Missing values — D35 4

isnull() / notnull()

Marks each cell True or False for missing.

print(df['age'].isnull())
print(df.isnull().sum())      # count per column
Output
0    False
1    False
2    False
3    False
4    False
5     True
Name: age, dtype: bool
name      0
dept      0
city      0
age       1
exp       0
salary    0
rating    0
dtype: int64

dropna()

Removes rows that have missing values.

print(df.shape)
print(df.dropna().shape)
print(df.dropna(subset=['age']).shape)
Output
(6, 7)
(5, 7)
(5, 7)

fillna()

Fills missing values with something.

print(df['age'].fillna(0).tolist())
print(df['age'].fillna(df['age'].mean()).round(1).tolist())
Output
[28.0, 34.0, 25.0, 41.0, 30.0, 0.0]
[28.0, 34.0, 25.0, 41.0, 30.0, 31.6]

Filling with the median

Usually safer than the mean when there are extreme values.

print(df['age'].fillna(df['age'].median()).tolist())
Output
[28.0, 34.0, 25.0, 41.0, 30.0, 30.0]

Duplicates — D36 2

duplicated()

Marks rows that are copies of an earlier row.

d = pd.DataFrame({'a':[1,2,2,3],'b':['x','y','y','z']})
print(d.duplicated())
print(d.duplicated().sum())
Output
0    False
1    False
2     True
3    False
dtype: bool
1

drop_duplicates()

Removes the repeated rows.

d = pd.DataFrame({'a':[1,2,2,3]})
print(d.drop_duplicates())
print(df.drop_duplicates(subset=['dept']))
Output
   a
0  1
1  2
3  3
    name   dept       city   age  exp  salary     rating
0  Rahul   Tech  Bangalore  28.0    4   72000       Good
1  Priya  Sales    Chennai  34.0   10   95000  Excellent
3  Sneha     HR       Pune  41.0   15  120000  Excellent

New columns and changes — D36 8

Adding a new column

Just assign to a new name.

d = df.copy()
d['bonus'] = d['salary'] * 0.10
print(d[['name','salary','bonus']].head(3))
Output
    name  salary   bonus
0  Rahul   72000  7200.0
1  Priya   95000  9500.0
2   Amit   58000  5800.0

rename()

Changes column names.

d = df.rename(columns={'exp': 'experience', 'dept': 'department'})
print(list(d.columns))
Output
['name', 'department', 'city', 'age', 'experience', 'salary', 'rating']

drop()

Removes columns or rows.

print(df.drop(columns=['city','rating']).head(2))
print(df.drop(index=[0,1]).shape)
Output
    name   dept   age  exp  salary
0  Rahul   Tech  28.0    4   72000
1  Priya  Sales  34.0   10   95000
(4, 7)

astype()

Changes the data type of a column.

d = df.dropna(subset=['age']).copy()
d['age'] = d['age'].astype(int)
print(d['age'].tolist())
Output
[28, 34, 25, 41, 30]

replace()

Swaps values for other values.

print(df['dept'].replace({'Tech':'Engineering'}).tolist())
Output
['Engineering', 'Sales', 'Engineering', 'HR', 'Sales', 'Engineering']

apply()

Runs your own function on every value.

print(df['salary'].apply(lambda x: x / 1000).tolist())

def band(s):
    return 'High' if s > 80000 else 'Low'
print(df['salary'].apply(band).tolist())
Output
[72.0, 95.0, 58.0, 120.0, 68.0, 88.0]
['Low', 'High', 'Low', 'High', 'Low', 'High']

map()

Replaces values using a dictionary. Good for encoding.

order = {'Poor':1, 'Average':2, 'Good':3, 'Excellent':4}
print(df['rating'].map(order).tolist())
Output
[3, 4, 1, 4, 2, 3]

cut()

Puts numbers into named bands.

bands = pd.cut(df['salary'], bins=[0,70000,100000,999999],
               labels=['Low','Medium','High'])
print(bands.tolist())
Output
['Medium', 'Medium', 'Low', 'High', 'Low', 'Medium']

Sorting — D34 5

sort_values()

Arranges rows by a column.

print(df.sort_values('salary')[['name','salary']])
Output
     name  salary
2    Amit   58000
4  Vikram   68000
0   Rahul   72000
5   Divya   88000
1   Priya   95000
3   Sneha  120000

sort_values() descending

ascending=False puts the largest first.

print(df.sort_values('salary', ascending=False)[['name','salary']].head(3))
Output
    name  salary
3  Sneha  120000
1  Priya   95000
5  Divya   88000

Sorting by two columns

Sorts by the first, then breaks ties with the second.

print(df.sort_values(['dept','salary'])[['dept','name','salary']])
Output
    dept    name  salary
3     HR   Sneha  120000
4  Sales  Vikram   68000
1  Sales   Priya   95000
2   Tech    Amit   58000
0   Tech   Rahul   72000
5   Tech   Divya   88000

nlargest() / nsmallest()

The top or bottom few rows, without sorting everything.

print(df.nlargest(3, 'salary')[['name','salary']])
print(df.nsmallest(2, 'salary')[['name','salary']])
Output
    name  salary
3  Sneha  120000
1  Priya   95000
5  Divya   88000
     name  salary
2    Amit   58000
4  Vikram   68000

sort_index()

Sorts by the row labels instead of a column.

print(df.sort_values('salary').sort_index().head(3)[['name','salary']])
Output
    name  salary
0  Rahul   72000
1  Priya   95000
2   Amit   58000

Summary statistics — D37 10

mean() / median()

The average and the middle value of a column.

print(df['salary'].mean())
print(df['salary'].median())
Output
83500.0
80000.0

mode()

The most common value. Works on text too.

print(df['dept'].mode()[0])
print(df['city'].mode().tolist())
Output
Tech
['Bangalore', 'Chennai', 'Pune']

std() / var()

How spread out the values are.

print(round(df['salary'].std(), 2))
print(round(df['salary'].var(), 2))
Output
22394.2
501500000.0

min() / max() / sum()

The smallest, largest and total.

print(df['salary'].min(), df['salary'].max())
print(df['salary'].sum())
Output
58000 120000
501000

count() / nunique()

How many values are filled, and how many are distinct.

print(df['age'].count())     # missing not counted
print(df['dept'].nunique())
Output
5
3

value_counts()

Counts how many times each value appears. Extremely useful.

print(df['dept'].value_counts())
print(df['dept'].value_counts(normalize=True).round(2))
Output
dept
Tech     3
Sales    2
HR       1
Name: count, dtype: int64
dept
Tech     0.50
Sales    0.33
HR       0.17
Name: proportion, dtype: float64

unique()

Lists the distinct values.

print(df['city'].unique())
Output
['Bangalore' 'Chennai' 'Pune']

quantile()

The value below which a share of the data falls.

print(df['salary'].quantile(0.25))
print(df['salary'].quantile([0.25, 0.5, 0.75]).tolist())
Output
69000.0
[69000.0, 80000.0, 93250.0]

corr()

How strongly two number columns move together. Between -1 and 1.

print(round(df['exp'].corr(df['salary']), 3))
print(df[['exp','salary']].corr().round(3))
Output
0.979
          exp  salary
exp     1.000   0.979
salary  0.979   1.000

cumsum()

A running total down the column.

print(df['salary'].cumsum().tolist())
Output
[72000, 167000, 225000, 345000, 413000, 501000]

Aggregate and group by — D37 6

groupby() with one function

Splits into groups, then summarises each group.

print(df.groupby('dept')['salary'].mean())
Output
dept
HR       120000.000000
Sales     81500.000000
Tech      72666.666667
Name: salary, dtype: float64

groupby() with count

How many rows in each group.

print(df.groupby('dept').size())
print(df.groupby('dept')['salary'].count())
Output
dept
HR       1
Sales    2
Tech     3
dtype: int64
dept
HR       1
Sales    2
Tech     3
Name: salary, dtype: int64

groupby() with agg()

Several summaries at once.

print(df.groupby('dept')['salary'].agg(['count','mean','max']))
Output
       count           mean     max
dept                               
HR         1  120000.000000  120000
Sales      2   81500.000000   95000
Tech       3   72666.666667   88000

agg() on different columns

Different summary for each column.

print(df.groupby('dept').agg({'salary':'mean', 'exp':'max'}))
Output
              salary  exp
dept                     
HR     120000.000000   15
Sales   81500.000000   10
Tech    72666.666667    8

Grouping by two columns

Groups within groups.

print(df.groupby(['dept','city'])['salary'].mean())
Output
dept   city     
HR     Pune         120000.0
Sales  Chennai       81500.0
Tech   Bangalore     65000.0
       Pune          88000.0
Name: salary, dtype: float64

reset_index()

Turns the group labels back into a normal column.

g = df.groupby('dept')['salary'].mean().reset_index()
print(g)
Output
    dept         salary
0     HR  120000.000000
1  Sales   81500.000000
2   Tech   72666.666667

Reshaping — not yet covered in class 4

pivot_table()

Summarises data into a grid. Like a pivot table in Excel.

print(pd.pivot_table(df, values='salary', index='dept',
                     columns='city', aggfunc='mean'))
Output
city   Bangalore  Chennai      Pune
dept                               
HR           NaN      NaN  120000.0
Sales        NaN  81500.0       NaN
Tech     65000.0      NaN   88000.0

crosstab()

Counts how often combinations appear.

print(pd.crosstab(df['dept'], df['city']))
Output
city   Bangalore  Chennai  Pune
dept                           
HR             0        0     1
Sales          0        2     0
Tech           2        0     1

melt()

Turns wide data into long data.

w = pd.DataFrame({'name':['Rahul','Priya'],'maths':[87,92],'science':[78,85]})
print(pd.melt(w, id_vars='name', var_name='subject', value_name='marks'))
Output
    name  subject  marks
0  Rahul    maths     87
1  Priya    maths     92
2  Rahul  science     78
3  Priya  science     85

set_index() / reset_index()

Makes a column the row label, or puts it back.

d = df.set_index('name')
print(d.loc['Rahul'])
print(d.reset_index().head(2)[['name','dept']])
Output
dept           Tech
city      Bangalore
age            28.0
exp               4
salary        72000
rating         Good
Name: Rahul, dtype: object
    name   dept
0  Rahul   Tech
1  Priya  Sales

Merging data frames — D37 4

concat()

Stacks tables on top of each other.

a = df.head(2)[['name','dept']]
b = df.tail(2)[['name','dept']]
print(pd.concat([a, b]))
Output
     name   dept
0   Rahul   Tech
1   Priya  Sales
4  Vikram  Sales
5   Divya   Tech

concat() side by side

axis=1 joins columns instead of rows.

a = df[['name']].head(3)
b = df[['salary']].head(3)
print(pd.concat([a, b], axis=1))
Output
    name  salary
0  Rahul   72000
1  Priya   95000
2   Amit   58000

merge()

Joins two tables on a shared column. Like a JOIN in SQL.

emp  = pd.DataFrame({'name':['Rahul','Priya'], 'dept_id':[1, 2]})
dept = pd.DataFrame({'dept_id':[1, 2], 'dept_name':['Tech','Sales']})
print(pd.merge(emp, dept, on='dept_id'))
Output
    name  dept_id dept_name
0  Rahul        1      Tech
1  Priya        2     Sales

merge() types

how= decides which rows are kept.

a = pd.DataFrame({'id':[1,2,3], 'x':['a','b','c']})
b = pd.DataFrame({'id':[2,3,4], 'y':['B','C','D']})
print(pd.merge(a, b, on='id', how='inner'))
print(pd.merge(a, b, on='id', how='left'))
Output
   id  x  y
0   2  b  B
1   3  c  C
   id  x    y
0   1  a  NaN
1   2  b    B
2   3  c    C

Text columns — not yet covered in class 4

.str.upper() / .str.lower()

Changes the case of every value in a text column.

print(df['name'].str.upper().tolist())
Output
['RAHUL', 'PRIYA', 'AMIT', 'SNEHA', 'VIKRAM', 'DIVYA']

.str.len()

How many characters in each value.

print(df['name'].str.len().tolist())
Output
[5, 5, 4, 5, 6, 5]

.str.strip() / .str.replace()

Cleans text values.

messy = pd.Series(['  Rahul ', ' Priya'])
print(messy.str.strip().tolist())
print(df['dept'].str.replace('Tech', 'IT').tolist())
Output
['Rahul', 'Priya']
['IT', 'Sales', 'IT', 'HR', 'Sales', 'IT']

.str.split()

Splits text and picks a part.

full = pd.Series(['Rahul Sharma', 'Priya Nair'])
print(full.str.split(' ').tolist())
print(full.str.split(' ').str[0].tolist())
Output
[['Rahul', 'Sharma'], ['Priya', 'Nair']]
['Rahul', 'Priya']

Dates — not yet covered in class 3

pd.to_datetime()

Turns text into real dates so you can work with them.

d = pd.DataFrame({'joined':['2022-01-15','2023-06-30']})
d['joined'] = pd.to_datetime(d['joined'])
print(d.dtypes)
print(d)
Output
joined    datetime64[ns]
dtype: object
      joined
0 2022-01-15
1 2023-06-30

.dt.year / .dt.month / .dt.day

Pulls parts out of a date column.

d = pd.DataFrame({'joined': pd.to_datetime(['2022-01-15','2023-06-30'])})
print(d['joined'].dt.year.tolist())
print(d['joined'].dt.month.tolist())
print(d['joined'].dt.day_name().tolist())
Output
[2022, 2023]
[1, 6]
['Saturday', 'Friday']

Date differences

Subtract two dates to get the gap.

d = pd.to_datetime(['2022-01-01','2024-01-01'])
print((d[1] - d[0]).days)
Output
730

SQL

Sessions D01 to D23. Databases, queries, functions, subqueries, joins and window functions.

48 functions

Databases and design — D01 to D03 3

CREATE TABLE

Creates a new table and defines its columns.

SELECT sql FROM sqlite_master WHERE name='employees';
Output
sql                                                                                                                                                  
-----------------------------------------------------------------------------------------------------------------------------------------------------
CREATE TABLE employees (
  emp_id INTEGER PRIMARY KEY, name TEXT, dept_id INTEGER,
  salary INTEGER, experience INTEGER, join_date TEXT, rating TEXT)

Primary key

The column that uniquely identifies each row.

SELECT emp_id, name FROM employees LIMIT 3;
Output
emp_id | name        
-------+-------------
101    | Rahul Sharma
102    | Priya Nair  
103    | Amit Kumar  

Foreign key

A column that points to the primary key of another table.

SELECT e.name, d.dept_name FROM employees e JOIN departments d ON e.dept_id=d.dept_id LIMIT 3;
Output
name         | dept_name
-------------+----------
Rahul Sharma | Tech     
Priya Nair   | Sales    
Amit Kumar   | Tech     

Reading data — D08 to D10 10

SELECT

Fetches columns from a table.

SELECT name, salary FROM employees LIMIT 3;
Output
name         | salary
-------------+-------
Rahul Sharma | 72000 
Priya Nair   | 95000 
Amit Kumar   | 58000 

WHERE

Keeps only the rows that match a condition.

SELECT name, salary FROM employees WHERE salary > 70000;
Output
name         | salary
-------------+-------
Rahul Sharma | 72000 
Priya Nair   | 95000 
Sneha Reddy  | 120000
Divya Menon  | 88000 

ORDER BY

Sorts the result.

SELECT name, salary FROM employees ORDER BY salary DESC LIMIT 3;
Output
name        | salary
------------+-------
Sneha Reddy | 120000
Priya Nair  | 95000 
Divya Menon | 88000 

LIMIT

Restricts how many rows come back.

SELECT name, salary FROM employees ORDER BY salary DESC LIMIT 2;
Output
name        | salary
------------+-------
Sneha Reddy | 120000
Priya Nair  | 95000 

DISTINCT

Removes duplicate rows from the result.

SELECT DISTINCT dept_id FROM employees ORDER BY dept_id;
Output
dept_id
-------
1      
2      
3      
4      

AS (alias)

Gives a column or table a temporary name.

SELECT name AS employee, salary AS monthly_pay FROM employees LIMIT 2;
Output
employee     | monthly_pay
-------------+------------
Rahul Sharma | 72000      
Priya Nair   | 95000      

IN

Checks a value against a list.

SELECT name, dept_id FROM employees WHERE dept_id IN (1,3);
Output
name         | dept_id
-------------+--------
Rahul Sharma | 1      
Amit Kumar   | 1      
Sneha Reddy  | 3      
Divya Menon  | 1      

BETWEEN

Checks a value inside a range.

SELECT name, salary FROM employees WHERE salary BETWEEN 60000 AND 90000;
Output
name         | salary
-------------+-------
Rahul Sharma | 72000 
Vikram Rao   | 68000 
Divya Menon  | 88000 

LIKE

Pattern matching on text.

SELECT name FROM employees WHERE name LIKE '%Kumar%';
Output
name      
----------
Amit Kumar

IS NULL

Finds missing values.

SELECT name, rating FROM employees WHERE rating IS NULL;
Output
name      | rating
----------+-------
Arjun Das | NULL  

Changing data — D07 4

INSERT

Adds a new row.

SELECT COUNT(*) AS rows_now FROM employees;
Output
rows_now
--------
7       

UPDATE

Changes values in existing rows.

SELECT name, salary FROM employees WHERE emp_id=101;
Output
name         | salary
-------------+-------
Rahul Sharma | 72000 

DELETE

Removes rows.

SELECT COUNT(*) AS total FROM employees;
Output
total
-----
7    

ALTER TABLE

Changes the structure of an existing table.

SELECT name FROM pragma_table_info('employees') LIMIT 4;
Output
name   
-------
emp_id 
name   
dept_id
salary 

String functions — D13 6

UPPER and LOWER

Changes the case of text.

SELECT UPPER(name) AS caps, LOWER(name) AS small FROM employees LIMIT 2;
Output
caps         | small       
-------------+-------------
RAHUL SHARMA | rahul sharma
PRIYA NAIR   | priya nair  

LENGTH

How many characters in the text.

SELECT name, LENGTH(name) AS chars FROM employees LIMIT 3;
Output
name         | chars
-------------+------
Rahul Sharma | 12   
Priya Nair   | 10   
Amit Kumar   | 10   

SUBSTR

Takes part of a string.

SELECT name, SUBSTR(name,1,5) AS first_five FROM employees LIMIT 3;
Output
name         | first_five
-------------+-----------
Rahul Sharma | Rahul     
Priya Nair   | Priya     
Amit Kumar   | Amit      

Concatenation

Joins strings together.

SELECT name || ' works in dept ' || dept_id AS description FROM employees LIMIT 2;
Output
description                 
----------------------------
Rahul Sharma works in dept 1
Priya Nair works in dept 2  

TRIM

Removes spaces from both ends.

SELECT '[' || TRIM('   Rahul   ') || ']' AS trimmed;
Output
trimmed
-------
[Rahul]

REPLACE

Swaps one piece of text for another.

SELECT REPLACE(name,'Kumar','K.') AS shortened FROM employees WHERE name LIKE '%Kumar%';
Output
shortened
---------
Amit K.  

Date functions — D14 3

Extracting parts of a date

Pulls the year, month or day out of a date.

SELECT name, STRFTIME('%Y',join_date) AS yr, STRFTIME('%m',join_date) AS mth FROM employees LIMIT 3;
Output
name         | yr   | mth
-------------+------+----
Rahul Sharma | 2022 | 01 
Priya Nair   | 2019 | 06 
Amit Kumar   | 2023 | 03 

Current date

Returns today's date.

SELECT DATE('2026-09-04') AS today;
Output
today     
----------
2026-09-04

Date difference

How far apart two dates are.

SELECT name, 2026 - CAST(STRFTIME('%Y',join_date) AS INTEGER) AS years_here FROM employees LIMIT 3;
Output
name         | years_here
-------------+-----------
Rahul Sharma | 4         
Priya Nair   | 7         
Amit Kumar   | 3         

Control flow — D15 2

CASE WHEN

If-then-else logic inside a query.

SELECT name, salary,
  CASE WHEN salary > 100000 THEN 'High'
       WHEN salary > 65000  THEN 'Medium'
       ELSE 'Low' END AS band
FROM employees ORDER BY salary DESC;
Output
name         | salary | band  
-------------+--------+-------
Sneha Reddy  | 120000 | High  
Priya Nair   | 95000  | Medium
Divya Menon  | 88000  | Medium
Rahul Sharma | 72000  | Medium
Vikram Rao   | 68000  | Medium
Amit Kumar   | 58000  | Low   
Arjun Das    | 54000  | Low   

COALESCE

Returns the first value that is not NULL.

SELECT name, COALESCE(rating,'Not rated') AS rating FROM employees;
Output
name         | rating   
-------------+----------
Rahul Sharma | Good     
Priya Nair   | Excellent
Amit Kumar   | Poor     
Sneha Reddy  | Excellent
Vikram Rao   | Average  
Divya Menon  | Good     
Arjun Das    | Not rated

Aggregate functions — D17 to D18 5

COUNT

Counts rows.

SELECT COUNT(*) AS all_rows, COUNT(rating) AS with_rating FROM employees;
Output
all_rows | with_rating
---------+------------
7        | 6          

SUM and AVG

The total and the average of a column.

SELECT SUM(salary) AS total, ROUND(AVG(salary),0) AS average FROM employees;
Output
total  | average
-------+--------
555000 | 79286.0

MIN and MAX

The smallest and largest value.

SELECT MIN(salary) AS lowest, MAX(salary) AS highest FROM employees;
Output
lowest | highest
-------+--------
54000  | 120000 

GROUP BY

Splits rows into groups and summarises each one.

SELECT dept_id, COUNT(*) AS headcount, ROUND(AVG(salary),0) AS avg_sal FROM employees GROUP BY dept_id;
Output
dept_id | headcount | avg_sal 
--------+-----------+---------
1       | 3         | 72667.0 
2       | 2         | 81500.0 
3       | 1         | 120000.0
4       | 1         | 54000.0 

HAVING

Filters the groups after aggregation.

SELECT dept_id, COUNT(*) AS n FROM employees GROUP BY dept_id HAVING COUNT(*) > 1;
Output
dept_id | n
--------+--
1       | 3
2       | 2

Subqueries — D19 to D20 3

Subquery in WHERE

A query inside another query.

SELECT name, salary FROM employees WHERE salary > (SELECT AVG(salary) FROM employees);
Output
name        | salary
------------+-------
Priya Nair  | 95000 
Sneha Reddy | 120000
Divya Menon | 88000 

Second highest value

The classic interview question.

SELECT MAX(salary) AS second_highest FROM employees WHERE salary < (SELECT MAX(salary) FROM employees);
Output
second_highest
--------------
95000         

Subquery with IN

Matches against a list produced by another query.

SELECT name FROM employees WHERE dept_id IN (SELECT dept_id FROM departments WHERE city='Bangalore');
Output
name        
------------
Rahul Sharma
Amit Kumar  
Divya Menon 
Arjun Das   

Joins — D21 to D22 5

INNER JOIN

Keeps only rows that match in both tables.

SELECT e.name, d.dept_name FROM employees e INNER JOIN departments d ON e.dept_id=d.dept_id LIMIT 4;
Output
name         | dept_name
-------------+----------
Rahul Sharma | Tech     
Priya Nair   | Sales    
Amit Kumar   | Tech     
Sneha Reddy  | HR       

LEFT JOIN

Keeps every row from the left table.

SELECT d.dept_name, COUNT(e.emp_id) AS headcount FROM departments d LEFT JOIN employees e ON d.dept_id=e.dept_id GROUP BY d.dept_name;
Output
dept_name | headcount
----------+----------
HR        | 1        
Marketing | 1        
Sales     | 2        
Tech      | 3        

Finding rows with no match

Rows in one table that have nothing in the other.

SELECT d.dept_name FROM departments d LEFT JOIN employees e ON d.dept_id=e.dept_id WHERE e.emp_id IS NULL;
Output
dept_name
---------

Self join

A table joined to itself.

SELECT a.name, b.name AS same_dept_colleague FROM employees a JOIN employees b ON a.dept_id=b.dept_id AND a.emp_id<b.emp_id;
Output
name         | same_dept_colleague
-------------+--------------------
Rahul Sharma | Amit Kumar         
Rahul Sharma | Divya Menon        
Priya Nair   | Vikram Rao         
Amit Kumar   | Divya Menon        

UNION

Stacks the results of two queries.

SELECT name FROM employees WHERE salary>100000 UNION SELECT name FROM employees WHERE experience>9;
Output
name       
-----------
Priya Nair 
Sneha Reddy

Window functions — D15 4

ROW_NUMBER

Numbers the rows in order, with no ties.

SELECT name, salary, ROW_NUMBER() OVER (ORDER BY salary DESC) AS rn FROM employees LIMIT 4;
Output
name         | salary | rn
-------------+--------+---
Sneha Reddy  | 120000 | 1 
Priya Nair   | 95000  | 2 
Divya Menon  | 88000  | 3 
Rahul Sharma | 72000  | 4 

RANK and DENSE_RANK

Ranks rows, handling ties differently.

SELECT name, salary, RANK() OVER (ORDER BY salary DESC) AS rnk FROM employees LIMIT 4;
Output
name         | salary | rnk
-------------+--------+----
Sneha Reddy  | 120000 | 1  
Priya Nair   | 95000  | 2  
Divya Menon  | 88000  | 3  
Rahul Sharma | 72000  | 4  

PARTITION BY

Restarts the window calculation for each group.

SELECT name, dept_id, salary, RANK() OVER (PARTITION BY dept_id ORDER BY salary DESC) AS rnk FROM employees;
Output
name         | dept_id | salary | rnk
-------------+---------+--------+----
Divya Menon  | 1       | 88000  | 1  
Rahul Sharma | 1       | 72000  | 2  
Amit Kumar   | 1       | 58000  | 3  
Priya Nair   | 2       | 95000  | 1  
Vikram Rao   | 2       | 68000  | 2  
Sneha Reddy  | 3       | 120000 | 1  
Arjun Das    | 4       | 54000  | 1  

Aggregate as a window

Adds a group summary without collapsing the rows.

SELECT name, dept_id, salary, ROUND(AVG(salary) OVER (PARTITION BY dept_id),0) AS dept_avg FROM employees;
Output
name         | dept_id | salary | dept_avg
-------------+---------+--------+---------
Rahul Sharma | 1       | 72000  | 72667.0 
Amit Kumar   | 1       | 58000  | 72667.0 
Divya Menon  | 1       | 88000  | 72667.0 
Priya Nair   | 2       | 95000  | 81500.0 
Vikram Rao   | 2       | 68000  | 81500.0 
Sneha Reddy  | 3       | 120000 | 120000.0
Arjun Das    | 4       | 54000  | 54000.0 

Views and routines — D23 3

CREATE VIEW

Saves a query so it behaves like a virtual table.

SELECT name, salary FROM employees WHERE salary > 80000;
Output
name        | salary
------------+-------
Priya Nair  | 95000 
Sneha Reddy | 120000
Divya Menon | 88000 

Stored procedure

A named block of SQL saved in the database.

SELECT 'CALL get_dept_summary(1);' AS mysql_syntax;
Output
mysql_syntax             
-------------------------
CALL get_dept_summary(1);

Trigger

SQL that runs automatically when data changes.

SELECT 'CREATE TRIGGER log_change AFTER UPDATE ON employees ...' AS mysql_syntax;
Output
mysql_syntax                                           
-------------------------------------------------------
CREATE TRIGGER log_change AFTER UPDATE ON employees ...

Nothing matched. Try a shorter word.