Correlation & Regression

Unit IV · Lecture 31

BDS113 — Introduction to Statistics

Simple Linear Regression

Regression Lines · y on x · x on y · Regression Coefficients · Properties

Unit IV · Lecture 31 of 40 · B.Sc. Data Science, 1st Year, Semester II

Correlation & Regression

Unit IV · Lecture 31

🤔 Correlation vs Regression

📖 The Key Distinction

Correlation measures the strength and direction of the linear relationship between two variables. It is symmetric — r(x,y) = r(y,x).

Regression goes further: it provides an equation to predict one variable from the other. It is not symmetric — the regression of y on x is different from the regression of x on y.

💡 Analogy: The GPS

Correlation tells you that two cities are connected by a road (the relationship exists and is strong). Regression gives you the GPS directions — the exact equation to get from one city to the other (predict y from x). And the directions from A to B are different from B to A — just like two different regression lines.

📖 Francis Galton (1822–1911)

The term "regression" was coined by Sir Francis Galton, who studied the heights of parents and their children. He found that tall parents tended to have children shorter than themselves, and short parents tended to have taller children — children's heights regressed toward the mean. He called this "regression toward the mean."

Correlation & Regression

Unit IV · Lecture 31

📏 Two Regression Lines

Regression of y on x

Equation: y − ȳ = bᵧₓ(x − x̄)

Slope (bᵧₓ): = r · (Sᵧ / Sₓ)

Or: bᵧₓ = [nΣxy − ΣxΣy] / [nΣx² − (Σx)²]

Purpose: Predict y given x.

Read as: "y regressed on x"

Regression of x on y

Equation: x − x̄ = bₓᵧ(y − ȳ)

Slope (bₓᵧ): = r · (Sₓ / Sᵧ)

Or: bₓᵧ = [nΣxy − ΣxΣy] / [nΣy² − (Σy)²]

Purpose: Predict x given y.

Read as: "x regressed on y"

💬 Key Difference: The denominators differ! y on x uses Σx² − (Σx)²/n; x on y uses Σy² − (Σy)²/n. The two lines are different unless r = ±1 (in which case they coincide). Both lines always pass through (x̄, ȳ).

Correlation & Regression

Unit IV · Lecture 31

📐 Regression Coefficients

📖 Definitions

Regression coefficient of y on x (bᵧₓ):
    bᵧₓ = Cov(x,y) / Var(x) = [Σ(xᵢ−x̄)(yᵢ−ȳ)] / [Σ(xᵢ−x̄)²]

Regression coefficient of x on y (bₓᵧ):
    bₓᵧ = Cov(x,y) / Var(y) = [Σ(xᵢ−x̄)(yᵢ−ȳ)] / [Σ(yᵢ−ȳ)²]

Relationship with correlation:
    r² = bᵧₓ × bₓᵧ    →    r = ±√(bᵧₓ × bₓᵧ)
    (Sign of r = sign of both regression coefficients)

📖 Interpretation of bᵧₓ

bᵧₓ represents the average change in y for a one-unit increase in x.

  • If bᵧₓ = 2.5: On average, y increases by 2.5 units for each 1-unit increase in x.
  • If bᵧₓ = −1.3: On average, y decreases by 1.3 units for each 1-unit increase in x.
  • If bᵧₓ = 0: There is no linear relationship — knowing x doesn't help predict y.

bᵧₓ has units: if y is in kg and x is in cm, then bᵧₓ is in kg/cm.

Correlation & Regression

Unit IV · Lecture 31

📝 Regression Line Equations

📖 y on x (in different forms)

y − ȳ = bᵧₓ(x − x̄)    [point-slope form]
y = (ȳ − bᵧₓx̄) + bᵧₓx    [y = a + bx form, where a = ȳ − bᵧₓx̄]
y = aᵧₓ + bᵧₓx    [intercept-slope form]

📖 x on y (in different forms)

x − x̄ = bₓᵧ(y − ȳ)    [point-slope form]
x = (x̄ − bₓᵧȳ) + bₓᵧy    [x = a + by form]
x = aₓᵧ + bₓᵧy    [intercept-slope form]

💡 Both lines pass through (x̄, ȳ). This is the only point where they intersect (unless r = ±1, when they are the same line). The angle between the two lines is related to the strength of correlation — the closer r is to ±1, the smaller the angle.

Correlation & Regression

Unit IV · Lecture 31

✏️ Example: Both Regression Lines

📝 Problem

Given: n = 5, Σx = 15, Σy = 20, Σxy = 66, Σx² = 55, Σy² = 86. Find both regression lines and r.

📖 Solution

x̄ = 15/5 = 3, ȳ = 20/5 = 4

bᵧₓ = [nΣxy − ΣxΣy] / [nΣx² − (Σx)²]
  = [5(66) − (15)(20)] / [5(55) − 225]
  = 30/50 = 0.6

bₓᵧ = [nΣxy − ΣxΣy] / [nΣy² − (Σy)²]
  = [5(66) − (15)(20)] / [5(86) − 400]
  = 30/30 = 1.0

r = √(bᵧₓ × bₓᵧ) = √(0.6 × 1.0) = √0.6 = 0.7746

Regression of y on x

y − 4 = 0.6(x − 3)

y = 2.2 + 0.6x

Regression of x on y

x − 3 = 1.0(y − 4)

x = −1.0 + 1.0y

💡 Verification: r² = bᵧₓ × bₓᵧ = 0.6 × 1.0 = 0.60. So r = √0.60 = 0.7746. This matches our earlier calculation from Lecture 28.

Correlation & Regression

Unit IV · Lecture 31

🔑 Properties of Regression Lines

Both lines pass through (x̄, ȳ) — the point of means is always on both regression lines.
r² = bᵧₓ × bₓᵧ — the square of the correlation coefficient equals the product of the two regression coefficients.
bᵧₓ and bₓᵧ always have the same sign as r (both positive or both negative).
|bᵧₓ| ≤ |bₓᵧ| if Sₓ > Sᵧ, and vice versa. The regression coefficient with the larger variance in the denominator is smaller in magnitude.
If r = ±1, both regression lines coincide — there is a perfect linear relationship and no scatter.
If r = 0, both regression coefficients are 0. The regression line of y on x is y = ȳ (horizontal), and x on y is x = x̄ (vertical).
Regression minimises errors in one direction only: y on x minimises Σ(yᵢ − ŷᵢ)² (vertical deviations); x on y minimises Σ(xᵢ − x̂ᵢ)² (horizontal deviations).
Regression toward the mean: Predicted values of y are always closer to ȳ than the corresponding x values are to x̄ (unless r = ±1).

Correlation & Regression

Unit IV · Lecture 31

🎬 Interactive: Both Regression Lines

x values:

y values:

Correlation & Regression

Unit IV · Lecture 31

🔑 Summary: Simple Linear Regression

Regression provides an equation to predict one variable from another. There are two regression lines — y on x and x on y.
Regression of y on x: y − ȳ = bᵧₓ(x − x̄), where bᵧₓ = Cov(x,y)/Var(x).
Regression of x on y: x − x̄ = bₓᵧ(y − ȳ), where bₓᵧ = Cov(x,y)/Var(y).
r² = bᵧₓ × bₓᵧ — the geometric mean of the two regression coefficients gives |r|.
Both lines pass through (x̄, ȳ). They coincide only when r = ±1.
Regression coefficients have units; the correlation coefficient is unit-free.
Regression toward the mean: Extreme values in x predict less extreme values in y.
Use y on x when you want to predict y from x. Use x on y when you want to predict x from y.

Correlation & Regression

Unit IV · Lecture 31

🧠 Quick Quiz — Lecture 31

Q1: The two regression lines intersect at:

A) The origin (0, 0)

B) The point (x̄, ȳ)

C) Two different points

D) They never intersect

Q2: r² equals:

A) bᵧₓ + bₓᵧ

B) bᵧₓ × bₓᵧ

C) bᵧₓ / bₓᵧ

D) bᵧₓ − bₓᵧ

Q3: If r = 0, the regression line of y on x is:

A) x = x̄

B) y = ȳ

C) y = x

D) y = 0

Q4: Regression of y on x minimises:

A) Σ|yᵢ − ŷᵢ|

B) Σ(yᵢ − ŷᵢ)²

C) Σ(xᵢ − x̂ᵢ)²

D) Both vertical and horizontal deviations

Q5: The regression coefficient bᵧₓ has units of:

A) Unit-free (like r)

B) Units of y per unit of x

C) Units of x per unit of y

D) Squared units of y

Correlation & Regression

Unit IV · Lecture 31

📝 Homework & Next Lecture

📝 Practice Tasks

1. Find both regression lines for: x = 2, 4, 6, 8, 10 and y = 3, 7, 5, 11, 9.

2. If bᵧₓ = 1.5 and bₓᵧ = 0.6, find r.

3. If the regression line of y on x is y = 3 + 2x, predict y when x = 10.

4. Explain why the two regression lines are different unless r = ±1.

5. Given x̄ = 10, ȳ = 20, bᵧₓ = 1.5, bₓᵧ = 0.4. Verify that r² = bᵧₓ × bₓᵧ. Find r.

6. BONUS: Show that if we interchange x and y in the regression of y on x, we do NOT get the regression of x on y. Use a numerical example.

Upcoming

Lecture 32: Coefficient of Regression — deeper properties, standard error of estimate, and the relationship between correlation and regression.