Unit III · Lecture 18 of 40 · B.Sc. Data Science, 1st Year, Semester II
In the previous lecture, we studied the Mean Absolute Deviation (MAD) — a simple and intuitive measure of spread. However, MAD has a critical limitation: it is not algebraically tractable. You cannot easily combine MADs from subgroups, use it in hypothesis testing, or build regression models with it.
To overcome this, statisticians developed variance and standard deviation — measures that use squared deviations instead of absolute values. These measures form the backbone of modern statistical analysis.
Think of MAD as a hand saw — simple, easy to use, and good for basic cuts. Variance and standard deviation are like a power saw — more complex, but far more powerful and versatile. For simple tasks, either works. For advanced construction (hypothesis testing, regression, ANOVA), you need the power tool.
Variance is the arithmetic mean of the squared deviations of observations from their mean. It measures the average squared distance of data points from the centre.
σ² = (1/N) × Σ(xᵢ − μ)²
Where:
• xᵢ = each observation
• μ = population mean
• N = population size
• σ² = population variance (sigma squared)
s² = (1/(n−1)) × Σ(xᵢ − x̄)²
Where:
• xᵢ = each observation
• x̄ = sample mean
• n = sample size
• s² = sample variance
• Dividing by (n−1) corrects for bias (Bessel's correction)
The Standard Deviation (SD) is the positive square root of the variance. It measures the average amount of variability in a dataset, expressed in the same units as the original data (unlike variance, which is in squared units).
σ = √σ² = √[(1/N) × Σ(xᵢ − μ)²]
s = √s² = √[(1/(n−1)) × Σ(xᵢ − x̄)²]
If the marks of students are measured in points, then variance is in points² (squared units — hard to interpret). Standard deviation takes the square root and brings the measure back to points (original units — easy to interpret). Think of it as: variance is the raw material, and standard deviation is the finished product.
The Root Mean Square Deviation (RMSD) about a value A is defined as:
RMSD(A) = √[(1/n) × Σ(xᵢ − A)²]
It is the square root of the mean of squared deviations from any arbitrary value A — not necessarily the mean.
When A = x̄ (the mean), the RMSD becomes the standard deviation:
RMSD(x̄) = √[(1/n) × Σ(xᵢ − x̄)²] = SD
When A ≠ x̄, the RMSD is always greater than the SD. This is because the mean is the value that minimises the sum of squared deviations.
Theorem: The sum of squared deviations Σ(xᵢ − A)² is minimum when A = x̄.
This means:
RMSD(A) ≥ SD for any value A
RMSD(A) = SD if and only if A = x̄
This is why the standard deviation (about the mean) is the most efficient measure of spread based on squared deviations.
Find the variance and standard deviation of: 4, 8, 6, 5, 7, 10
Step 1: Mean
x̄ = (4 + 8 + 6 + 5 + 7 + 10) / 6 = 40 / 6 = 6.67
Step 2–4: Deviations and Squared Deviations
| xᵢ | xᵢ − x̄ | (xᵢ − x̄)² |
|---|---|---|
| 4 | −2.67 | 7.1289 |
| 8 | +1.33 | 1.7689 |
| 6 | −0.67 | 0.4489 |
| 5 | −1.67 | 2.7889 |
| 7 | +0.33 | 0.1089 |
| 10 | +3.33 | 11.0889 |
| Σ(xᵢ − x̄)² = | 23.3334 | |
The number of goals scored in 30 matches is given below. Find the standard deviation.
| Goals (xᵢ) | 0 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| Frequency (fᵢ) | 3 | 5 | 8 | 7 | 4 | 3 |
Step 1: Mean
N = 3+5+8+7+4+3 = 30
Σfᵢxᵢ = (3×0)+(5×1)+(8×2)+(7×3)+(4×4)+(3×5) = 0+5+16+21+16+15 = 73
x̄ = 73/30 = 2.433
| xᵢ | fᵢ | fᵢxᵢ | xᵢ − x̄ | (xᵢ − x̄)² | fᵢ(xᵢ − x̄)² |
|---|---|---|---|---|---|
| 0 | 3 | 0 | −2.433 | 5.920 | 17.760 |
| 1 | 5 | 5 | −1.433 | 2.054 | 10.270 |
| 2 | 8 | 16 | −0.433 | 0.188 | 1.504 |
| 3 | 7 | 21 | +0.567 | 0.321 | 2.247 |
| 4 | 4 | 16 | +1.567 | 2.455 | 9.820 |
| 5 | 3 | 15 | +2.567 | 6.590 | 19.770 |
| Totals | 73 | 61.371 | |||
The heights (in cm) of 50 students are grouped as follows. Find the standard deviation.
| Height (cm) | 150–155 | 155–160 | 160–165 | 165–170 | 170–175 |
|---|---|---|---|---|---|
| Frequency (fᵢ) | 6 | 12 | 18 | 10 | 4 |
Mid-points (xᵢ): 152.5, 157.5, 162.5, 167.5, 172.5
N = 6+12+18+10+4 = 50
Σfᵢxᵢ = (6×152.5)+(12×157.5)+(18×162.5)+(10×167.5)+(4×172.5) = 915+1890+2925+1675+690 = 8095
x̄ = 8095/50 = 161.9 cm
| Class | xᵢ | fᵢ | fᵢxᵢ | xᵢ − x̄ | (xᵢ − x̄)² | fᵢ(xᵢ − x̄)² |
|---|---|---|---|---|---|---|
| 150–155 | 152.5 | 6 | 915 | −9.4 | 88.36 | 530.16 |
| 155–160 | 157.5 | 12 | 1890 | −4.4 | 19.36 | 232.32 |
| 160–165 | 162.5 | 18 | 2925 | +0.6 | 0.36 | 6.48 |
| 165–170 | 167.5 | 10 | 1675 | +5.6 | 31.36 | 313.60 |
| 170–175 | 172.5 | 4 | 690 | +10.6 | 112.36 | 449.44 |
| Totals | 8095 | 1532.00 | ||||
When data values are large (e.g., heights in cm, incomes in thousands), calculating (xᵢ − x̄)² directly is tedious. The assumed mean method simplifies computation by choosing an arbitrary value A (usually near the centre of the data) and working with deviations from A.
Let dᵢ = xᵢ − A (deviation from assumed mean A)
Variance: σ² = (Σfᵢdᵢ² / N) − (Σfᵢdᵢ / N)²
SD: σ = √σ²
This is equivalent to: σ² = (mean of d²) − (mean of d)²
Let A = 162.5 (mid-point of the modal class), class width c = 5
dᵢ = (xᵢ − A) / c: d values are −2, −1, 0, 1, 2
| Class | fᵢ | dᵢ | fᵢdᵢ | fᵢdᵢ² |
|---|---|---|---|---|
| 150–155 | 6 | −2 | −12 | 24 |
| 155–160 | 12 | −1 | −12 | 12 |
| 160–165 | 18 | 0 | 0 | 0 |
| 165–170 | 10 | 1 | 10 | 10 |
| 170–175 | 4 | 2 | 8 | 16 |
| Totals | 50 | −6 | 62 |
σ²(d) = (62/50) − (−6/50)² = 1.24 − 0.0144 = 1.2256
σ² = σ²(d) × c² = 1.2256 × 25 = 30.64
σ = √30.64 ≈ 5.54 cm ✓ (Same answer, much easier arithmetic!)
Consider the data: 2, 4, 6, 8, 10 (Mean = 6)
| Value of A | Σ(xᵢ − A)² | RMSD(A) = √[Σ(xᵢ−A)²/5] | Comparison |
|---|---|---|---|
| A = 0 | 4+16+36+64+100 = 220 | √44.0 = 6.63 | RMSD > SD |
| A = 4 | 4+0+4+16+36 = 60 | √12.0 = 3.46 | RMSD > SD |
| A = 6 (mean) | 16+4+0+4+16 = 40 | √8.0 = 2.83 | RMSD = SD (minimum!) |
| A = 8 | 36+16+4+0+4 = 60 | √12.0 = 3.46 | RMSD > SD |
| A = 10 | 64+36+16+4+0 = 120 | √24.0 = 4.90 | RMSD > SD |
The standard deviation is the minimum possible RMSD — achieved when deviations are measured from the mean.
Enter data and an arbitrary value A to see both the SD and RMSD(A) computed step by step:
| Feature | MAD | Variance (σ²) | Std. Deviation (σ) |
|---|---|---|---|
| Based on | |xᵢ − x̄| | (xᵢ − x̄)² | √(xᵢ − x̄)² |
| Units | Same as data | Squared units | Same as data |
| Outlier effect | Moderate | High | High |
| Algebraic use | Poor | Excellent | Excellent |
| Used in testing | Rarely | Directly | Directly |
| Ease of computation | Easiest | Moderate | Moderate |
| Interpretability | Very intuitive | Less intuitive (squared) | Intuitive |
| Relationship | Independent | σ² = (SD)² | σ = √(Variance) |
Click each scenario to see how adding an outlier affects the mean, MAD, and SD:
50, 55, 60, 65, 70
50, 55, 60, 65, 70, 95
50, 55, 60, 65, 70, 200
Q1: The standard deviation is the positive square root of:
Q2: If every observation in a dataset is multiplied by 3, the new SD will be:
Q3: The RMSD about a value A is minimum when A equals:
Q4: Variance is expressed in:
Q5: Why do we divide by (n−1) instead of n for sample variance?
Q6: If y = x + 10, then SD(y) equals:
Q7: Which of the following is NOT a property of standard deviation?
Q8: The shortcut formula for variance is:
1. Calculate the variance and standard deviation of: 12, 15, 18, 21, 24
2. Find the SD for the following frequency distribution:
Value: 5, 10, 15, 20, 25
Frequency: 4, 6, 10, 8, 2
3. Find the SD for the following grouped data using the assumed mean method:
Class: 0–10, 10–20, 20–30, 30–40, 40–50
Frequency: 5, 8, 15, 12, 10
4. The RMSD of a dataset about the value A = 10 is 5.2. If the mean of the dataset is 8, what can you say about the standard deviation?
5. Two datasets have the same mean of 50. Dataset A has SD = 5 and Dataset B has SD = 15. Describe the difference in their distributions.
6. If the variance of a dataset is 64, what is the standard deviation? If each value is multiplied by 2 and then 5 is added, what will be the new variance and SD?
7. BONUS: Prove that Σ(xᵢ − A)² = Σ(xᵢ − x̄)² + n(x̄ − A)² for any value A. Use this to explain why RMSD(A) ≥ SD.
Questions? Let's discuss!
Lecture 19: Coefficient of Variation — Comparing Variability Across Datasets
Lecture 20: Combined Variance & SD of Two Groups
Lecture 21: Lorenz Curve & Measures of Inequality
Unit III · Lecture 18 · BDS113 Introduction to Statistics · B.Sc. Data Science, Semester II