Unit IV · Lecture 24 of 40 · B.Sc. Data Science, 1st Year, Semester II
Curve fitting is the process of finding a mathematical function (curve) that best fits a set of observed data points. The goal is to capture the underlying relationship between two variables (x and y) in a compact mathematical form, so we can describe, predict, and understand the data.
Imagine you have plotted several points on a graph paper — each point represents an observation (xᵢ, yᵢ). The points don't fall on a perfect line or curve. Curve fitting is like drawing the "best possible line" through these scattered points — the line that passes as close as possible to all points simultaneously.
1. Description: Summarise the relationship between variables in a simple equation.
2. Prediction: Estimate y for values of x not observed in the data.
3. Interpolation: Estimate values between observed data points.
4. Extrapolation: Estimate values beyond the observed range (with caution).
5. Understanding: Identify trends, patterns, and the nature of the relationship.
Straight line: y = a + bx (most common)
Parabola: y = a + bx + cx²
Exponential: y = abˣ
Power curve: y = axᵇ
Logarithmic: y = a + b·ln(x)
Today we focus on the straight line — the simplest and most widely used.
Given n data points (x₁,y₁), (x₂,y₂), …, (xₙ,yₙ), there are infinitely many straight lines that can be drawn through or near the data. We need a criterion to select the one "best" line. Different criteria give different "best" lines.
1. Minimise Σ|yᵢ − ŷᵢ| (sum of absolute deviations)
→ Gives the MAD-based line. Hard to solve analytically.
2. Minimise Σ(yᵢ − ŷᵢ)² (sum of squared deviations)
→ Least Squares! Easy to solve, elegant mathematics.
3. Minimise max|yᵢ − ŷᵢ| (maximum deviation)
→ Minimax criterion. Used in some engineering applications.
4. Minimise Σ|yᵢ − ŷᵢ|ᵖ for other values of p
→ Generalised criterion. p = 2 is the most common.
The least squares criterion (option 2) dominates because:
1. Mathematical elegance: Squaring makes the function differentiable everywhere (unlike |x|), allowing us to use calculus to find the minimum.
2. Unique solution: The least squares problem always has a unique, closed-form solution.
3. Statistical optimality: Under certain conditions (normal errors), least squares gives the best possible estimates (Gauss-Markov theorem).
4. Penalises large errors: Squaring gives more weight to large deviations, which is often desirable.
The Principle of Least Squares states that, among all possible curves of a given type, the best-fitting curve is the one for which the sum of the squares of the residuals (vertical deviations of observed values from the curve) is minimum.
Minimise: S = Σeᵢ² = Σ(yᵢ − ŷᵢ)²
Where:
• yᵢ = observed value of y for the i-th data point
• ŷᵢ = predicted (fitted) value from the curve
• eᵢ = yᵢ − ŷᵢ = residual (error) for the i-th point
• S = sum of squared residuals (to be minimised)
The method of least squares was independently developed by Adrien-Marie Legendre (1805) and Carl Friedrich Gauss (1809, though he claimed to have used it since 1795). Gauss used it to predict the orbit of the asteroid Ceres. The method became the foundation of modern regression analysis and remains the most widely used curve-fitting technique in science and engineering.
Click "Fit Line" to see the least squares line drawn through sample data points. The residuals (vertical red lines) show the errors being minimised:
Blue dots = observed data | Red line = least squares fit | Gray lines = residuals (errors)
We want to fit a straight line ŷ = a + bx to n data points (x₁,y₁), (x₂,y₂), …, (xₙ,yₙ) using the least squares criterion.
The residual for the i-th point is: eᵢ = yᵢ − (a + bxᵢ)
We need to find a (intercept) and b (slope) that minimise:
S = Σ[yᵢ − (a + bxᵢ)]²
b (slope): For every one-unit increase in x, y changes by b units on average. If b > 0, y increases with x (positive relationship). If b < 0, y decreases with x (negative relationship).
a (intercept): The predicted value of y when x = 0. It is the point where the line crosses the y-axis. (Note: a may not always have a meaningful interpretation if x = 0 is outside the data range.)
To minimise S = Σ[yᵢ − a − bxᵢ]², we take partial derivatives with respect to a and b, and set them to zero:
∂S/∂a = −2Σ(yᵢ − a − bxᵢ) = 0
∂S/∂b = −2Σxᵢ(yᵢ − a − bxᵢ) = 0
Expanding and simplifying:
Equation 1: Σyᵢ = na + bΣxᵢ → Σy = na + bΣx
Equation 2: Σxᵢyᵢ = aΣxᵢ + bΣxᵢ² → Σxy = aΣx + bΣx²
These two equations in two unknowns (a and b) are called the normal equations. Solving them simultaneously gives the least squares estimates.
b = [nΣxy − Σx·Σy] / [nΣx² − (Σx)²]
Equivalent form:
b = [Σxy − n·x̄·ȳ] / [Σx² − n·x̄²]
Or using deviations:
b = Σ(xᵢ−x̄)(yᵢ−ȳ) / Σ(xᵢ−x̄)²
a = ȳ − b·x̄
Where:
• x̄ = Σx/n = mean of x
• ȳ = Σy/n = mean of y
Key property: The least squares line always passes through the point (x̄, ȳ) — the means of x and y.
Let Sₓᵧ = Σ(xᵢ−x̄)(yᵢ−ȳ) (sum of cross-products)
Let Sₓₓ = Σ(xᵢ−x̄)² (sum of squared deviations of x)
Then: b = Sₓᵧ / Sₓₓ and a = ȳ − bx̄
This is the cleanest form and is used in most theoretical derivations.
Fit a least squares straight line to the following data:
| x | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| y | 2 | 4 | 5 | 4 | 5 |
Step 1–2: Compute the table
| x | y | xy | x² |
|---|---|---|---|
| 1 | 2 | 2 | 1 |
| 2 | 4 | 8 | 4 |
| 3 | 5 | 15 | 9 |
| 4 | 4 | 16 | 16 |
| 5 | 5 | 25 | 25 |
| Σx=15 | Σy=20 | Σxy=66 | Σx²=55 |
n = 5, x̄ = 15/5 = 3, ȳ = 20/5 = 4
b = [nΣxy − Σx·Σy] / [nΣx² − (Σx)²]
= [5(66) − (15)(20)] / [5(55) − (15)²]
= [330 − 300] / [275 − 225]
= 30 / 50 = 0.6
a = ȳ − b·x̄ = 4 − 0.6(3) = 4 − 1.8 = 2.2
For every unit increase in x, y increases by 0.6 units on average. When x = 0, the predicted y is 2.2.
Check 1: Residuals sum to zero
| x | y | ŷ = 2.2+0.6x | Residual (y−ŷ) |
|---|---|---|---|
| 1 | 2 | 2.8 | −0.8 |
| 2 | 4 | 3.4 | +0.6 |
| 3 | 5 | 4.0 | +1.0 |
| 4 | 4 | 4.6 | −0.6 |
| 5 | 5 | 5.2 | −0.2 |
| Sum of residuals = | 0.0 ✓ | ||
At x = x̄ = 3: ŷ = 2.2 + 0.6(3) = 2.2 + 1.8 = 4.0 = ȳ ✓
Predictions:
At x = 6: ŷ = 2.2 + 0.6(6) = 2.2 + 3.6 = 5.8 (extrapolation)
At x = 2.5: ŷ = 2.2 + 0.6(2.5) = 2.2 + 1.5 = 3.7 (interpolation)
A company records advertising expenditure (₹ lakhs) and sales (₹ lakhs). Fit a regression line and predict sales when advertising = ₹7 lakhs.
| Advertising (x) | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Sales (y) | 10 | 15 | 18 | 22 | 25 | 30 |
| x | y | xy | x² |
|---|---|---|---|
| 1 | 10 | 10 | 1 |
| 2 | 15 | 30 | 4 |
| 3 | 18 | 54 | 9 |
| 4 | 22 | 88 | 16 |
| 5 | 25 | 125 | 25 |
| 6 | 30 | 180 | 36 |
| 21 | 120 | 487 | 91 |
n = 6, x̄ = 21/6 = 3.5, ȳ = 120/6 = 20
b = [6(487) − (21)(120)] / [6(91) − (21)²]
= [2922 − 2520] / [546 − 441]
= 402 / 105 = 3.829
a = 20 − 3.829(3.5) = 20 − 13.4 = 6.6
When x = 7: ŷ = 6.6 + 3.83(7) = 6.6 + 26.81 = ₹33.41 lakhs
For every ₹1 lakh increase in advertising, sales increase by approximately ₹3.83 lakhs.
Enter paired (x, y) data to compute the least squares line, see the equation, and visualise the fit:
Q1: The least squares method minimises the sum of:
Q2: The least squares line always passes through:
Q3: If b = 0 in the fitted line ŷ = a + bx, this means:
Q4: The sum of residuals in a least squares fit is always:
Q5: Who is credited with publishing the method of least squares first?
Q6: The slope b in ŷ = a + bx represents:
Q7: Least squares estimates are BLUE. This stands for:
Q8: A limitation of least squares is:
1. Fit a least squares line to: x = 2, 4, 6, 8, 10 and y = 3, 7, 5, 11, 9. Predict y when x = 7.
2. The data below shows study hours and exam scores. Fit a regression line and interpret the slope.
Hours (x): 1, 2, 3, 4, 5, 6
Score (y): 45, 50, 55, 60, 65, 70
3. For the fitted line ŷ = 10 + 2.5x, interpret the values 10 and 2.5 in the context of production cost (y) and units produced (x).
4. Verify that the least squares line from Problem 1 passes through (x̄, ȳ).
5. Explain why the least squares method uses squared deviations instead of absolute deviations. List two mathematical advantages.
6. If Σx = 30, Σy = 50, Σxy = 200, Σx² = 150, and n = 6, find the least squares estimates a and b.
7. BONUS: Show that the least squares line ŷ = a + bx always passes through (x̄, ȳ). (Hint: Substitute x = x̄ into the equation and use a = ȳ − bx̄.)
Lecture 25: Fitting of Polynomials — Second-degree parabola and higher
Lecture 26: Correlation — Karl Pearson's Coefficient
Lecture 27: Rank Correlation — Spearman's Coefficient
Unit IV · Lecture 24 · BDS113 Introduction to Statistics · B.Sc. Data Science, Semester II