Correlation & Regression

Unit IV · Lecture 25

BDS113 — Introduction to Statistics

Fitting of Polynomials

Second-Degree Parabola · Normal Equations · Higher-Degree Polynomials · Choosing the Right Model

Unit IV · Lecture 25 of 40 · B.Sc. Data Science, 1st Year, Semester II

Correlation & Regression

Unit IV · Lecture 25

The Journey — Lecture 25

📐

When a Straight Line is Not Enough
Curved patterns · Residual analysis · Motivation for polynomials

📏

Fitting a Second-Degree Parabola
y = a + bx + cx² · Normal equations · Step-by-step method

✏️

Worked Examples
Parabola fitting · Interactive visualiser · Higher-degree overview

⚖️

Model Selection & Quiz
When to use which degree · Overfitting · Properties · Practice quiz

Correlation & Regression

Unit IV · Lecture 25

🤔 When a Straight Line is Not Enough

📖 The Problem

In the previous lecture, we fitted a straight line ŷ = a + bx using least squares. But many real-world relationships are not linear — they are curved. If we force a straight line through curved data, the fit will be poor and predictions unreliable.

💡 Analogy: Fitting Shoes

A straight line is like a one-size-fits-all shoe — it works for some feet (linear data) but pinches others (curved data). A polynomial is like a custom-moulded shoe — it adapts to the shape of your foot (the data pattern). The trick is to use the simplest shoe that fits well — you don't need a custom boot for a simple walk.

📖 Signs That You Need a Polynomial

1. Residual pattern: If residuals from a linear fit show a systematic U-shape or inverted U-shape, the data has curvature.

2. Scatter plot: If the scatter plot shows a clear curved trend (parabolic, S-shaped, etc.).

3. Domain knowledge: Physics, biology, economics often have inherent non-linear relationships (e.g., projectile motion is parabolic).

4. R² improvement: If adding a quadratic term significantly improves the fit (higher R², lower residual sum of squares).

📖 Common Polynomial Models

Degree 1 (Linear): y = a + bx
→ Straight line. 2 parameters.

Degree 2 (Quadratic/Parabola): y = a + bx + cx²
→ U-shape or inverted U. 3 parameters.

Degree 3 (Cubic): y = a + bx + cx² + dx³
→ S-shape or more complex curves. 4 parameters.

Degree k: y = a₀ + a₁x + a₂x² + … + aₖxᵏ
→ (k+1) parameters. Requires at least (k+1) data points.

Correlation & Regression

Unit IV · Lecture 25

📏 Fitting a Second-Degree Parabola: y = a + bx + cx²

📖 Problem Setup

We want to fit a second-degree polynomial (parabola) to n data points using the least squares criterion:

Minimise: S = Σ[yᵢ − (a + bxᵢ + cxᵢ²)]²

We need to find three unknowns: a, b, and c. This requires three normal equations.

📖 The Three Normal Equations

Taking ∂S/∂a = 0, ∂S/∂b = 0, ∂S/∂c = 0:

Σy = na + bΣx + cΣx²    … (1)

Σxy = aΣx + bΣx² + cΣx³    … (2)

Σx²y = aΣx² + bΣx³ + cΣx⁴    … (3)

Solving these three simultaneous equations gives the least squares estimates of a, b, and c.

💡 Pattern Recognition: Notice the pattern — each equation corresponds to multiplying the basic equation y = a + bx + cx² by 1, x, and x² respectively, then summing. For a degree-k polynomial, you multiply by 1, x, x², …, xᵏ to get (k+1) normal equations.

Correlation & Regression

Unit IV · Lecture 25

🪜 Step-by-Step: Fitting a Parabola

1 Create the computation table: Columns for x, y, xy, x²y, x², x³, x⁴.

2 Compute all sums: Σx, Σy, Σxy, Σx²y, Σx², Σx³, Σx⁴, and n.

3 Set up the three normal equations by substituting the sums.

4 Solve the three simultaneous equations for a, b, and c (by elimination, substitution, or matrix methods).

5 Write the equation: ŷ = a + bx + cx². Interpret the coefficients.

6 Verify: Check residuals and compute the residual sum of squares.

💡 Minimum Data Points: To fit a degree-k polynomial, you need at least k+1 data points. For a parabola (k=2), you need at least 3 points. In practice, you should have significantly more points than parameters for a reliable fit.

Correlation & Regression

Unit IV · Lecture 25

✏️ Example 1: Fitting a Parabola

📝 Problem

Fit a second-degree parabola y = a + bx + cx² to the following data:

x	1	2	3	4	5
y	2	6	12	20	30

📖 Solution — Step 1–2: Computation Table

x	y	xy	x²y	x²	x³	x⁴
1	2	2	2	1	1	1
2	6	12	24	4	8	16
3	12	36	108	9	27	81
4	20	80	320	16	64	256
5	30	150	750	25	125	625
15	70	280	1204	55	225	979

📖 Step 3: Normal Equations

n = 5, Σx = 15, Σy = 70, Σxy = 280, Σx²y = 1204, Σx² = 55, Σx³ = 225, Σx⁴ = 979

(1): 70 = 5a + 15b + 55c
(2): 280 = 15a + 55b + 225c
(3): 1204 = 55a + 225b + 979c

📖 Step 4: Solving the System

From (1): 14 = a + 3b + 11c → a = 14 − 3b − 11c … (1')

Substitute (1') into (2): 280 = 15(14−3b−11c) + 55b + 225c
280 = 210 − 45b − 165c + 55b + 225c
70 = 10b + 60c → 7 = b + 6c … (4)

Substitute (1') into (3): 1204 = 55(14−3b−11c) + 225b + 979c
1204 = 770 − 165b − 605c + 225b + 979c
434 = 60b + 374c … (5)

From (4): b = 7 − 6c. Substitute into (5):
434 = 60(7−6c) + 374c = 420 − 360c + 374c
14 = 14c → c = 1
b = 7 − 6(1) = 1
a = 14 − 3(1) − 11(1) = 0

Fitted Parabola

ŷ = x + x²

Or equivalently: ŷ = x(1 + x). The data follows a perfect quadratic pattern! (This is because y = x + x² generates the values 2, 6, 12, 20, 30 exactly.)

Correlation & Regression

Unit IV · Lecture 25

✏️ Example 1: Verification

x	y (observed)	ŷ = x + x²
1	2	1 + 1 = 2
2	6	2 + 4 = 6
3	12	3 + 9 = 12
4	20	4 + 16 = 20
5	30	5 + 25 = 30
SSE =

Perfect Fit!

All residuals are zero. SSE = 0.

This happens because the data was generated by a perfect quadratic function. In real-world data, residuals will never be exactly zero — but a good polynomial fit makes them small and randomly scattered.

💡 Comparison with Linear Fit: If we had fitted a straight line to this data, we would get ŷ = −2 + 6.4x, with SSE = 11.2 — a much worse fit. The parabola captures the curvature that the line misses.

Correlation & Regression

Unit IV · Lecture 25

✏️ Example 2: Production Cost vs Output

📝 Problem

A factory records output (x, in hundreds of units) and total cost (y, in ₹ thousands). Fit a parabola y = a + bx + cx².

x	1	2	3	4	5	6
y	15	20	22	25	30	38

x	y	xy	x²y	x²	x³	x⁴
1	15	15	15	1	1	1
2	20	40	80	4	8	16
3	22	66	198	9	27	81
4	25	100	400	16	64	256
5	30	150	750	25	125	625
6	38	228	1368	36	216	1296
21	150	599	2811	91	441	2275

📖 Normal Equations

(1): 150 = 6a + 21b + 91c
(2): 599 = 21a + 91b + 441c
(3): 2811 = 91a + 441b + 2275c

Solving (by elimination):
a ≈ 14.29, b ≈ −1.57, c ≈ 0.89

Fitted Parabola

ŷ = 14.29 − 1.57x + 0.89x²

The positive coefficient of x² (c = 0.89) indicates increasing marginal cost — as output increases, each additional unit costs more to produce. This is a classic pattern in manufacturing economics.

Correlation & Regression

Unit IV · Lecture 25

🎬 Interactive: Parabola Fitter

Enter data to fit both a straight line and a parabola, and compare visually:

x values:

y values:

Correlation & Regression

Unit IV · Lecture 25

📐 General Polynomial Fitting

📖 General Form

A polynomial of degree k has the form:

y = a₀ + a₁x + a₂x² + a₃x³ + … + aₖxᵏ

This has (k+1) unknown coefficients and requires (k+1) normal equations.

📖 The (k+1) Normal Equations

The j-th normal equation (j = 0, 1, 2, …, k) is obtained by multiplying y = a₀ + a₁x + … + aₖxᵏ by xʲ and summing:

Σxʲy = a₀Σxʲ + a₁Σxʲ⁺¹ + a₂Σxʲ⁺² + … + aₖΣxʲ⁺ᵏ

This gives a system of (k+1) linear equations in (k+1) unknowns, which can be written in matrix form as:

[Σx⁰ Σx¹ … Σxᵏ ] [a₀] [Σy ]
[Σx¹ Σx² … Σxᵏ⁺¹] [a₁] = [Σxy ]
[ ⋮ ⋮ ⋱ ⋮ ] [⋮ ] [ ⋮ ]
[Σxᵏ Σxᵏ⁺¹ … Σx²ᵏ] [aₖ] [Σxᵏy]

💡 Matrix Notation: This is X'Xa = X'y, where X is the design matrix, a is the coefficient vector, and y is the response vector. The solution is a = (X'X)⁻¹X'y — the standard formula from linear regression.

Correlation & Regression

Unit IV · Lecture 25

📐 Cubic Fit: y = a + bx + cx² + dx³

📖 Four Normal Equations

Σy = na + bΣx + cΣx² + dΣx³
Σxy = aΣx + bΣx² + cΣx³ + dΣx⁴
Σx²y = aΣx² + bΣx³ + cΣx⁴ + dΣx⁵
Σx³y = aΣx³ + bΣx⁴ + cΣx⁵ + dΣx⁶

📖 When to Use Cubic

1. The scatter plot shows an S-shape or inflection point.

2. The quadratic fit leaves systematic residual patterns.

3. The data has a change in curvature (concave up then concave down, or vice versa).

4. Domain knowledge suggests a cubic relationship (e.g., volume vs. side length).

📖 Computation Requirements

Table columns needed: x, y, xy, x²y, x³y, x², x³, x⁴, x⁵, x⁶

Sums needed: Σx through Σx⁶, Σy, Σxy, Σx²y, Σx³y

Minimum data points: 4 (but 10+ recommended)

Equations to solve: 4 simultaneous equations

In practice, cubic and higher fits are computed using software (Python, R, Excel) rather than by hand.

Correlation & Regression

Unit IV · Lecture 25

🎯 Choosing the Right Polynomial Degree

📖 The Overfitting Problem

A polynomial of degree (n−1) can pass through all n data points exactly (SSE = 0). But this "perfect" fit is useless for prediction — it captures noise, not the true relationship. This is called overfitting.

✅ Guidelines for Choosing Degree

1. Start with degree 1 (linear). If residuals show no pattern, stop.

2. Try degree 2 (quadratic) if residuals show curvature. If the improvement is significant and c is statistically significant, use it.

3. Try degree 3 only if degree 2 still shows systematic residuals.

4. Prefer the simplest model that adequately fits the data. This is the principle of parsimony (Occam's Razor).

5. Never use degree ≥ (n−1) — it will overfit.

⚠️ Signs of Overfitting

1. The fitted curve oscillates wildly between data points.

2. Adding more degree barely reduces SSE but makes the curve more complex.

3. Predictions outside the data range are wildly unreasonable.

4. The coefficients are very large with alternating signs.

5. The model performs poorly on new, unseen data.

💬 Rule of Thumb: In most practical applications, a degree 1 or 2 polynomial is sufficient. Degree 3 and higher are rarely needed unless the data clearly demands it. When in doubt, prefer simplicity.

Correlation & Regression

Unit IV · Lecture 25

⚖️ Linear vs Quadratic vs Cubic — Comparison

Feature	Linear (Degree 1)	Quadratic (Degree 2)	Cubic (Degree 3)
Equation	y = a + bx	y = a + bx + cx²	y = a + bx + cx² + dx³
Parameters	2 (a, b)	3 (a, b, c)	4 (a, b, c, d)
Normal equations	2	3	4
Shape	Straight line	U-shape or inverted U	S-shape or complex curve
Min. data points	2	3	4
Computation	Easy (by hand)	Moderate (by hand)	Hard (use software)
Overfitting risk	Low	Moderate	Higher
Best for	Linear trends	Curved trends with one bend	Trends with two bends (inflection)

💡 Parsimony Principle: Always prefer the simplest model (lowest degree) that adequately captures the data pattern. A model with fewer parameters is easier to interpret, less likely to overfit, and more robust for predictions.

Correlation & Regression

Unit IV · Lecture 25

🔑 Properties of Polynomial Least Squares

The fitted polynomial minimises SSE: Among all polynomials of the same degree, the least squares polynomial has the smallest sum of squared errors.
Residuals sum to zero: Σ(yᵢ − ŷᵢ) = 0 for any polynomial fit (from the first normal equation with x⁰ = 1).
The system is always solvable: As long as you have at least (k+1) distinct x values, the normal equations have a unique solution for a degree-k polynomial.
Higher degree always reduces SSE: Adding a higher-degree term can never increase SSE. But it may not significantly reduce it either — the improvement may be just noise-fitting.
Polynomials are global approximators: Every data point affects the entire curve. This is different from piecewise methods (splines) where each point only affects its local region.
Extrapolation is dangerous: Polynomials — especially high-degree ones — can behave wildly outside the data range. Never extrapolate far beyond your data with polynomials.
Connection to Taylor series: Polynomial fitting is related to Taylor's theorem — any smooth function can be approximated locally by a polynomial. The more terms, the better the local approximation.

Correlation & Regression

Unit IV · Lecture 25

🔑 Summary: Fitting of Polynomials

Polynomial fitting extends least squares to non-linear relationships by fitting y = a₀ + a₁x + a₂x² + … + aₖxᵏ.
A second-degree parabola (y = a + bx + cx²) requires 3 normal equations and is the most common non-linear fit.
The normal equations follow a pattern: the j-th equation involves Σxʲy on the left and Σxʲ through Σxʲ⁺ᵏ on the right.
A degree-k polynomial needs at least k+1 data points and produces (k+1) normal equations.
Choose the simplest model (lowest degree) that adequately fits the data — the principle of parsimony.
Overfitting occurs when the degree is too high — the model captures noise instead of the true pattern.
Never extrapolate far beyond the data range with polynomials — they can behave erratically outside the observed range.
In practice: Linear and quadratic fits cover most real-world needs. Higher degrees are computed using software.

Correlation & Regression

Unit IV · Lecture 25

🧠 Quick Quiz — Lecture 25

Q1: To fit a second-degree polynomial, you need at least:

A) 2 data points

B) 3 data points

C) 4 data points

D) 5 data points

Q2: How many normal equations are needed for a cubic polynomial?

A) 2

B) 3

C) 4

D) 6

Q3: The second normal equation for a parabola y = a + bx + cx² is:

A) Σy = na + bΣx + cΣx²

B) Σxy = aΣx + bΣx² + cΣx³

C) Σx²y = aΣx² + bΣx³ + cΣx⁴

D) Σxy = a + bx + cx²

Q4: Overfitting occurs when:

A) The polynomial degree is too low

B) The polynomial degree is too high, capturing noise

C) There are too few data points

D) The data is linear

Q5: A polynomial of degree (n−1) through n data points will:

A) Always underfit

B) Pass through all points exactly (SSE = 0)

C) Give the best predictions

D) Be a straight line

Q6: The principle of parsimony suggests:

A) Always use the highest degree polynomial

B) Use the simplest model that adequately fits the data

C) Always use a linear model

D) Use as many parameters as data points

Q7: If the coefficient c in y = a + bx + cx² is positive, the parabola opens:

A) Upward (U-shape)

B) Downward (inverted U)

C) To the right

D) It's a straight line

Q8: Why is extrapolation with polynomials dangerous?

A) Polynomials are always linear

B) Polynomials can behave wildly outside the data range

C) Polynomials cannot make predictions

D) Polynomials only work for integer values

Correlation & Regression

Unit IV · Lecture 25

📝 Homework & Practice

📝 Practice Tasks

1. Fit a parabola y = a + bx + cx² to: x = 1, 2, 3, 4, 5 and y = 1, 4, 9, 16, 25. What do you notice?

2. Fit a second-degree polynomial to the following data and predict y at x = 6:
x: 1, 2, 3, 4, 5
y: 5, 8, 13, 20, 29

3. For the data in Problem 2, also fit a straight line. Compare the SSE of both models. Which is better?

4. Write down the four normal equations needed to fit a cubic polynomial y = a + bx + cx² + dx³.

5. Explain why a polynomial of degree (n−1) always gives SSE = 0 for n data points, and why this is not desirable.

6. A scatter plot shows data that curves upward on both sides (U-shape). What degree polynomial would you try first? Why?

7. BONUS: For the parabola y = a + bx + cx², show that the vertex (minimum or maximum point) is at x = −b/(2c). Use this to find the vertex of ŷ = 14.29 − 1.57x + 0.89x² from Example 2.

Correlation & Regression

Unit IV · Lecture 25

Thank You!

Upcoming

Lecture 26: Correlation — Karl Pearson's Coefficient of Correlation
Lecture 27: Rank Correlation — Spearman's Coefficient
Lecture 28: Regression Lines & Their Properties

Unit IV · Lecture 25 · BDS113 Introduction to Statistics · B.Sc. Data Science, Semester II