Sunday, September 20, 2026

Regression → Logistic | Math & Evaluation
MATH HEAVYINTERACTIVEINTERVIEW READY

Regression → Logistic Regression

The story is not a list of algorithms. It is a sequence of answers to different mathematical problems:

\[\text{estimate a mean} \rightarrow \text{stabilize it} \rightarrow \text{select variables} \rightarrow \text{change the response distribution} \rightarrow \text{model probability}\]

We start with least squares, discover why its assumptions matter, add Ridge and Lasso when estimation becomes unstable or high-dimensional, generalize to GLMs, and arrive at logistic regression. Then evaluation follows the thing the model actually outputs.

1. OLS: why does the least-squares equation exist?

Suppose the conditional mean is linear:

\[E[Y\mid X]=X\beta\]

Write the observed outcome as

\[y=X\beta+\varepsilon\]

OLS chooses coefficients that make the squared residuals as small as possible:

\[\hat\beta=\arg\min_\beta\|y-X\beta\|_2^2\]

Differentiate:

\[\frac{\partial}{\partial\beta}(y-X\beta)^T(y-X\beta)=-2X^T(y-X\beta)=0\]

Therefore:

\[X^TX\hat\beta=X^Ty\qquad\Rightarrow\qquad\boxed{\hat\beta=(X^TX)^{-1}X^Ty}\]
Geometric meaning: \(X\hat\beta\) is the orthogonal projection of \(y\) onto the column space of \(X\). The residual is perpendicular to every column of \(X\): \(X^T\hat\varepsilon=0\).

Move the points 👇

🖱️ Grab any numbered point and move it. The OLS line and residual gaps update live.

2. Assumptions: why do we need them?

The important interview distinction: some assumptions give unbiased/consistent estimation; others mainly justify standard errors, confidence intervals and classical tests. A violation does not automatically mean “the model is useless.” It means a particular guarantee may disappear.

AssumptionWhy it existsIf violated
Correct conditional mean / linearity in parametersDefines what OLS is estimating.Misspecified mean → biased/inconsistent interpretation; systematic residual pattern.
Exogeneity \(E[\varepsilon\mid X]=0\)Prevents predictors being systematically related to omitted error.Endogeneity → coefficient bias; causal interpretation can fail.
Independent observations/errorsControls dependence in the sampling process.Often wrong standard errors; time/cluster structure needs appropriate modeling.
Homoscedasticity \(Var(\varepsilon\mid X)=\sigma^2\)Supports classical OLS variance formulas and efficiency under Gauss–Markov conditions.Coefficients can remain unbiased, but naive standard errors/tests can be wrong. Use robust SE where appropriate.
No perfect multicollinearityMakes \(X^TX\) invertible and coefficients identifiable.No unique OLS coefficient vector. Near-collinearity → huge variance/unstable coefficients.
Normal errorsConvenient for exact small-sample t/F inference.OLS need not become biased. Exact finite-sample inference may fail; large-sample methods can still work.
Classic trap: predictors do not need to be normally distributed. Normality, when invoked for classical OLS inference, concerns the conditional error distribution—not the raw X columns.

Residual diagnostics: the practical translation

Residual vs fitted
Curvature → mean misspecification.
Funnel → heteroscedasticity.
Q–Q plot
Strong tail departures → non-normal errors; mainly an inference concern.
Leverage / Cook's distance
One observation can disproportionately move the fitted coefficients.
Correlation among X
Large coefficient standard errors and unstable signs/magnitudes.

3. Ridge: what if \(X^TX\) is unstable?

Ridge changes the optimization problem:

\[\hat\beta_{ridge}=\arg\min_\beta\left(\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2\right)\]
\[\boxed{\hat\beta=(X^TX+\lambda I)^{-1}X^Ty}\]

Adding \(\lambda I\) makes the system numerically better behaved. Coefficients are shrunk toward zero, trading some bias for lower variance.

Philosophy: if the data cannot cleanly distinguish two correlated predictors, do not demand wildly different coefficients just to fit noise.

4. Lasso: can the model select variables while fitting?

\[\hat\beta_{lasso}=\arg\min_\beta\left(\|y-X\beta\|_2^2+\lambda\|\beta\|_1\right)\]

The L1 geometry creates corners. Those corners make exact zero coefficients common:

\[|\beta_1|+|\beta_2|\le t\]

So Lasso performs shrinkage and variable selection. With strongly correlated features, which feature gets selected can be unstable; Elastic Net combines L1 and L2 penalties.

5. GLM: the conceptual leap

A Generalized Linear Model separates three ideas:

Random component: choose a response distribution from the exponential-family world.

Systematic component: \(\eta=X\beta\).

Link: connect the expected response to the linear predictor: \(g(\mu)=X\beta\).

ModelDistributionLinkMean
Linear regressionGaussianIdentity\(\mu=X\beta\)
Logistic regressionBinomialLogit\(\log\frac{p}{1-p}=X\beta\)
Poisson regressionPoissonLog\(\log\mu=X\beta\)

Big idea: GLM does not say “everything must be a straight line in Y.” It says the right transformed mean can be linear in the predictors.

6. Logistic regression: where did log-odds come from?

The problem

For binary \(Y\in\{0,1\}\), we want \(P(Y=1\mid X)=p\). A straight line \(X\beta\) can be less than 0 or greater than 1, so it cannot itself be a probability.

Step 1 — probability

\[0

Step 2 — odds

\[\text{odds}=\frac{p}{1-p}\in(0,\infty)\]

Odds compare success to failure. Unlike probability, odds can range from zero to infinity.

Step 3 — log-odds

\[\text{logit}(p)=\log\left(\frac{p}{1-p}\right)\in(-\infty,\infty)\]

Now we have a real-valued quantity, so a linear predictor can model it:

\[\boxed{\log\frac{p}{1-p}=\eta=X\beta}\]

Step 4 — solve backward

\[\frac{p}{1-p}=e^\eta\]
\[p=e^\eta(1-p)\]
\[p(1+e^\eta)=e^\eta\]
\[\boxed{p=\frac{1}{1+e^{-\eta}}=\sigma(\eta)}\]
The philosophy: probability is bounded → odds remove the upper bound → log removes the multiplicative scale and creates a symmetric real-valued scale → the linear predictor lives there → inverse-logit maps it back to a valid probability.

Play with the log-odds

Probability:   Odds:   Log-odds:

Coefficient interpretation

\[\log\frac{p}{1-p}=\beta_0+\beta_jx_j+\cdots\]

Increase \(x_j\) by one unit while holding other variables fixed:

\[\Delta\log(odds)=\beta_j\qquad\Rightarrow\qquad \frac{odds_{new}}{odds_{old}}=e^{\beta_j}\]

So \(e^{\beta_j}\) is an odds ratio. It is not a fixed probability increase; the probability effect depends on the starting probability.

Why log loss appears naturally

For Bernoulli observations:

\[L(\beta)=\prod_i p_i^{y_i}(1-p_i)^{1-y_i}\]
\[\ell(\beta)=\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)]\]

Maximizing likelihood is equivalent to minimizing:

\[\boxed{-\ell(\beta)=\text{binary cross-entropy / log loss}}\]

This is why logistic regression is evaluated not only by threshold metrics, but also by probability-quality metrics such as log loss and calibration.

7. Evaluation: measure what the model is supposed to do

Regression

\[MAE=\frac1n\sum|y_i-\hat y_i|\]
\[MSE=\frac1n\sum(y_i-\hat y_i)^2\]
\[RMSE=\sqrt{MSE}\]
\[R^2=1-\frac{\sum(y-\hat y)^2}{\sum(y-\bar y)^2}\]

MAE is easier to interpret; MSE/RMSE punish large errors more; R² compares residual variation with a mean-only baseline.

Classification

  • Log loss: rewards calibrated probabilities and punishes confident wrong predictions.
  • ROC-AUC: ranking quality across thresholds.
  • PR-AUC: often more informative when positives are rare.
  • Precision: among predicted positives, how many are positive?
  • Recall: among actual positives, how many did we catch?
  • Calibration: among cases predicted ~0.8, is the observed positive rate near 0.8?

Threshold is a decision layer

The model produces \(p=P(Y=1\mid X)\). A threshold converts probability into an action:

\[\hat y=1\quad\text{if}\quad p\ge t\]

Do not tune the threshold on the test set. Choose it using validation/CV and the business/error costs, then report final test performance once.

\n

8. ROC + AUC: the model has more than one threshold

First: the philosophy

A classifier usually gives a score or probability. One threshold is just one operating point. ROC asks: “If I slide the threshold, how much recall do I get, and how many negatives do I accidentally catch?”

\[TPR=\frac{TP}{TP+FN}=\text{Recall}\qquad FPR=\frac{FP}{FP+TN}=1-\text{Specificity}\]

ROC curve: plot TPR against FPR for many thresholds. AUC is the area under that curve.

Nice intuition: AUC can be read as the chance that the model gives a randomly chosen positive a higher score than a randomly chosen negative (with ties handled appropriately). So AUC is mainly about ranking, not picking your final threshold.

Why not just accuracy?

Because accuracy hides the threshold and class balance. If only 1% of cases are positive, predicting “negative” every time gives 99% accuracy and catches exactly zero positives. 😅

Threshold playground

Move the threshold below. Watch precision and recall trade places. The same probability model can look very different after the decision threshold changes.

9. Tiny hand calculations — one table, no calculator gymnastics

Use these tiny examples as interview flashcards. The numbers are deliberately small so you can calculate them on paper.

MetricFormulaInputCalculationAnswerWhat it asksHigher?Threshold?Typical trapMemory hook
MAEΣ|e|/ne=1,0,-1,-1(1+0+1+1)/40.75Average error sizeLowerNoDoesn't punish big errors extra“Average distance”
MSEΣe²/ne=1,0,-1,-1(1+0+1+1)/40.75Error with squared punishmentLowerNoUnits are squared“Big mistakes hurt”
RMSE√MSEMSE=.75√.750.866Error in target unitsLowerNoStill sensitive to outliers“MSE back home”
R²1-SSE/SSTSSE=3,SST=51-3/50.40Gain over mean baselineHigherNoNot % accuracy“Beat the mean”
Accuracy(TP+TN)/NTP=3,TN=5,N=108/100.80All decisions correctHigherYesCan lie under imbalance“Total right”
PrecisionTP/(TP+FP)TP=3,FP=13/40.75Are my positive alerts clean?HigherYesIgnores missed positives“Predicted + → right?”
Recall / TPRTP/(TP+FN)TP=3,FN=13/40.75Did I catch the positives?HigherYesIgnores false alarms“Actual + → caught?”
F12PR/(P+R)P=.75,R=.751.125/1.50.75Balance precision + recallHigherYesNot a probability metric“Meet in the middle”
FPRFP/(FP+TN)FP=1,TN=51/60.167How many negatives got flagged?LowerYesNot the same as 1-precision“False alarm rate”
ROC-AUCArea under ROC4 positives × 4 negatives13 winning pairs / 160.8125Ranking qualityHigherNo single oneDoesn't choose threshold“Rank + vs -”
Log loss-Σ[ylnp+(1-y)ln(1-p)]/ny=(1,0), p=(.9,.2)-[ln.9+ln.8]/20.168Probability qualityLowerNoConfident wrong = huge pain“Don't lie confidently”
Brier scoreΣ(p-y)²/ny=(1,0), p=(.9,.2)(.1²+.2²)/20.025Probability calibration/errorLowerNoNot the same as AUC“Probability distance”

Regression mini-example: actual y=(3,5,4,6), prediction ŷ=(2,5,5,7). Errors are (1,0,-1,-1), mean y=4.5, SSE=3 and SST=5. Hence MAE=.75, MSE=.75, RMSE≈.866 and R²=.40.

ROC-AUC mini-example: imagine 4 positives and 4 negatives. If the model ranks the positive above the negative in 13 of 16 possible positive-negative pairs, AUC=13/16=.8125. No threshold is needed for that calculation.

10. scikit-learn: one compact workflow

Use a pipeline so preprocessing is learned only from training folds. Scale features for Ridge/Lasso and logistic regularization because the penalty acts on coefficient magnitudes.

from sklearn.datasets import load_diabetes, load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso, LogisticRegression
from sklearn.metrics import (
    mean_absolute_error, mean_squared_error, r2_score,
    accuracy_score, precision_score, recall_score,
    roc_auc_score, roc_curve, auc, log_loss, f1_score, brier_score_loss
)

# ---------------- Regression ----------------
X, y = load_diabetes(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(
    X, y, test_size=0.20, random_state=42
)

models = {
    "OLS": Pipeline([
        ("scale", StandardScaler()),
        ("model", LinearRegression())
    ]),
    "Ridge": Pipeline([
        ("scale", StandardScaler()),
        ("model", Ridge(alpha=1.0))
    ]),
    "Lasso": Pipeline([
        ("scale", StandardScaler()),
        ("model", Lasso(alpha=0.01, max_iter=10000))
    ])
}

for name, model in models.items():
    model.fit(Xtr, ytr)
    pred = model.predict(Xte)
    print(name)
    print("MAE :", mean_absolute_error(yte, pred))
    print("RMSE:", mean_squared_error(yte, pred) ** 0.5)
    print("R2  :", r2_score(yte, pred))

# ---------------- Logistic ----------------
X, y = load_breast_cancer(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

clf = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(
        C=1.0, penalty="l2", max_iter=2000
    ))
])

clf.fit(Xtr, ytr)
p = clf.predict_proba(Xte)[:, 1]
threshold = 0.50
pred = (p >= threshold).astype(int)

print("Accuracy :", accuracy_score(yte, pred))
print("Precision:", precision_score(yte, pred))
print("Recall   :", recall_score(yte, pred))
print("F1       :", f1_score(yte, pred))
print("ROC-AUC  :", roc_auc_score(yte, p))
fpr, tpr, thresholds = roc_curve(yte, p)
print("AUC check:", auc(fpr, tpr))
print("Log loss :", log_loss(yte, p))
print("Brier    :", brier_score_loss(yte, p))

Interview mental model

QuestionAnswer
What is OLS?Estimate the conditional mean by minimizing squared residuals.
Why Ridge?L2 shrinkage reduces variance and stabilizes correlated features.
Why Lasso?L1 penalty can drive coefficients exactly to zero.
Why GLM?Let the response distribution and link match the outcome type while keeping a linear predictor.
Why logistic?Binary response + binomial likelihood + logit link gives valid probabilities and a linear log-odds model.
Why log loss?It is the negative Bernoulli log-likelihood, so it directly evaluates probabilistic predictions.

No comments:

Post a Comment