Regression → Logistic Regression
The story is not a list of algorithms. It is a sequence of answers to different mathematical problems:
We start with least squares, discover why its assumptions matter, add Ridge and Lasso when estimation becomes unstable or high-dimensional, generalize to GLMs, and arrive at logistic regression. Then evaluation follows the thing the model actually outputs.
1. OLS: why does the least-squares equation exist?
Suppose the conditional mean is linear:
Write the observed outcome as
OLS chooses coefficients that make the squared residuals as small as possible:
Differentiate:
Therefore:
Move the points 👇
🖱️ Grab any numbered point and move it. The OLS line and residual gaps update live.
2. Assumptions: why do we need them?
The important interview distinction: some assumptions give unbiased/consistent estimation; others mainly justify standard errors, confidence intervals and classical tests. A violation does not automatically mean “the model is useless.” It means a particular guarantee may disappear.
| Assumption | Why it exists | If violated |
|---|---|---|
| Correct conditional mean / linearity in parameters | Defines what OLS is estimating. | Misspecified mean → biased/inconsistent interpretation; systematic residual pattern. |
| Exogeneity \(E[\varepsilon\mid X]=0\) | Prevents predictors being systematically related to omitted error. | Endogeneity → coefficient bias; causal interpretation can fail. |
| Independent observations/errors | Controls dependence in the sampling process. | Often wrong standard errors; time/cluster structure needs appropriate modeling. |
| Homoscedasticity \(Var(\varepsilon\mid X)=\sigma^2\) | Supports classical OLS variance formulas and efficiency under Gauss–Markov conditions. | Coefficients can remain unbiased, but naive standard errors/tests can be wrong. Use robust SE where appropriate. |
| No perfect multicollinearity | Makes \(X^TX\) invertible and coefficients identifiable. | No unique OLS coefficient vector. Near-collinearity → huge variance/unstable coefficients. |
| Normal errors | Convenient for exact small-sample t/F inference. | OLS need not become biased. Exact finite-sample inference may fail; large-sample methods can still work. |
Residual diagnostics: the practical translation
Curvature → mean misspecification.
Funnel → heteroscedasticity.
Strong tail departures → non-normal errors; mainly an inference concern.
One observation can disproportionately move the fitted coefficients.
Large coefficient standard errors and unstable signs/magnitudes.
3. Ridge: what if \(X^TX\) is unstable?
Ridge changes the optimization problem:
Adding \(\lambda I\) makes the system numerically better behaved. Coefficients are shrunk toward zero, trading some bias for lower variance.
Philosophy: if the data cannot cleanly distinguish two correlated predictors, do not demand wildly different coefficients just to fit noise.
4. Lasso: can the model select variables while fitting?
The L1 geometry creates corners. Those corners make exact zero coefficients common:
So Lasso performs shrinkage and variable selection. With strongly correlated features, which feature gets selected can be unstable; Elastic Net combines L1 and L2 penalties.
5. GLM: the conceptual leap
A Generalized Linear Model separates three ideas:
Random component: choose a response distribution from the exponential-family world.
Systematic component: \(\eta=X\beta\).
Link: connect the expected response to the linear predictor: \(g(\mu)=X\beta\).
| Model | Distribution | Link | Mean |
|---|---|---|---|
| Linear regression | Gaussian | Identity | \(\mu=X\beta\) |
| Logistic regression | Binomial | Logit | \(\log\frac{p}{1-p}=X\beta\) |
| Poisson regression | Poisson | Log | \(\log\mu=X\beta\) |
Big idea: GLM does not say “everything must be a straight line in Y.” It says the right transformed mean can be linear in the predictors.
6. Logistic regression: where did log-odds come from?
The problem
For binary \(Y\in\{0,1\}\), we want \(P(Y=1\mid X)=p\). A straight line \(X\beta\) can be less than 0 or greater than 1, so it cannot itself be a probability.
Step 1 — probability
Step 2 — odds
Odds compare success to failure. Unlike probability, odds can range from zero to infinity.
Step 3 — log-odds
Now we have a real-valued quantity, so a linear predictor can model it:
Step 4 — solve backward
Play with the log-odds
Probability: Odds: Log-odds:
Coefficient interpretation
Increase \(x_j\) by one unit while holding other variables fixed:
So \(e^{\beta_j}\) is an odds ratio. It is not a fixed probability increase; the probability effect depends on the starting probability.
Why log loss appears naturally
For Bernoulli observations:
Maximizing likelihood is equivalent to minimizing:
This is why logistic regression is evaluated not only by threshold metrics, but also by probability-quality metrics such as log loss and calibration.
7. Evaluation: measure what the model is supposed to do
Regression
MAE is easier to interpret; MSE/RMSE punish large errors more; R² compares residual variation with a mean-only baseline.
Classification
- Log loss: rewards calibrated probabilities and punishes confident wrong predictions.
- ROC-AUC: ranking quality across thresholds.
- PR-AUC: often more informative when positives are rare.
- Precision: among predicted positives, how many are positive?
- Recall: among actual positives, how many did we catch?
- Calibration: among cases predicted ~0.8, is the observed positive rate near 0.8?
Threshold is a decision layer
The model produces \(p=P(Y=1\mid X)\). A threshold converts probability into an action:
Do not tune the threshold on the test set. Choose it using validation/CV and the business/error costs, then report final test performance once.
8. ROC + AUC: the model has more than one threshold
First: the philosophy
A classifier usually gives a score or probability. One threshold is just one operating point. ROC asks: “If I slide the threshold, how much recall do I get, and how many negatives do I accidentally catch?”
ROC curve: plot TPR against FPR for many thresholds. AUC is the area under that curve.
Why not just accuracy?
Because accuracy hides the threshold and class balance. If only 1% of cases are positive, predicting “negative” every time gives 99% accuracy and catches exactly zero positives. 😅
Threshold playground
Move the threshold below. Watch precision and recall trade places. The same probability model can look very different after the decision threshold changes.
9. Tiny hand calculations — one table, no calculator gymnastics
Use these tiny examples as interview flashcards. The numbers are deliberately small so you can calculate them on paper.
| Metric | Formula | Input | Calculation | Answer | What it asks | Higher? | Threshold? | Typical trap | Memory hook |
|---|---|---|---|---|---|---|---|---|---|
| MAE | Σ|e|/n | e=1,0,-1,-1 | (1+0+1+1)/4 | 0.75 | Average error size | Lower | No | Doesn't punish big errors extra | “Average distance” |
| MSE | Σe²/n | e=1,0,-1,-1 | (1+0+1+1)/4 | 0.75 | Error with squared punishment | Lower | No | Units are squared | “Big mistakes hurt” |
| RMSE | √MSE | MSE=.75 | √.75 | 0.866 | Error in target units | Lower | No | Still sensitive to outliers | “MSE back home” |
| R² | 1-SSE/SST | SSE=3,SST=5 | 1-3/5 | 0.40 | Gain over mean baseline | Higher | No | Not % accuracy | “Beat the mean” |
| Accuracy | (TP+TN)/N | TP=3,TN=5,N=10 | 8/10 | 0.80 | All decisions correct | Higher | Yes | Can lie under imbalance | “Total right” |
| Precision | TP/(TP+FP) | TP=3,FP=1 | 3/4 | 0.75 | Are my positive alerts clean? | Higher | Yes | Ignores missed positives | “Predicted + → right?” |
| Recall / TPR | TP/(TP+FN) | TP=3,FN=1 | 3/4 | 0.75 | Did I catch the positives? | Higher | Yes | Ignores false alarms | “Actual + → caught?” |
| F1 | 2PR/(P+R) | P=.75,R=.75 | 1.125/1.5 | 0.75 | Balance precision + recall | Higher | Yes | Not a probability metric | “Meet in the middle” |
| FPR | FP/(FP+TN) | FP=1,TN=5 | 1/6 | 0.167 | How many negatives got flagged? | Lower | Yes | Not the same as 1-precision | “False alarm rate” |
| ROC-AUC | Area under ROC | 4 positives × 4 negatives | 13 winning pairs / 16 | 0.8125 | Ranking quality | Higher | No single one | Doesn't choose threshold | “Rank + vs -” |
| Log loss | -Σ[ylnp+(1-y)ln(1-p)]/n | y=(1,0), p=(.9,.2) | -[ln.9+ln.8]/2 | 0.168 | Probability quality | Lower | No | Confident wrong = huge pain | “Don't lie confidently” |
| Brier score | Σ(p-y)²/n | y=(1,0), p=(.9,.2) | (.1²+.2²)/2 | 0.025 | Probability calibration/error | Lower | No | Not the same as AUC | “Probability distance” |
Regression mini-example: actual y=(3,5,4,6), prediction ŷ=(2,5,5,7). Errors are (1,0,-1,-1), mean y=4.5, SSE=3 and SST=5. Hence MAE=.75, MSE=.75, RMSE≈.866 and R²=.40.
ROC-AUC mini-example: imagine 4 positives and 4 negatives. If the model ranks the positive above the negative in 13 of 16 possible positive-negative pairs, AUC=13/16=.8125. No threshold is needed for that calculation.
10. scikit-learn: one compact workflow
Use a pipeline so preprocessing is learned only from training folds. Scale features for Ridge/Lasso and logistic regularization because the penalty acts on coefficient magnitudes.
from sklearn.datasets import load_diabetes, load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso, LogisticRegression
from sklearn.metrics import (
mean_absolute_error, mean_squared_error, r2_score,
accuracy_score, precision_score, recall_score,
roc_auc_score, roc_curve, auc, log_loss, f1_score, brier_score_loss
)
# ---------------- Regression ----------------
X, y = load_diabetes(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(
X, y, test_size=0.20, random_state=42
)
models = {
"OLS": Pipeline([
("scale", StandardScaler()),
("model", LinearRegression())
]),
"Ridge": Pipeline([
("scale", StandardScaler()),
("model", Ridge(alpha=1.0))
]),
"Lasso": Pipeline([
("scale", StandardScaler()),
("model", Lasso(alpha=0.01, max_iter=10000))
])
}
for name, model in models.items():
model.fit(Xtr, ytr)
pred = model.predict(Xte)
print(name)
print("MAE :", mean_absolute_error(yte, pred))
print("RMSE:", mean_squared_error(yte, pred) ** 0.5)
print("R2 :", r2_score(yte, pred))
# ---------------- Logistic ----------------
X, y = load_breast_cancer(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(
X, y, test_size=0.20, stratify=y, random_state=42
)
clf = Pipeline([
("scale", StandardScaler()),
("model", LogisticRegression(
C=1.0, penalty="l2", max_iter=2000
))
])
clf.fit(Xtr, ytr)
p = clf.predict_proba(Xte)[:, 1]
threshold = 0.50
pred = (p >= threshold).astype(int)
print("Accuracy :", accuracy_score(yte, pred))
print("Precision:", precision_score(yte, pred))
print("Recall :", recall_score(yte, pred))
print("F1 :", f1_score(yte, pred))
print("ROC-AUC :", roc_auc_score(yte, p))
fpr, tpr, thresholds = roc_curve(yte, p)
print("AUC check:", auc(fpr, tpr))
print("Log loss :", log_loss(yte, p))
print("Brier :", brier_score_loss(yte, p))Interview mental model
| Question | Answer |
|---|---|
| What is OLS? | Estimate the conditional mean by minimizing squared residuals. |
| Why Ridge? | L2 shrinkage reduces variance and stabilizes correlated features. |
| Why Lasso? | L1 penalty can drive coefficients exactly to zero. |
| Why GLM? | Let the response distribution and link match the outcome type while keeping a linear predictor. |
| Why logistic? | Binary response + binomial likelihood + logit link gives valid probabilities and a linear log-odds model. |
| Why log loss? | It is the negative Bernoulli log-likelihood, so it directly evaluates probabilistic predictions. |
No comments:
Post a Comment