Logistic regression is a classification algorithm used to model the probability of a binary outcome based on one or more predictor variables. Despite its name it is a linear model for classification rather than regression. Here are the concepts that matter.
Binary classification. Logistic regression is primarily used for binary classification, where the target variable has only two possible outcomes, typically represented as 0 and 1.
The sigmoid function. Logistic regression uses the sigmoid function to map predicted values to probabilities between 0 and 1:
σ(z) = 1 / (1 + e−z)
Where z is the linear combination of the input features and their corresponding coefficients.
Decision boundary. The model separates classes by fitting a decision boundary, typically a straight line in two dimensions, that divides the feature space into regions associated with different classes.
Cost function. Logistic regression uses cross-entropy loss to measure the difference between the predicted probabilities and the actual target values. Training is the process of minimizing it.
Regularization. L1 (Lasso) and L2 (Ridge) regularization can be applied to prevent overfitting, by penalizing large coefficients.
What that looks like in code
This is scikit-learn's own example, on the iris dataset:
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
clf = LogisticRegression(random_state=0).fit(X, y)
clf.predict(X[:2, :])
# array([0, 0])
clf.predict_proba(X[:2, :])
# array([[9.82e-01, 1.82e-02, 1.44e-08],
# [9.72e-01, 2.82e-02, 3.02e-08]])
clf.score(X, y)
# 0.97predict_proba is the part worth dwelling on, because it is the sigmoid output rather than a label. The first sample comes back as 98.2 percent class 0, and only then does predict collapse that into the label 0. Keeping the probability is usually more useful than the label: it lets you set your own threshold instead of accepting 0.5, which matters whenever a false positive and a false negative cost different amounts.
The iris example also shows that the binary case is the starting point rather than the limit. There are three classes here, and the probabilities across each row sum to 1.
Usage. Logistic regression is widely used in healthcare for predicting disease outcomes, in finance for credit risk analysis, and in marketing for customer churn prediction. Its enduring advantage over heavier models is that the coefficients remain readable, so you can say which feature pushed a decision and in which direction.
