Linear Regression Explained

Linear regression is a supervised learning algorithm used to model the relationship between one or more independent variables (features) and a dependent variable (target). It assumes a linear relationship between the input features and the target, represented by a straight-line equation.

Important concepts

• Simple linear regression. There is only one independent variable, and its relationship to the target is modelled with a straight line.
• Multiple linear regression. The same idea extended to multiple independent variables, which allows more complex relationships to be modelled.
• Coefficients. The coefficients, or weights, represent the slope of the line. Each one tells you how much the target changes for a one-unit change in the corresponding independent variable.
• Intercept. The intercept term is the value of the target when all independent variables are zero.

What that looks like in code

The concepts above map directly onto attributes on a fitted model, which is the fastest way to make them concrete. This is scikit-learn's own example, using data built so that the answer is known in advance: the target is 1 * x₀ + 2 * x₁ + 3.

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
# y = 1 * x_0 + 2 * x_1 + 3
y = np.dot(X, np.array([1, 2])) + 3

reg = LinearRegression().fit(X, y)

print(reg.coef_)       # array([1., 2.])
print(reg.intercept_)  # 3.0
print(reg.score(X, y)) # 1.0
print(reg.predict(np.array([[3, 5]])))  # array([16.])

Read the output against the definitions. coef_ came back as [1., 2.], which is the slope on each of the two features, and intercept_ came back as 3.0. Those are exactly the numbers the data was built from, so the model recovered the relationship it was shown.

score returns R², the proportion of variance the model explains. Here it is 1.0 because the data is perfectly linear with no noise in it. Real data never does this, and a score of 1.0 on a real dataset is a sign you have leaked the target into your features rather than a sign you have done well.

Where it is used

Linear regression is widely used for prediction and forecasting across economics, finance, healthcare, and other fields. It is also a common baseline model for evaluating whether a more complex machine learning algorithm is actually earning its complexity. If a gradient-boosted ensemble cannot beat a straight line on your problem, that is worth knowing before you ship the ensemble.