Linear regression is a statistical method to find the best-fitting line to model the relationship between a dependent variable and one or more independent variables. It minimizes the sum of squared errors between the data points and the estimated values on the line. This method is widely used in various fields, from forecasting economic trends to improving manufacturing processes.

Linear regression plays a crucial role in statistics and machine learning by allowing analysts to create models for the relationship between a dependent variable and one or more independent variables. However, what is the actual functioning mechanism of this powerful algorithm? Let's explore the principles of statistical mechanics that drive linear regression. Linear regression aims to identify the optimal linear equation that reduces the sum of squared discrepancies between the observed data points and the estimated values lying on the line or hyperplane. From a mathematical standpoint, this can be expressed as: minimize the sum of squared differences between y and the predicted values using the given coefficients β0, β1, ..., βn.

Plot of residuals

The residuals are the vertical gaps between every data point and the regression line. Through adding up the squared residuals, we can calculate a measurement that represents the overall difference between the line and the data points. Function that determines the expenses associated with a particular process or activity. The cost function that linear regression seeks to minimize is formed by the sum of squared residuals. When we find the partial derivatives of this function for each β parameter and set them to zero, we get a set of linear equations called the normal equations.

Equations of Normalisation

Analytically solving this set of equations gives us the β values that characterise the regression line. Yet, with significant datasets and increased dimensions, numerical optimisation techniques such as gradient descent are frequently utilised.

Descent of Gradient

Gradient descent continuously updates the β coefficients towards the direction of maximum descent on the vector.

Although this mathematical representation may appear intimidating, its capacity to concisely express the connections within data is what makes linear regression a crucial tool in various fields. From forecasting economic tendencies to enhancing manufacturing procedures, the statistical basis of this algorithm remains influential in shaping our comprehension of a wide range of occurrences.

Authors: Catalin Bondari & Bohdan Boiprav