Evaluating regression models

Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), R-squared (R²), and Adjusted R-squared are commonly used metrics for evaluating regression models.

Mean Absolute Error (MAE)

Mean Absolute Error (MAE) measures the average magnitude of the errors in a set of predictions, without considering their direction. It is the average of the absolute differences between predicted values and actual values.

$$ \text{MAE} = \frac{1}{n} \sum_{i=1}^{n} | y_i - \hat{y}_i | $$

• \( y_i \): Actual value.
• \( \hat{y}_i \): Predicted value.
• \( n \): Number of observations.

Key Features:

• Interpretability: MAE is easy to understand as it represents the average error in the same units as the target variable.
• Sensitivity: Less sensitive to outliers compared to MSE and RMSE.

Mean Squared Error (MSE)

Mean Squared Error (MSE) measures the average of the squares of the errors. It penalizes larger errors more than smaller ones, making it useful when large errors are particularly undesirable.

$$ \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 $$

• \( y_i \): Actual value.
• \( \hat{y}_i \): Predicted value.
• \( n \): Number of observations.

Key Features:

• Penalty: Squaring the errors penalizes larger errors more than smaller errors.
• Unit: The result is in squared units of the target variable, which can be less interpretable.

Root Mean Squared Error (RMSE)

Root Mean Squared Error (RMSE) is the square root of the MSE. It provides an error metric in the same units as the target variable, making it more interpretable than MSE.

$$ \text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2} $$

• \( y_i \): Actual value.
• \( \hat{y}_i \): Predicted value.
• \( n \): Number of observations.

Key Features:

• Interpretability: RMSE is in the same units as the target variable.
• Sensitivity: Like MSE, RMSE is sensitive to large errors.

R-squared (R²)

R-squared (R²), also known as the coefficient of determination, measures the proportion of the variance in the dependent variable that is predictable from the independent variables. It provides an indication of the goodness of fit of a model.

$$ R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}i)^2}{\sum{i=1}^{n} (y_i - \bar{y})^2} $$

• \( y_i \): Actual value.
• \( \hat{y}_i \): Predicted value.
• \( \bar{y} \): Mean of the actual values.
• \( n \): Number of observations.

Key Features:

• Range: R² ranges from 0 to 1. An R² of 1 indicates a perfect fit, while an R² of 0 indicates that the model does not explain any of the variance in the target variable.
• Interpretation: Higher values indicate better fit, but it does not imply causation.

Adjusted R-squared

Adjusted R-squared adjusts the R-squared value based on the number of predictors in the model. It penalizes for adding predictors that do not improve the model significantly, providing a more accurate measure of model fit for multiple regression models.

$$ \text{Adjusted } R^2 = 1 - \left( \frac{(1 - R^2) \cdot (n - 1)}{n - k - 1} \right) $$

• \( R^2 \): R-squared value.
• \( n \): Number of observations.
• \( k \): Number of predictors.

Key Features:

• Adjustment: Takes into account the number of predictors, preventing overfitting.
• Comparison: Useful for comparing models with different numbers of predictors.

Summary of Metrics

1. Mean Absolute Error (MAE):

   • Measures the average magnitude of errors in the same units as the target variable.
   • Less sensitive to outliers.

2. Mean Squared Error (MSE):

   • Measures the average of the squares of the errors.
   • Penalizes larger errors more heavily.

3. Root Mean Squared Error (RMSE):

   • Square root of MSE, providing an error metric in the same units as the target variable.
   • Sensitive to large errors.

4. R-squared (R²):

   • Proportion of variance in the dependent variable that is predictable from the independent variables.
   • Ranges from 0 to 1, with higher values indicating better fit.

5. Adjusted R-squared:

   • Adjusted version of R² that takes into account the number of predictors.
   • Penalizes for adding predictors that do not improve the model significantly.

Python Example

Here’s how you can calculate these metrics using the sklearn library:

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

# Sample data
y_true = np.array([3, -0.5, 2, 7])
y_pred = np.array([2.5, 0.0, 2, 8])

# Calculate MAE
mae = mean_absolute_error(y_true, y_pred)
print("Mean Absolute Error (MAE):", mae)

# Calculate MSE
mse = mean_squared_error(y_true, y_pred)
print("Mean Squared Error (MSE):", mse)

# Calculate RMSE
rmse = np.sqrt(mse)
print("Root Mean Squared Error (RMSE):", rmse)

# Calculate R-squared
r2 = r2_score(y_true, y_pred)
print("R-squared (R²):", r2)

# Calculate Adjusted R-squared (manually, for example purposes)
n = len(y_true)
k = 1  # number of predictors (for simplicity)
adjusted_r2 = 1 - ((1 - r2) * (n - 1)) / (n - k - 1)
print("Adjusted R-squared:", adjusted_r2)