Evaluation metrics, also known as performance measures or evaluative metrics, are quantitative measurements used to evaluate the performance and quality of a model or algorithm in solving a particular problem. It provides a standardized way to evaluate and compare different models and algorithms based on specific criteria.

Error Metrics

Error metrics focus on measuring the accuracy and magnitude of errors in the forecasted values when compared to the actual values. Read more about them here.

Mean Absolute Error (MAE)

Use MAE when you want an evaluation metric that represents the average absolute difference between predicted and actual values, providing a simple and interpretable measure of overall model performance.

# Install library
!pip install scikit-learn

# Import function
from sklearn.metrics import mean_absolute_error

# Calculate MAE
mae = mean_absolute_error(actual_values, predicted_values)
print("Mean Absolute Error:", mae)

Mean Squared Error (MSE)

Consider using MSE when you want to penalize larger prediction errors more than smaller ones, making it suitable for applications where outliers or extreme values need to be emphasized in the evaluation.

# Install library
!pip install scikit-learn

# Import function
from sklearn.metrics import mean_squared_error

# Calculate MSE
mse = mean_squared_error(actual_values, predicted_values)
print("Mean Squared Error:", mse)

Stay up-to-date with our latest articles!
Subscribe to our free newsletter by entering your email address below.

Root Mean Squared Error (RMSE)

Opt for RMSE when you want the advantages of MSE but you want to express the evaluation metric in the same units as the target variable, which helps in better understanding the magnitude of prediction errors compared to the original scale of the data.

# Install library
!pip install scikit-learn

# Import libraries
from sklearn.metrics import mean_squared_error
import numpy as np

# Calculate MSE
mse = mean_squared_error(actual_values, predicted_values)

# Calculate RMSE
rmse = np.sqrt(mse)

print("RMSE:", rmse)

Mean Absolute Percentage Error (MAPE)

Use MAPE when you need an evaluation metric that expresses prediction errors as a percentage of the actual values, which is helpful for understanding the relative performance across different datasets.

# Install library
!pip install statsmodels

# Import function
from statsmodels.tools.eval_measures import mean_absolute_percentage_error

# Calculate MAPE
mape = mean_absolute_percentage_error(actual_values, predicted_values) * 100
print("MAPE:", mape, "%")


Symmetric Mean Absolute Percentage Error (SMAPE)

Consider using SMAPE when you want a percentage-based evaluation metric that avoids issues with division by zero and provides a symmetric measure of prediction accuracy.

# Import libraries
import numpy as np

# Define function
def sym_mean_absolute_percentage_error(actual, predicted):
    Calculate SMAPE (Symmetric Mean Absolute Percentage Error).
    return 2 * np.mean(np.abs(actual - predicted) / (np.abs(actual) + np.abs(predicted))) * 100

# Calculate SMAPE
smape = sym_mean_absolute_percentage_error(actual_values, predicted_values)
print("SMAPE:", smape, "%")

Mean Absolute Scaled Error (MASE)

Use MASE when you have time series data with different scales or seasonal patterns and want to compare the accuracy of forecasting models, as it measures the performance of a model relative to the mean absolute error of a naive, non-seasonal forecasting method, making it suitable for assessing forecast accuracy across various datasets.

# Install library
!pip install scikit-learn

# Import libraries
from sklearn.metrics import mean_absolute_error

def mean_absolute_scaled_error(actual, predicted):
    Calculate MASE (Mean Absolute Scaled Error).
    mae = mean_absolute_error(actual, predicted)
    naive_error = np.mean(np.abs(actual[1:] - actual[:-1]))
    return mae / naive_error

# Calculate MASE
mase = mean_absolute_scaled_error(actual_values, predicted_values)
print("MASE:", mase)

You can see a summary table here:

Penalizes large errorsNoYesYesNoNoNo
Outlier sensitivityMediumHighHighMediumMediumMedium
Original scaleYesNoYesNoNoNo
Sensitivity to division by 0NoNoNoYesYesYes

Performance Metrics

Performance metrics go beyond the conventional error magnitude measurements and play a vital role in assessing alternative aspects of forecasting models. They are a perfect complement to the error metrics. Read more about them here.

Forecast Bias

Use this metric when you want to assess the systematic overestimation or underestimation of your forecasted values compared to the actual data, helping you identify consistent errors in your predictions.

# Calculate forecast differences
differences = [predicted - actual for predicted, actual in zip(predicted_values, actual_values)]

# Calculate forecast bias
bias = sum(differences) / len(differences)
print("Forecast Bias:", bias)

Forecast Interval Coverage (FIC)

Utilize this metric when you need to evaluate the accuracy of your prediction intervals (confidence intervals) around the forecasted values, indicating how well they capture the true variability and uncertainty in future observations.

def calculate_coverage(forecasted_intervals, actual_values):
    # Calculate the number of actual values that fall within the intervals
    num_within_interval = sum((lower <= actual <= upper) for actual, (lower, upper) in zip(actual_values, forecasted_intervals))
    # Calculate the total number of observations
    total_observations = len(actual_values)
    # Calculate the coverage or FIC by dividing both values
    fic = num_within_interval / total_observations * 100
    return fic

# Calculate FIC
fic = calculate_coverage(forecasted_intervals, actual_values)
print("Forecast Interval Coverage:", fic, '%')

Prediction Direction Accuracy (PDA)

Consider using this metric when your focus is on the correct direction of predictions (e.g., up or down) rather than the specific magnitude, which is particularly relevant in financial and binary forecasting applications.

def calculate_pda(predicted_values, actual_values):
    # Initialize the value of the correct directions variable
    correct_directions = 0
    # Iterate each value
    for i in range(1, len(predicted_values)):
        # Calculate predicted and actual directions
        pred_change = predicted_values[i] - predicted_values[i - 1]
        actual_change = actual_values[i] - actual_values[i - 1]
        # Check if the predictions match the actual directions
        if (pred_change > 0 and actual_change > 0) or (pred_change < 0 and actual_change < 0):
            correct_directions += 1

    # Calculate PDA
    pda = (correct_directions / (len(predicted_values) - 1)) * 100
    return pda
# Calculate PDA
pda = calculate_pda(predicted_values, actual_values)
print("Prediction Direction Accuracy:", pda, '%')


Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *