Please rotate your device to landscape mode to view the charts.

Practice-relevant model validation: Distributional parameter risk analysis in financial model risk management

Journal: Annals of Operations Research

Date: 2023

Author: Mark Cummins, Fabian Gogolin, Fearghal Kearney, Greg Kiely, Bernard Murphy

Abstract:
An objective of model validation within organisations is to provide guidance on model selection decisions that balance the operational effectiveness and structural complexity of competing models. We consider a practice-relevant model validation scenario where a financial quantitative analysis team seeks to decide between incumbent and alternative models on the basis of parameter risk. We devise a model risk management methodology that gives a meaningful distributional assessment of parameter risk in a setting where market calibration and historical estimation procedures must be jointly applied. Such a scenario is typically driven by data constraints that preclude market calibration only. We demonstrate our proposed methodology in a natural gas storage modelling context, where model usage is necessary to support profit and loss reporting, and to inform trading and hedging strategy. We leverage our distributional parameter risk approach to devise an accessible technique to support model selection decisions.

Link: Google Scholar

Background and Context

Research Problem

Financial organizations need to balance operational effectiveness and model complexity when validating and selecting between competing risk models.

Study Focus

Analysis of parameter risk in natural gas storage modeling where both market calibration and historical estimation procedures must be jointly applied due to data constraints.

Methodology

Development of a distributional parameter risk analysis framework to support model validation decisions in financial risk management.

Comparison of Model Risk Measurements Across Different Models

  • Compares calibration accuracy between one-factor Mean Reverting Variance Gamma (MRVG-1F) and Mean Reverting Jump Diffusion (MRJD) models
  • Both models show similar levels of calibration error around 1.1%
  • Lower RMSE indicates better model fit to market data

Storage Value Distribution Characteristics Across Models

  • Shows key statistical properties of storage value distributions for both models
  • MRVG-1F shows slightly higher expected value but also higher variability
  • Opposite skewness patterns indicate different risk characteristics between models

Parameter Sensitivity Analysis for MRVG-2F Model

  • Shows sensitivity of storage value to different model parameters
  • Parameter 'c' has the highest impact on model outcomes
  • Helps identify which parameters require most careful estimation

Risk-Adjusted Bid-Offer Spreads Comparison

  • Compares the width of risk-adjusted trading ranges across models
  • MRVG-2F shows significantly wider spreads, indicating higher parameter risk
  • Wider spreads suggest need for larger risk buffers in trading decisions

Model Performance Metrics

  • Shows calibration accuracy of the more complex MRVG-2F model
  • Higher RMSE indicates increased model complexity comes with some cost to fit
  • Trade-off between model sophistication and parameter risk

Contribution and Implications

  • Provides a comprehensive framework for assessing model risk when both market calibration and historical estimation are required
  • Demonstrates how to balance model complexity against parameter risk in financial model selection
  • Offers practical guidance for setting model usage restrictions and trading limits based on parameter risk assessment

Data Sources

  • Model Comparison Chart: Based on Table 1 RMSE values for MRVG-1F and MRJD models
  • Distribution Characteristics Chart: Constructed from Table 2 statistical measures
  • Parameter Sensitivity Chart: Created using Table 5 parameter delta values
  • Bid-Offer Spreads Chart: Derived from reported percentile values in Section 4.2
  • Performance Metrics Chart: Based on Table 3 RMSE values for MRVG-2F model