SAEe, Valencia, Dec-22

1. Introduction

  • Receiver Operating Characteristic curve: Graphical representation of the performance of a classificator.
  • Example: the signal to noise problem in telecommunications (Peterson and Bisdall, 1953).
  • Popular in Radiology, Medicine, Machine-Learning, Psychology.
  • Economics: Berge and Jordà (2011) for bussiness cycle turning points.
  • Other references: Drehmann and Juseliius (2014), Lahiri and Wang (2013, 2016), Leiva-Leon and Perez-Quiros (2020), Ponka (2017)…

Baseline example: quarterly US GDP growth rates from 1951:1 to 2019:4

Filter recession probabilities and fix threshold level (\(\alpha\)):

Define:

  • a positive is obtained when signal is above a given threshold (say \(\alpha\), confidence level).
  • a true positive (TP) is obtained when it is a positive and the event (recession) is true.
  • a false positive (FP) is obtained when it is a positive and the event (recession) is false.

The ROC curve plots the TP rate (over total events) against the FP rates (over total non-events) as a parametric function of threshold \(\alpha\) in (0,1).

Toy example: Class. 1 is prefered to Class. 2. because the Area Under the ROC (AUROC) is bigger.

Back to US GDP data, apply to pre and post-pandemic (2020-Q1 and Q2) data.

The filtered probabilities look VERY different…

… but the AUROC for pre and postpandemic data are almost the same !!!

What is happening? Let’s take a closer look…

The recessions are still there !!!.

Problem: the number of threshold points concentrates on a tiny narrow interval.

2. Projecting the ROC curve

We add a vertical axis to the ROC curve with the threshold points…

Say

  • \(X\) the FPR axis

  • \(Y\) the TPR axis

  • \(A\) the thresholds axis

  • Define the (d)ifference in (A)rea (U)nder (p)rojections computed using a Trapezoid rule grid of \(r=1,\ldots,r\) points:

  • \(\mathrm{dAUp} = \mathrm{AU}_{XO} - \mathrm{AU}_{YO}\) ,where

  • \(\mathrm{AU}_{XA} = \sum_r |y_r - y_{r-1}| \frac{ \left( \alpha_r + \alpha_{r-1} \right)}{2}\)

  • \(\mathrm{AU}_{YA} = \sum_r |x_r - x_{r-1}| \frac{ \left( \alpha_r + \alpha_{r-1} \right)}{2}\)

3. Some extreme examples

4. AUROC and dAUp performance

The measure is also more conservative with trending signals…

and long queues.

***

5. Conclusions

  • The AUC does not have technical problems when dealing with “weird” signals,
  • but it can give missleading and confusing conclusions.
  • We propose an alternative measure that
    1. gives similar result in standard cases
    2. is more robust and conservative when there are short periods of extreme volatility or other anomalies.