lir.plotting.expected_calibration_error module

Empirical cross-entropy (ECE).

The discrimination and calibration of the LRs reported by some systems can also be measured separately. The empirical cross entropy (ECE) plot is a graphical way of doing this.

ECE is computed as the average of:

  • -P(Hp) * log2(P(Hp | LR_i)) over all LR_i for which Hp is true, and

  • -P(Hd) * log2(P(Hd | LR_i)) over all LR_i for which Hd is true.

References

Ramos, D. Forensic Evidence Evaluation Using Automatic Speaker Recognition Systems. Ph.D. thesis, Universidad Autónoma de Madrid.

Robertson, B., Vignaux, G. A., & Berger, C. (2016). Interpreting Evidence: Evaluating Forensic Science in the Courtroom (2nd ed.), pp. 96–97.

lir.plotting.expected_calibration_error.calculate_ece(lrs: ndarray, y: ndarray, priors: ndarray) → ndarray[source]

Calculate empirical cross-entropy (ECE) of a set of LRs and corresponding ground-truth labels.

An entropy is calculated for each element of priors.

Parameters:

  • lrs (np.ndarray) – Array of likelihood ratios.

  • y (np.ndarray) – Array of ground-truth labels for the LRs (0 for Hd or 1 for Hp), with the same length as lrs.

  • priors (np.ndarray) – Array of prior probabilities for class 1 (values in the range [0, 1]).

Returns:

Array of entropy values with the same length as priors.

Return type:

np.ndarray

lir.plotting.expected_calibration_error.plot_ece(ax: Any, llrdata: LLRData, log_prior_odds_range: tuple[float, float] = (-3, 3), show_pav: bool = True, ylim: str = 'neutral') → None[source]

Generate an ECE plot for a set of LRs and corresponding ground-truth labels.

The x-axis shows the log prior odds of a sample being drawn from class 1. The y-axis shows the expected cost (cross-entropy) for:

  1. A non-informative system (dotted line),

  2. The provided LR values (solid line), and

  3. The LR values after PAV transformation (Pool Adjacent Violators; dashed line).

Parameters:

  • ax (matplotlib.axes.Axes) – Matplotlib axes to plot into.

  • llrdata (LLRData) – LLR data containing LLR values and corresponding labels.

  • log_prior_odds_range (tuple[float, float], optional) – Range of log prior odds shown on the x-axis, given as (min, max).

  • show_pav (bool, optional) – Whether to include the PAV-transformed LRs in the plot.

  • ylim ({"neutral", "zoomed"}, optional) –

    Y-axis scaling mode:

    • "neutral": Lower limit is 0; upper limit is set slightly above the maximum of the non-informative reference.

    • "zoomed": Lower limit is 0; upper limit is set to approximately 10% above the maximum ECE value of the LRs (this may clip part of the non-informative reference line).