Registry reference

Experiment components

The preferred way to set up an experiment is to use a YAML configuration. Use the LR system selection helper to learn how to use the YAML interface.

This page lists the components that may be needed to set up an experiment.

Experiment strategies

Registry section: experiment_strategies

• experiment_strategies.single_run Prepare Experiment consisting of a single run using configuration values.

• experiment_strategies.grid Prepare Experiment consisting of multiple runs using configuration values.

• experiment_strategies.optuna Prepare Experiment for optimizing configuration parameters.

Data strategies

Registry section: data_strategies

• data_strategies.train_test Split the data into a training set and a test set.

• data_strategies.cross_validation K-fold cross-validation iterator over successive train/test splits.

• data_strategies.train_test_sources Split the data into a training set and a test set by their source ids.

• data_strategies.cross_validation_sources K-fold cross-validation by source id.

• data_strategies.predefined_train_test Split data into a training set and a test set based on predefined assignments.

• data_strategies.predefined_cross_validation Split data into cross validation folds based on predefined assignments.

• data_strategies.train_test_pairs A train/test split policy for paired instances.

Data providers

Registry section: data_providers

• data_providers.synthesized_normal_binary Implementation of a data source generating normally distributed binary class data.

• data_providers.synthesized_normal_multiclass Implementation of a data source generating normally distributed multiclass data.

• data_providers.glass LA-ICP-MS measurements of elemental concentration from floatglass.

• data_providers.parse_features_from_csv_file Read CSV data from file.

• data_providers.parse_features_from_csv_url Read CSV data from a URL.

Metrics

Registry section: metric

• metric.cllr Calculate a log likelihood ratio cost (C_llr) for a series of log likelihood ratios.

• metric.cllr_min Estimate the discriminative power from a collection of log likelihood ratios.

• metric.cllr_cal Calculate the difference between the C_llr before and after isotonic calibration.

• metric.llr_lower_bound Provide corresponding lower bound for provided LLR data.

• metric.llr_upper_bound Provide corresponding upper bound for provided LLR data.

• metric.devpav Calculate devPAV for LR data under H1 and H2.

Output

Registry section: output

• output.metrics Helper class to write aggregated results to CSV file.

• output.pav Generate a plot of pre-calibrated versus post-calibrated LRs using Pool Adjacent Violators (PAV).

• output.ece Generate an ECE plot for a set of LRs and corresponding ground-truth labels.

• output.lr_histogram Plot the 10log LRs.

• output.llr_interval Plot the LRs on the x-axis, with the relative interval score on the y-axis.

• output.llr_overestimation Plot LLR-overestimation as a function of the system LLR.

• output.nbe Generate the visual NBE plot using matplotlib.

• output.invariance_delta_function Plot Invariance Verification delta functions and LR bounds.

• output.tippett Plot empirical cumulative distribution functions of same-source and different-sources LRs.

• output.score_to_llr Aggregation that generates plots by repeatedly calling a plotting function.

• output.score_distribution Aggregation that generates plots by repeatedly calling a plotting function.

• output.save_model Write the model to a file.

• output.case_llr Aggregation that applies a full-data-fitted LR system to case data and stores LLRs as CSV.

• output.by_category Aggregation method that manages data categorization.

Hyperparameters

Registry section: hyperparameter_types

• hyperparameter_types.categorical Parse a categorical hyperparameter from configuration.

• hyperparameter_types.cluster Parse the configuration section of a clustered hyperparameter.

• hyperparameter_types.constant Parse the configuration section of a constant.

• hyperparameter_types.float Parse a floating-point hyperparameter from configuration.

• hyperparameter_types.folder Parse a folder hyperparameter from configuration.

LR system components

You may choose to use either the Python API or a YAML configuration to set up an LR system.

• Use the Practitioner’s Guide to learn how to use the Python API.

• Use the LR system selection helper to learn how to use the YAML interface.

This page lists the components that may be needed to define an LR system.

LR system architecture

Registry section: lrsystem_architecture

• lrsystem_architectures.specific_source LR system for binary data and a linear pipeline.

• lrsystem_architectures.score_based Provide a representation of a common source, score-based LR system.

• lrsystem_architectures.two_level Implement two level model, common-source feature-based LR system architecture.

LR system modules

Registry section: modules

• modules.pipeline A pipeline of processing modules.

• modules.logging_pipeline A pipeline that writes debugging output to a CSV file.

• modules.bootstrap Bootstrap system that estimates confidence intervals around the best estimate of a pipeline.

• modules.standard_scaler Standardize features by removing the mean and scaling to unit variance.

• modules.probabilities_to_odds Convert a probability to odds.

• modules.probabilities_to_logodds Convert probability values to their log odds with base 10.

• modules.element_wise_difference Calculate the element-wise absolute difference between pairs.

• modules.euclidean_distance Calculate the Euclidean distance between pairs.

• modules.manhattan_distance Calculate the Manhattan distance between pairs.

• modules.tee Parse configuration for allowing multiple tasks for given input.

• modules.csv_writer Implementation of a transformation step in a scikit-learn Pipeline that writes to CSV.

• modules.save_features Transformer to save the features of the instances in a new field.

• modules.validate_feature_data_type Module that validates the data types of the features in the instances.

• modules.identity Represent the Identity function of a transformer.

• modules.logistic_regression Logistic Regression (aka logit, MaxEnt) classifier.

• modules.svm C-Support Vector Classification.

• modules.kde Calculate LR from a score, belonging to one of two distributions using KDE.

• modules.isotonic_calibrator Calculate LR from a score belonging to one of two distributions using isotonic regression.

• modules.logistic_calibrator Calculate LR from a score, belonging to one of two distributions using logistic regression.

• modules.mcmc Use Markov Chain Monte Carlo simulations to fit a statistical distribution for each of the two hypotheses.

• modules.static_bounder Bound LLRs to constant values.

• modules.elub_bounder Calculate the Empirical Upper and Lower Bounds for a given LR system.

• modules.iv_bounder Calculate Invariance Verification bounds for a given LR system.

• modules.n_source_bounder Bound LLRs based on the number of sources.

Pairing methods

Registry section: pairing

• pairing.instance_pairs Construct pairs from a set of instances.

• pairing.source_pairs Construct pairs of sources (i.e. classes) from an array of instances.