Registry reference

Experiment components

The preferred way to set up an experiment is to use a YAML configuration. Use the LR system selection helper to learn how to use the YAML interface.

This page lists the components that may be needed to set up an experiment.

Experiment strategies

Registry section: experiment_strategies

• experiment_strategies.single_run – Prepare Experiment consisting of a single run using configuration values.

• experiment_strategies.grid – Prepare Experiment consisting of multiple runs using configuration values.

• experiment_strategies.optuna – Prepare Experiment for optimizing configuration parameters.

Data strategies

Registry section: data_strategies

• data_strategies.train_test – Split the data into a training set and a test set.

• data_strategies.cross_validation – K-fold cross-validation iterator over successive train/test splits.

• data_strategies.train_test_sources – Split the data into a training set and a test set by their source ids.

• data_strategies.cross_validation_sources – K-fold cross-validation by source id.

• data_strategies.predefined_train_test – Split data into a training set and a test set based on predefined assignments.

• data_strategies.predefined_cross_validation – Split data into cross validation folds based on predefined assignments.

• data_strategies.train_test_pairs – A train/test split policy for paired instances.

Data providers

Registry section: data_providers

• data_providers.synthesized_normal_binary – Implementation of a data source generating normally distributed binary class data.

• data_providers.synthesized_normal_multiclass – Implementation of a data source generating normally distributed multiclass data.

• data_providers.glass – LA-ICP-MS measurements of elemental concentration from floatglass.

• data_providers.parse_features_from_csv_file – Read CSV data from file.

• data_providers.parse_features_from_csv_url – Read CSV data from a URL.

Metrics

Registry section: metric

• metric.cllr – Calculate a log likelihood ratio cost (C_llr) for a series of log likelihood ratios.

• metric.cllr_min – Estimate the discriminative power from a collection of log likelihood ratios.

• metric.cllr_cal – Calculate the difference between the C_llr before and after isotonic calibration.

• metric.llr_lower_bound – Provide corresponding lower bound for provided LLR data.

• metric.llr_upper_bound – Provide corresponding upper bound for provided LLR data.

• metric.devpav – Calculate devPAV for LR data under H1 and H2.

Output

Registry section: output

• output.metrics – Helper class to write aggregated results to CSV file.

• output.pav – Generate a plot of pre-calibrated versus post-calibrated LRs using Pool Adjacent Violators (PAV).

• output.ece – Generate an ECE plot for a set of LRs and corresponding ground-truth labels.

• output.lr_histogram – Plot the 10log LRs.

• output.llr_interval – Plot the LRs on the x-axis, with the relative interval score on the y-axis.

• output.llr_overestimation – Plot LLR-overestimation as a function of the system LLR.

• output.nbe – Generate the visual NBE plot using matplotlib.

• output.invariance_delta_function – Plot Invariance Verification delta functions and LR bounds.

• output.tippett – Plot empirical cumulative distribution functions of same-source and different-sources LRs.

• output.score_to_llr – Aggregation that generates plots by repeatedly calling a plotting function.

• output.score_distribution – Aggregation that generates plots by repeatedly calling a plotting function.

• output.save_model – Write the model to a file.

• output.case_llr – Aggregation that applies a full-data-fitted LR system to case data and stores LLRs as CSV.

• output.by_category – Aggregation method that manages data categorization.

Hyperparameters

Registry section: hyperparameter_types

• hyperparameter_types.categorical – Parse a categorical hyperparameter from configuration.

• hyperparameter_types.cluster – Parse the configuration section of a clustered hyperparameter.

• hyperparameter_types.constant – Parse the configuration section of a constant.

• hyperparameter_types.float – Parse a floating-point hyperparameter from configuration.

• hyperparameter_types.folder – Parse a folder hyperparameter from configuration.

LR system components

You may choose to use either the Python API or a YAML configuration to set up an LR system.

• Use the Practitioner’s Guide to learn how to use the Python API.

• Use the LR system selection helper to learn how to use the YAML interface.

This page lists the components that may be needed to define an LR system.

LR system architecture

Registry section: lrsystem_architecture

• lrsystem_architectures.specific_source – LR system for binary data and a linear pipeline.

• lrsystem_architectures.score_based – Provide a representation of a common source, score-based LR system.

• lrsystem_architectures.two_level – Implement two level model, common-source feature-based LR system architecture.

LR system modules

Registry section: modules

• modules.pipeline – A pipeline of processing modules.

• modules.logging_pipeline – A pipeline that writes debugging output to a CSV file.

• modules.bootstrap – Bootstrap system that estimates confidence intervals around the best estimate of a pipeline.

• modules.standard_scaler – Standardize features by removing the mean and scaling to unit variance.

• modules.probabilities_to_odds – Convert a probability to odds.

• modules.probabilities_to_logodds – Convert probability values to their log odds with base 10.

• modules.element_wise_difference – Calculate the element-wise absolute difference between pairs.

• modules.euclidean_distance – Calculate the Euclidean distance between pairs.

• modules.manhattan_distance – Calculate the Manhattan distance between pairs.

• modules.tee – Parse configuration for allowing multiple tasks for given input.

• modules.csv_writer – Implementation of a transformation step in a scikit-learn Pipeline that writes to CSV.

• modules.save_features – Transformer to save the features of the instances in a new field.

• modules.validate_feature_data_type – Module that validates the data types of the features in the instances.

• modules.identity – Represent the Identity function of a transformer.

• modules.logistic_regression – Logistic Regression (aka logit, MaxEnt) classifier.

• modules.svm – C-Support Vector Classification.

• modules.kde – Calculate LR from a score, belonging to one of two distributions using KDE.

• modules.isotonic_calibrator – Calculate LR from a score belonging to one of two distributions using isotonic regression.

• modules.logistic_calibrator – Calculate LR from a score, belonging to one of two distributions using logistic regression.

• modules.mcmc – Use Markov Chain Monte Carlo simulations to fit a statistical distribution for each of the two hypotheses.

• modules.static_bounder – Bound LLRs to constant values.

• modules.elub_bounder – Calculate the Empirical Upper and Lower Bounds for a given LR system.

• modules.iv_bounder – Calculate Invariance Verification bounds for a given LR system.

• modules.n_source_bounder – Bound LLRs based on the number of sources.

Pairing methods

Registry section: pairing

• pairing.instance_pairs – Construct pairs from a set of instances.

• pairing.source_pairs – Construct pairs of sources (i.e. classes) from an array of instances.