lir.experiments package

class lir.experiments.Experiment(output_path: Path)[source]

Bases: ABC

Representation of an experiment to evaluate LR systems.

Parameters:

output_path (Path) – Path where generated outputs are written.

abstractmethod run() → None[source]

Execute the experiment.

class lir.experiments.OptunaExperiment(data_config: ConfigValue, outputs: Sequence[Aggregation], output_path: Path, baseline_config: ConfigValue, lrsystem_parameters: list[Hyperparameter], n_trials: int, metric_function: Callable[[LLRData], float])[source]

Bases: Experiment

An optimization strategy that uses Optuna for choosing parameter values.

This strategy sequentially runs slight variations of an LR system by changing its hyperparameters. After each run, the output is evaluated using a metric_function, and the next set of hyperparameter values is chosen. The experiment stops after n_trials runs are executed.

Parameters:

  • data_config (ConfigValue) – Data configuration used to construct datasets for runs.

  • outputs (Sequence[Aggregation]) – Output aggregation definitions executed after each run.

  • output_path (Path) – Path where generated outputs are written.

  • baseline_config (ConfigValue) – Baseline configuration to be tuned during optimisation.

  • lrsystem_parameters (list[Hyperparameter]) – LR system parameters varied during optimisation.

  • n_trials (int) – Number of optimisation trials to execute.

  • metric_function (Callable[[LLRData], float]) – Value passed via metric_function.

run() → None[source]

Execute the experiment.

This method ensures that all outputs are properly closed after the experiment run.

class lir.experiments.PredefinedExperiment(data_configs: list[DataConfig], outputs: Sequence[Aggregation], output_path: Path, lrsystem_configs: list[LRSystemConfig], enable_parallelization: bool = False)[source]

Bases: Experiment

Experiment strategy that runs a pre-defined set of LR systems on a pre-defined set of data setups.

To set up a single run experiment in a YAML configuration:

experiments:
  - strategy: single_run
    name: my experiment
    data: *my_data_setup
    lrsystem: *my_lrsystem
    output: *my_aggregations

Multiple runs can be defined using the grid strategy, with additional configuration options:

  • use the hyperparameters field to configure which hyperparameters can be varied;

  • use the dataparameters field to configure which dataparameters can be varied;

  • set the enable_parallelization field to True to enable parallelization.

For more guidance and working examples, see: Setting up an experiment.

Parameters:

  • data_configs (list[DataConfig]) – Data configurations evaluated by this experiment.

  • outputs (Sequence[Aggregation]) – Output aggregation definitions executed after each run.

  • output_path (Path) – Path where generated outputs are written.

  • lrsystem_configs (list[LRSystemConfig]) – LR-system configurations evaluated by this experiment.

  • enable_parallelization (bool) – Whether to run the LR systems in parallel.

run() → None[source]

Execute the experiment.

This method ensures that all outputs are properly closed after the experiment run.

Submodules

lir.experiments.execution module

class lir.experiments.execution.DataConfig(spec: ConfigValue, params: dict[str, Any], experiment_output_dir: Path, run_output_dir: Path | None = None)[source]

Bases: ParameterizedConfig

Data configuration object.

property data_setup: DataSetup

Parse the data configuration.

This is done here to ensure that data parsing is only done once per data configuration, even when multiple LR systems are being evaluated on the same data setup.

Returns:

A tuple of a data provider and a data strategy.

Return type:

tuple[DataProvider, DataStrategy]

property filter: Transformer

Return a data filter.

Returns:

A data transformer object.

Return type:

Transformer

property provider: DataProvider

Return a data provider.

Returns:

A data provider object.

Return type:

DataProvider

property splits: Iterable[tuple[InstanceData, InstanceData]]

Convert the split_data iterable to a list to allow multiple iterations over the splits.

E.g. one iteration for validation and one for case LLR generation.

Returns:

An iterable of training/test set pairs.

Return type:

Iterable[tuple[InstanceData, InstanceData]]

property splitter: DataStrategy

Return a data splitter.

Returns:

A data splitter object.

Return type:

DataStrategy

class lir.experiments.execution.LRSystemConfig(spec: ConfigValue, params: dict[str, Any], experiment_output_dir: Path, run_output_dir: Path | None = None)[source]

Bases: ParameterizedConfig

LR system configuration object.

property lrsystem: LRSystem

Return the materialized LR system as defined in configuration.

Returns:

An LR system object.

Return type:

LRSystem

class lir.experiments.execution.ParameterizedConfig(spec: ConfigValue, params: dict[str, Any], experiment_output_dir: Path, run_output_dir: Path | None = None)[source]

Bases: NamedTuple

Base class for LR system or data configurations.

A configuration of an LR system or data setup is a dictionary, stored in the spec attribute. Additionally, there can be hyperparameters that are already incorporated in the configuration. The hyperparameters describe how this configuration is different from other configurations.

The configuration is extended by the subclass to lazily materialize the configuration on demand.

Objects of this class are pickleable. When pickled, the materialization is dropped and will have to be recreated when needed.

property desc: str

Generate a description of this configuration from the parameter values.

Returns:

A description of this configuration.

Return type:

str

experiment_output_dir: Path

Path to the directory where results of the experiment may be written. This directory is shared among all runs of an experiment.

params: dict[str, Any]

The parameters that describe the configuration.

run_output_dir: Path | None

Path to the directory where results of the run may be written, this is a subdirectory of experiment_output_dir. May be None if the configuration is not yet assigned to a run.

spec: ConfigValue

The configuration of an LR system or data setup for a run.

lir.experiments.execution.parallellize_runs(output_base_dir: Path, lrsystem_configs: list[LRSystemConfig], data_configs: list[DataConfig]) → Iterable[AggregationData][source]

Run LR systems in parallel.

This method has exactly the same effect as run_multiple(), but uses multiprocessing to do runs in parallel. It selects a parallelization strategy to distribute the runs over workers.

Issues:

  • this may lead to repetitive loading of data, which may take additional (costly) I/O operations

  • in some cases (notably, when bootstrapping) the multiprocessing.imap_unsorted operation may produce a “leaked semaphore” warning

  • logging in workers is disabled

Parameters:

  • output_base_dir (Path) – The base directory where the results may be written.

  • lrsystem_configs (list[LRSystemConfig]) – A list of LR system configuraitons.

  • data_configs (list[DataConfig]) – A list of dataset configurations.

Returns:

A list of results for all runs.

Return type:

list[AggregationData]

lir.experiments.execution.run_lrsystem(experiment_output_dir: Path, lrsystem_config: LRSystemConfig, data_config: DataConfig, skip_full_lrsystem: bool = False, run_name: str | None = None) → AggregationData[source]

Run experiment on a single LR system configuration using the provided data.

The LR system is fitted using the training subset data and subsequently used to determine LLRs for the test subset data. The results are stored in a temporary list which contains the determined data of each test / train split.

The collected results are combined and passed to the configured outputs aggregations, which may write metrics and visualizations to the output_path directory. The combined LLR data is returned.

Next to this, the configuration of both the data and LR system are stored in the output directory for future reference.

Parameters:

  • experiment_output_dir (Path) – The base directory of the path where results may be written.

  • lrsystem_config (LRSystemConfig) – LR-system configuration for a single run.

  • data_config (DataConfig) – Data configuration used to construct datasets for runs.

  • skip_full_lrsystem (bool) – If True, the full LR system will not be trained.

  • run_name (str | None) – The name of the run (optional). If None, the name will be derived from parameter values.

Returns:

Likelihood-ratio data produced by applying the LR system.

Return type:

LLRData

lir.experiments.execution.run_multiple(output_base_dir: Path, lrsystem_configs: list[LRSystemConfig], data_configs: list[DataConfig]) → Iterator[AggregationData][source]

Run LR systems sequentially.

Consider using parallellize_runs() to speed up processing by doing runs in parallel.

Parameters:

  • output_base_dir (Path) – The base directory where the results may be written.

  • lrsystem_configs (list[LRSystemConfig]) – A list of LR system configuraitons.

  • data_configs (list[DataConfig]) – A list of dataset configurations.

Returns:

A list of results for all runs.

Return type:

Iterator[AggregationData]

lir.experiments.execution.run_multiple_lrsystems(output_base_dir: Path, lrsystem_configs: list[LRSystemConfig], data_config: DataConfig) → list[AggregationData][source]

Run multiple LR systems for a single data configuration.

Parameters:

  • output_base_dir (Path) – The base directory where the results may be written.

  • lrsystem_configs (list[LRSystemConfig]) – A list of LR system configuraitons.

  • data_config (DataConfig) – Data configuration used to construct the dataset.

Returns:

A list of results for all runs.

Return type:

list[AggregationData]

lir.experiments.optuna_experiment module

class lir.experiments.optuna_experiment.OptunaExperiment(data_config: ConfigValue, outputs: Sequence[Aggregation], output_path: Path, baseline_config: ConfigValue, lrsystem_parameters: list[Hyperparameter], n_trials: int, metric_function: Callable[[LLRData], float])[source]

Bases: Experiment

An optimization strategy that uses Optuna for choosing parameter values.

This strategy sequentially runs slight variations of an LR system by changing its hyperparameters. After each run, the output is evaluated using a metric_function, and the next set of hyperparameter values is chosen. The experiment stops after n_trials runs are executed.

Parameters:

  • data_config (ConfigValue) – Data configuration used to construct datasets for runs.

  • outputs (Sequence[Aggregation]) – Output aggregation definitions executed after each run.

  • output_path (Path) – Path where generated outputs are written.

  • baseline_config (ConfigValue) – Baseline configuration to be tuned during optimisation.

  • lrsystem_parameters (list[Hyperparameter]) – LR system parameters varied during optimisation.

  • n_trials (int) – Number of optimisation trials to execute.

  • metric_function (Callable[[LLRData], float]) – Value passed via metric_function.

run() → None[source]

Execute the experiment.

This method ensures that all outputs are properly closed after the experiment run.

lir.experiments.predefined_experiment module

class lir.experiments.predefined_experiment.PredefinedExperiment(data_configs: list[DataConfig], outputs: Sequence[Aggregation], output_path: Path, lrsystem_configs: list[LRSystemConfig], enable_parallelization: bool = False)[source]

Bases: Experiment

Experiment strategy that runs a pre-defined set of LR systems on a pre-defined set of data setups.

To set up a single run experiment in a YAML configuration:

experiments:
  - strategy: single_run
    name: my experiment
    data: *my_data_setup
    lrsystem: *my_lrsystem
    output: *my_aggregations

Multiple runs can be defined using the grid strategy, with additional configuration options:

  • use the hyperparameters field to configure which hyperparameters can be varied;

  • use the dataparameters field to configure which dataparameters can be varied;

  • set the enable_parallelization field to True to enable parallelization.

For more guidance and working examples, see: Setting up an experiment.

Parameters:

  • data_configs (list[DataConfig]) – Data configurations evaluated by this experiment.

  • outputs (Sequence[Aggregation]) – Output aggregation definitions executed after each run.

  • output_path (Path) – Path where generated outputs are written.

  • lrsystem_configs (list[LRSystemConfig]) – LR-system configurations evaluated by this experiment.

  • enable_parallelization (bool) – Whether to run the LR systems in parallel.

run() → None[source]

Execute the experiment.

This method ensures that all outputs are properly closed after the experiment run.

lir.experiments.train_only module

class lir.experiments.train_only.TrainOnlyExperiment(output_dir: Path, data_provider: DataProvider, lrsystem: LRSystem)[source]

Bases: Experiment

Experiment strategy that runs a training session only.

This experiment trains an LR system on a dataset and writes the fitted model to a file named lrsystem.pkl.

To set up a training only experiment in a YAML configuration:

experiments:
  - strategy: train_only
    name: my experiment
    data_provider: *my_data_provider
    lrsystem: *my_lrsystem
Parameters:

  • output_dir (Path) – Path where generated outputs are written.

  • data_provider (DataProvider) – Data provider that provides the data for training the LR system.

  • lrsystem (LRSystem) – The LR system to train.

run() → None[source]

Execute the experiment.