lir.datasets package

Submodules

lir.datasets.alcohol_breath_analyser module

class lir.datasets.alcohol_breath_analyser.AlcoholBreathAnalyser(ill_calibrated: bool = False)[source]

Bases: DataProvider

Alcohol Breath Analyser example class.

Example from paper:

Peter Vergeer, Andrew van Es, Arent de Jongh, Ivo Alberink and Reinoud Stoel, Numerical likelihood ratios outputted by LR systems are often based on extrapolation: When to stop extrapolating? In: Science and Justice 56 (2016) 482–491.

Parameters:

ill_calibrated (bool) – Whether to load the intentionally ill-calibrated variant of the dataset.

get_instances() → LLRData[source]

Provide LLR data for example system.

Returns:

Likelihood-ratio data produced by applying the LR system.

Return type:

LLRData

lir.datasets.feature_data_csv module

class lir.datasets.feature_data_csv.DataField(field_name: str, column_names: list[str], validate_cell: ~collections.abc.Callable[[str], ~typing.Any] = <class 'str'>)[source]

Bases: object

A data field for parsing a CSV file into an InstanceData object.

column_names: list[str]

The associated column names in the CSV file.

field_name: str

The attribute name in the InstanceData object.

parse_from_row(row: dict[str, str]) → Any[source]

Parse a row into a field value.

Parameters:

row (dict[str, str]) – A row from the CSV file.

Returns:

The parsed value, either a single value (int, float, str), or a list of values.

Return type:

Any

validate_cell

A validation function for parsing column values.

alias of str

class lir.datasets.feature_data_csv.ExtraField(field_name: str, column_name: str, validate_cell: Callable[[str], Any])[source]

Bases: DataField

Extra field for CSV parsing.

property column_name: str

Return column name.

class lir.datasets.feature_data_csv.FeatureDataCsvParser(file: Callable[[], IO] | str | Path, /, *, source_id_column: str | list[str] | None = None, hypothesis_column: str | None = None, label_column: str | None = None, feature_columns: str | list[str] | None = None, instance_id_column: str | None = None, role_assignment_column: str | None = None, fold_assignment_column: str | None = None, extra_fields: list[ExtraField] | None = None, ignore_columns: list[str] | None = None, head: int | None = None, message_prefix: str | None = None, continue_on_error: bool = False)[source]

Bases: DataProvider

Parse a CSV file into an InstanceData object.

The CSV contents can be a URL, a file path, or a function with no arguments that returns a data stream.

Parameters:

  • file (Callable[[], IO] | str | Path) – Reference to the contents of a CSV file.

  • source_id_column (str | list[str] | None) – Column name(s) containing source identifiers (each source has a unique string identifier).

  • hypothesis_column (str | None) – Column name containing hypothesis labels (value 0 for H2 or 1 for H2).

  • label_column (str | None) – Deprecated alias for hypothesis_column.

  • feature_columns (str | list[str] | None) – Column names containing numerical feature values. If not specified, all columns not otherwise designated are interpreted as feature columns.

  • instance_id_column (str | None) – Column name containing instance identifiers.

  • role_assignment_column (str | None) – Column name containing predefined roles (value ‘train’ for training or ‘test’ for test).

  • fold_assignment_column (str | None) – Column name containing predefined fold assignments (each fold has a unique string identifier).

  • extra_fields (list[ExtraField] | None) – Optional extra fields to parse from each row.

  • ignore_columns (list[str] | None) – Column names ignored when extracting features. This attribute is ignored if feature_columns is available.

  • head (int | None) – Maximum number of rows to read from the source.

  • message_prefix (str | None) – Prefix added to parser log and error messages.

  • continue_on_error (bool) – If True, a row will be dropped if a parse error occurs. Otherwise, parsing will be aborted.

Examples

Assume a CSV file containing two features and source identifiers:

source_id,feature1,feature2,feature3,name_of_an_irrelevant_column
0,1,10,1,sherlock
0,1,11,1,holmes
1,20,30,1,irene
1,18,32,3,adler
2,5,10,8,professor
2,1,11,8,moriarty

This file can be parsed using the following YAML configuration:

data:
  provider: parse_features_from_csv_file
    path: path/to/file.csv
    source_id_column: source_id
    ignore_columns:
      - name_of_an_irrelevant_column

data:
  provider: parse_features_from_csv_url
    url: https://raw.githubusercontent.com/NetherlandsForensicInstitute/elemental_composition_glass/refs/heads/main/training.csv
    source_id_column: Item
    ignore_columns:
      - id
      - Piece
data_fields: list[DataField]
get_instances() → FeatureData[source]

Retrieve FeatureData instances.

Returns:

FeatureData object parsed from the source.

Return type:

FeatureData

ignore_columns: list[str]
class lir.datasets.feature_data_csv.ImplicitFeaturesField[source]

Bases: DataField

A features field for parsing a CSV file into an InstanceData object.

This field does not require feature columns to be specified explicitly, but assumes all columns not otherwise assigned to hold feature values.

parse_from_row(row: dict[str, str]) → list[float][source]

Parse a row into feature values.

Parameters:

row (dict[str, str]) – A row from the CSV file.

Returns:

The feature values.

Return type:

list[float]

exception lir.datasets.feature_data_csv.ParseError[source]

Bases: ValueError

Exception to be raised on parse errors.

This happens when an input file is malformatted or contains invalid input.

lir.datasets.glass module

class lir.datasets.glass.GlassData(cache_dir: Path | str | None = None)[source]

Bases: DataProvider

LA-ICP-MS measurements of elemental concentration from floatglass.

The measurements are from reference glass from casework, collected in the past 10 years or so. For more info on the DataProvider, see: https://github.com/NetherlandsForensicInstitute/elemental_composition_glass

This data provider has a pre-defined train/test split, with a training set of three instances per source, and a test set of five instances per source.

If cache_dir is not None, data are retrieved from the web as needed and stored locally for later use. The class requests_cache.CachedSesson from the requests library handles caching.

Parameters:

cache_dir (Path | str | None) – Cache directory used for storing downloaded dataset files.

get_instances() → FeatureData[source]

Return data with pre-defined assignments of training data and test data.

The training data is read from training.csv and has three instances (replicates) per source. The test data is read from duplo.csv and triplo.csv and has a total of five instances per source.

The features are elemental concentrations on a log_10 basis, and normalized to Si. The elements are: K39, Ti49, Mn55, Rb85, Sr88, Zr90, Ba137, La139, Ce140, Pb208

The source_ids are unique identifiers of a glass particle. Each particle is from a different reference window. An instance is a replicate measurement on a glass particle. Source ids are prefixed with the role assignment, e.g. ‘test-123’ and ‘train-123’. The ids ‘test-123’ and ‘train-123’ refer to different glass particles (and therefore different reference windows).

The instance_ids values of an instance are a concatenation of the filename and a row number, e.g. “training.csv:22”.

The data are returned as a FeatureData object with the following properties: - features: an (n, 10) array of feature values - source_ids: a 1d array of source ids (str) - instance_ids: a 1d array of unique instance ids (str) - role_assignments: a 1d array of role assignments (values “train” or “test”)

Returns:

FeatureData object parsed from the source.

Return type:

FeatureData

lir.datasets.synthesized_normal_binary module

class lir.datasets.synthesized_normal_binary.SynthesizedNormalBinaryData(h1_params: SynthesizedNormalData, h2_params: SynthesizedNormalData, seed: int | None = None)[source]

Bases: DataProvider

Implementation of a data source generating normally distributed binary class data.

Parameters:

  • h1_params (SynthesizedNormalData) – Distribution parameters used to sample class-1 data.

  • h2_params (SynthesizedNormalData) – Distribution parameters used to sample class-2 data.

  • seed (int | None) – Random seed controlling stochastic behaviour for reproducible results.

get_instances() → FeatureData[source]

Return instances with randomly synthesized data and binary labels.

The features are drawn from a normal distribution, as configured.

Returns:

FeatureData object parsed from the source.

Return type:

FeatureData

class lir.datasets.synthesized_normal_binary.SynthesizedNormalData(mean: float, std: float, size: int | tuple[int, int])[source]

Bases: object

Representation of normally distributed data, leveraging a number generator.

The generated data can be used to generate normally distributed data and is useful for debugging purposes or gaining insight in the effect of varying parts within the LR system pipeline.

Parameters:

  • mean (float) – Mean value of the generated normal distribution.

  • std (float) – Standard deviation of the generated normal distribution.

  • size (int | tuple[int, int]) – Number of samples to generate.

get(rng: Generator) → ndarray[source]

Draw random samples from a normally distributed data set.

Parameters:

rng (numpy.random.Generator) – Random number generator used for sampling.

Returns:

Randomly sampled values for this distribution configuration.

Return type:

np.ndarray

lir.datasets.synthesized_normal_multiclass module

class lir.datasets.synthesized_normal_multiclass.SynthesizedDimension(population_mean: float, population_std: float, sources_std: float)[source]

Bases: NamedTuple

Representation of a data distribution.

population_mean: float

Alias for field number 0

population_std: float

Alias for field number 1

sources_std: float

Alias for field number 2

class lir.datasets.synthesized_normal_multiclass.SynthesizedNormalMulticlassData(dimensions: list[SynthesizedDimension], population_size: int, sources_size: int, seed: int | None)[source]

Bases: DataProvider

Implementation of a data source generating normally distributed multiclass data.

Parameters:

  • dimensions (list[SynthesizedDimension]) – Number of feature dimensions to include in the header.

  • population_size (int) – Number of sources to sample in the synthetic population.

  • sources_size (int) – Number of source groups represented in the dataset.

  • seed (int | None) – Random seed controlling stochastic behaviour for reproducible results.

get_instances() → FeatureData[source]

Return instances with randomly synthesized data and multi-class labels.

The features are drawn from a normal distribution, as configured.

Returns:

FeatureData object parsed from the source.

Return type:

FeatureData