IntervalEstimationAnnotModel

class skactiveml.pool.multiannotator.IntervalEstimationAnnotModel(classes=None, missing_label=nan, alpha=0.05, mode='upper', random_state=None)

Bases: BaseEstimator

IELearning

This annotator model relies on ‘Interval Estimation Learning’ (IELearning) for estimating the annotation performances, i.e., labeling accuracies, of multiple annotators [1]. Therefore, it computes the mean accuracy and the lower as well as the upper bound of the labeling accuracy per annotator. (Weighted) majority vote is used as estimated ground truth.

Parameters:

classesarray-like of shape (n_classes,), default=None

Holds the label for each class.

missing_labelscalar or string or np.nan or None, default=np.nan

Value to represent a missing label.

alphafloat, interval=(0, 1), default=0.05

Half of the confidence level for student’s t-distribution.

mode‘lower’ or ‘mean’ or ‘upper’, default=’upper’

Mode of the estimated annotation performance.

random_stateNone or int or numpy.random.RandomState, default=None

The random state used for deciding on majority vote labels in case of ties.

Attributes:

n_annotators_: int

Number of annotators.

A_perf_ndarray, shape (n_annotators, 3)

Estimated annotation performances (i.e., labeling accuracies), where A_cand[i, 0] indicates the lower bound, A_cand[i, 1] indicates the mean, and A_cand[i, 2] indicates the upper bound of the estimation labeling accuracy.

References

[1]

P. Donmez, J. G. Carbonell, and J. Schneider. Efficiently Learning the Accuracy of Labeling Sources for Selective Sampling. In ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., pages 259–268, 2009.

Methods

fit(X, y[, sample_weight])	Fit annotator model for given samples.
get_metadata_routing()	Get metadata routing of this object.
get_params([deep])	Get parameters for this estimator.
predict_annotator_perf(X)	Calculates the probability that an annotator provides the true label for a given sample.
set_fit_request(*[, sample_weight])	Configure whether metadata should be requested to be passed to the fit method.
set_params(**params)	Set the parameters of this estimator.

fit(X, y, sample_weight=None)

Fit annotator model for given samples.

Parameters:

Xarray-like of shape (n_samples, n_features)

Test samples.

yarray-like of shape (n_samples, n_annotators)

Class labels of annotators.

sample_weightarray-like of shape (n_samples, n_annotators), default=None

Sample weight for each label and annotator.

Returns:

selfIntervalEstimationAnnotModel object

The fitted annotator model.

get_metadata_routing()

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:

routingMetadataRequest

A MetadataRequest encapsulating routing information.

get_params(deep=True)

Get parameters for this estimator.

Parameters:

deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

paramsdict

Parameter names mapped to their values.

predict_annotator_perf(X)

Calculates the probability that an annotator provides the true label for a given sample.

Parameters:

Xarray-like of shape (n_samples, n_features)

Test samples.

Returns:

P_annotnumpy.ndarray of shape (n_samples, n_annotators)

P_annot[i,l] is the probability, that annotator l provides the correct class label for sample X[i].

set_fit_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → IntervalEstimationAnnotModel

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to fit.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns:

selfobject

The updated object.

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:

**paramsdict

Estimator parameters.

Returns:

selfestimator instance

Estimator instance.