AnnotMixClassifier#

class skactiveml.classifier.multiannotator.AnnotMixClassifier(clf_module, alpha=0.5, sample_embed_dim=0, annotator_embed_dim=16, hidden_dim=None, n_hidden_layers=0, hidden_dropout=0.1, eta=0.9, n_annotators=None, neural_net_param_dict=None, sample_dtype=<class 'numpy.float32'>, classes=None, cost_matrix=None, missing_label=nan, random_state=None)[source]#

Bases: _SkorchMultiAnnotatorClassifier

Annot-Mix

Annot-Mix [1] trains a multi-annotator classifier using an extension of MixUp [2]. The main idea is to apply MixUp not only to samples and class labels, but to sample–annotator pairs: it convexly combines inputs and their annotator-specific noisy labels and trains a one-stage model that jointly estimates the true label distribution and each annotator’s reliability. In this way, Annot-Mix can handle multiple, potentially conflicting labels per sample while using MixUp-style regularization to become more robust to label noise.

Parameters:

clf_modulenn.Module or nn.Module.__class__: A PyTorch module as classification model outputting logits for samples as input. In general, the uninstantiated class should be passed, although instantiated modules will also work. The forward module must return logits as first element and optional sample embeddings as second element. If no sample embeddings are returned, the implementation uses the original samples.
alphafloat, default=0.5: MixUp concentration parameter. The mix coefficient lambda is drawn from Beta(alpha, alpha). Use alpha=0 to disable MixUp.
annotator_embed_dimint, default=16: Dimensionality of the annotator embedding used to model annotator-specific behavior.
sample_embed_dimint, default=0: Dimensionality of an optional learnable sample-embedding used to model sample-specific behavior of each annotator. If sample_embed_dim=0, the annotator performances are only modeled as class-specific.
hidden_dimint or None, default=None: Hidden size of the fusion multi-layer perceptron that propagates sample and annotator representations. If None, a sensible default is used, which depends on the other input parameters. Note that this parameter has no effect for n_hidden_layers=0.
n_hidden_layersint, default=0: Number of hidden layers in the fusion multi-layer perceptron.
hidden_dropoutfloat, default=0.1: Dropout probability applied in the fusion multi-layer perceptron. Note that this parameter has no effect for n_hidden_layers=0.
etafloat in (0, 1), default=0.9: Prior annotator performance, i.e., the probability of obtaining a correct annotation from an arbitrary annotator for an arbitrary sample of an arbitrary class.
n_annotatorsint, default=None: Number of annotators. If n_annotators=None, the number of annotators is inferred by the shape of y during training.
neural_net_param_dictdict, default=None: Additional arguments for skorch.net.NeuralNet. If neural_net_param_dict is None, no extra arguments are added. module, criterion, predict_nonlinearity, and train_split are not allowed in this dictionary.
sample_dtypestr or type, default=np.float32: Dtype to which input samples are cast inside the estimator. If set to None, the input dtype is preserved.
classesarray-like of shape (n_classes,), default=None: Holds the label for each class. If None, the classes are determined during the fit.
missing_labelscalar or string or np.nan or None, default=np.nan: Value to represent a missing label.
cost_matrixarray-like of shape (n_classes, n_classes), default=None: Cost matrix with cost_matrix[i,j] indicating cost of predicting class classes[j] for a sample of class classes[i]. Can be only set, if classes is not None.
random_stateint or RandomState instance or None, default=None: Determines random number for predict method. Pass an int for reproducible results across multiple method calls.

References

[1]

Herde, M., Lührs, L., Huseljic, D., & Sick, B. (2024). Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension. Eur. Conf. Artif. Intell.

[2]

Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond Empirical Risk Minimization. Int. Conf. Learn. Represent.

Methods

`fit`(X, y, **fit_params)	Initialize and fit the module.
`get_metadata_routing`()	Get metadata routing of this object.
`get_params`([deep])	Get parameters for this estimator.
`initialize`([X, y, enforce_check_X_y])	Initialize the wrapper and (optionally) validate inputs.
`partial_fit`(X, y, **fit_params)	Fit the module without re-initialization.
`predict`(X[, extra_outputs])	Return class predictions for the test samples X.
`predict_proba`(X[, extra_outputs])	Return class probability estimates for the test samples X.
`score`(X, y[, sample_weight])	Return the mean accuracy on the given test data and labels.
`set_fit_request`(*[, sample_weight])	Configure whether metadata should be requested to be passed to the `fit` method.
`set_params`(**params)	Set the parameters of this estimator.
`set_predict_proba_request`(*[, extra_outputs])	Configure whether metadata should be requested to be passed to the `predict_proba` method.
`set_predict_request`(*[, extra_outputs])	Configure whether metadata should be requested to be passed to the `predict` method.
`set_score_request`(*[, sample_weight])	Configure whether metadata should be requested to be passed to the `score` method.

fit(X, y, **fit_params)#

Initialize and fit the module.

If the module was already initialized, by calling fit, the module will be re-initialized (unless warm_start is True).

Parameters:

Xmatrix-like, shape (n_samples, …): Training data set, usually complete, i.e., including the labeled and unlabeled samples
yarray-like of shape (n_samples, n_annotators): It contains the class labels of the training samples, where missing labels are represented via missing_label. Specifically, label y[n, m] refers to the label of sample X[n] from annotator m.
fit_paramsdict-like: Further parameters as input to the ‘fit’ method of the skorch.net.NeuralNet.

Returns:

self: _SkorchMultiAnnotatorClassifier,: _SkorchMultiAnnotatorClassifier object fitted on the training data.

get_metadata_routing()#

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:

routingMetadataRequest: A MetadataRequest encapsulating routing information.

get_params(deep=True)#

Get parameters for this estimator.

Parameters:

deepbool, default=True: If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

paramsdict: Parameter names mapped to their values.

initialize(X=None, y=None, enforce_check_X_y=False)#

Initialize the wrapper and (optionally) validate inputs.

If any data is provided or enforce_check_X_y is True, inputs are validated via _validate_data. A new skorch.NeuralNet is then created and assigned to self.neural_net_.

Parameters:

Xarray-like of shape (n_samples, …), default=None: Input samples for optional validation.
yarray-like of shape (n_samples, …), default=None: Target values for optional validation.
enforce_check_X_ybool, default=False: Whether to validate even if both X and y are None.

Returns:

selfSkorchMixin: Returned when no input data was supplied (both X and y are None).
X_out, y_outtuple of nd.array, optional: Validated X and y as a tuple, returned when enforce_check_X_y=True.

partial_fit(X, y, **fit_params)#

Fit the module without re-initialization.

If the module was already initialized, by calling partial_fit, the module will not be re-initialized again.

Parameters:

Xmatrix-like, shape (n_samples, …): Training data set, usually complete, i.e., including the labeled and unlabeled samples
yarray-like of shape (n_samples, n_annotators): It contains the class labels of the training samples, where missing labels are represented via missing_label. Specifically, label y[n, m] refers to the label of sample X[n] from annotator m.
fit_paramsdict-like: Further parameters as input to the ‘partial_fit’ method of the skorch.net.NeuralNet.

Returns:

self: _SkorchMultiAnnotatorClassifier,: _SkorchMultiAnnotatorClassifier object fitted on the training data.

predict(X, extra_outputs=None)[source]#

Return class predictions for the test samples X.

By default, this method returns only the class predictions y_pred. If extra_outputs is provided, a tuple is returned whose first element is y_pred and whose remaining elements are the requested additional forward outputs, in the order specified by extra_outputs.

Parameters:

Xarray-like of shape (n_samples, …)

Test samples.

extra_outputsNone or str or sequence of str, default=None

Names of additional outputs to return next to y_pred. The names must be a subset of the following keys:

“logits” : Additionally return the class-membership logits L_class for the samples in X.
“embeddings” : Additionally return the learned embeddings X_embed for the samples in X.
“annotator_perf” : additionally return the estimated annotator performance probabilities P_perf for each sample–annotator pair.
“annotator_class” : Additionally return the annotator–class probability estimates P_annot for each sample, class, and annotator.
“annotator_embeddings” : Additionally return the learned embeddings A_embed for the annotators as the next element of the output tuple.

Returns:

y_prednumpy.ndarray of shape (n_samples,)

Class labels of the test samples.

*extrasnumpy.ndarray, optional

Only returned if extra_outputs is not None. In that case, the method returns a tuple whose first element is y_pred and whose remaining elements correspond to the requested forward outputs in the order given by extra_outputs. Potential outputs are:

L_class : np.ndarray of shape (n_samples, n_classes), where L_class[n, c] is the logit for the class classes_[c] of sample X[n].
X_embed : np.ndarray of shape (n_samples, …), where X_embed[n] refers to the learned embedding for sample X[n].
P_perf : np.ndarray of shape (n_samples, n_annotators), where P_perf[n, m] refers to the estimated label correctness probability (performance) of annotator m when labeling sample X[n].
P_annot : np.ndarray of shape (n_samples, n_annotators, n_classes), where P_annot[n, m, c] refers to the probability that annotator m provides the class label c for sample X[n].
A_embed : np.ndarray of shape (n_annotators, annotator_embed_dim), where A_embed[m] refers to the learned embedding for annotator m.

predict_proba(X, extra_outputs=None)[source]#

Return class probability estimates for the test samples X.

By default, this method returns only the class probabilities P. If extra_outputs is provided, a tuple is returned whose first element is P and whose remaining elements are the requested additional forward outputs, in the order specified by extra_outputs.

Parameters:

Xarray-like of shape (n_samples, …)

Test samples.

extra_outputsNone or str or sequence of str, default=None

Names of additional outputs to return next to P. The names must be a subset of the following keys:

“logits” : Additionally return the class-membership logits L_class for the samples in X.
“embeddings” : Additionally return the learned embeddings X_embed for the samples in X.
“annotator_perf” : additionally return the estimated annotator performance probabilities P_perf for each sample–annotator pair.
“annotator_class” : Additionally return the annotator–class probability estimates P_annot for each sample, class, and annotator.
“annotator_embeddings” : Additionally return the learned embeddings A_embed for the annotators as the next element of the output tuple.

Returns:

Pnumpy.ndarray of shape (n_samples, n_classes)

Class probabilities of the test samples. Classes are ordered according to self.classes_.

*extrasnumpy.ndarray, optional

Only returned if extra_outputs is not None. In that case, the method returns a tuple whose first element is P and whose remaining elements correspond to the requested forward outputs in the order given by extra_outputs. Potential outputs are:

L_class : np.ndarray of shape (n_samples, n_classes), where L_class[n, c] is the logit for the class classes_[c] of sample X[n].
X_embed : np.ndarray of shape (n_samples, …), where X_embed[n] refers to the learned embedding for sample X[n].
P_perf : np.ndarray of shape (n_samples, n_annotators), where P_perf[n, m] refers to the estimated label correctness probability (performance) of annotator m when labeling sample X[n].
P_annot : `np.ndarray of shape (n_samples, n_annotators, n_classes), where P_annot[n, m, c] refers to the probability that annotator m provides the class label c for sample X[n].
A_embed : np.ndarray of shape (n_annotators, annotator_embed_dim), where A_embed[m] refers to the learned embedding for annotator m.

score(X, y, sample_weight=None)#

Return the mean accuracy on the given test data and labels.

Parameters:

Xarray-like of shape (n_samples, n_features): Test samples.
yarray-like of shape (n_samples,): True labels for X.
sample_weightarray-like of shape (n_samples,), default=None: Sample weights.

Returns:

scorefloat: Mean accuracy of self.predict(X) regarding y.

set_fit_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → AnnotMixClassifier#

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for sample_weight parameter in fit.

Returns:

selfobject: The updated object.

set_params(**params)#

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:

**paramsdict: Estimator parameters.

Returns:

selfestimator instance: Estimator instance.

set_predict_proba_request(*, extra_outputs: bool | None | str = '$UNCHANGED$') → AnnotMixClassifier#

Configure whether metadata should be requested to be passed to the predict_proba method.

The options for each parameter are:

True: metadata is requested, and passed to predict_proba if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to predict_proba.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

extra_outputsstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for extra_outputs parameter in predict_proba.

Returns:

selfobject: The updated object.

set_predict_request(*, extra_outputs: bool | None | str = '$UNCHANGED$') → AnnotMixClassifier#

Configure whether metadata should be requested to be passed to the predict method.

The options for each parameter are:

True: metadata is requested, and passed to predict if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to predict.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

extra_outputsstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for extra_outputs parameter in predict.

Returns:

selfobject: The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → AnnotMixClassifier#

Configure whether metadata should be requested to be passed to the score method.

The options for each parameter are:

True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to score.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for sample_weight parameter in score.

Returns:

selfobject: The updated object.

AnnotMixClassifier#

This Page