CrowdLayerClassifier

class skactiveml.classifier.multiannotator.CrowdLayerClassifier(clf_module, n_annotators=None, neural_net_param_dict=None, sample_dtype=<class 'numpy.float32'>, classes=None, cost_matrix=None, missing_label=nan, random_state=None)

Bases: _SkorchMultiAnnotatorClassifier

Crowd Layer

Crowd Layer [1] is a layer added at the end of a classifying neural network and allows us to train deep neural networks end-to-end, directly from the noisy labels of multiple annotators, using only backpropagation. The main idea is to insert an annotator-specific transformation on top of a shared latent prediction: for each annotator, the layer models how their noisy labels are generated from the underlying class probabilities (e.g., via a confusion-matrix-like mapping). By learning these annotator-specific mappings jointly with the base network, Crowd Layer can separate systematic annotator biases from the true label signal and thus leverage multiple noisy labels per sample during training.

Parameters:

clf_modulenn.Module or nn.Module.__class__

A PyTorch module as classification model outputting logits for samples as input. In general, the uninstantiated class should be passed, although instantiated modules will also work. The forward module must return logits as first element and optional sample embeddings as second element. If no sample embeddings are returned, the implementation uses the original samples.

n_annotatorsint, default=None

Number of annotators. If n_annotators=None, the number of annotators is inferred from y when calling fit.

neural_net_param_dictdict, default=None

Additional arguments for skorch.net.NeuralNet. If neural_net_param_dict is None, no additional arguments are added. module, criterion, predict_nonlinearity, and train_split are not allowed in this dictionary.

sample_dtypestr or type, default=np.float32

Dtype to which input samples are cast inside the estimator. If set to None, the input dtype is preserved.

classesarray-like of shape (n_classes,), default=None

Holds the label for each class. If None, the classes are determined during the fit.

missing_labelscalar or string or np.nan or None, default=np.nan

Value to represent a missing label.

cost_matrixarray-like of shape (n_classes, n_classes), default=None

Cost matrix with cost_matrix[i,j] indicating cost of predicting class classes[j] for a sample of class classes[i]. Can be only set, if classes is not None.

random_stateint or RandomState sample or None, default=None

Determines random number for predict method. Pass an int for reproducible results across multiple method calls.

References

[1]

Rodrigues, Filipe, and Francisco Pereira. “Deep Learning from Crowds.” AAAI Conference on Artificial Intelligence, 2018.

Methods

fit(X, y, **fit_params)	Initialize and fit the module.
get_metadata_routing()	Get metadata routing of this object.
get_params([deep])	Get parameters for this estimator.
initialize([X, y, enforce_check_X_y])	Initialize the wrapper and (optionally) validate inputs.
partial_fit(X, y, **fit_params)	Fit the module without re-initialization.
predict(X[, extra_outputs])	Return class predictions for the test samples X.
predict_proba(X[, extra_outputs])	Return class probability estimates for the test samples X.
score(X, y[, sample_weight])	Return the mean accuracy on the given test data and labels.
set_fit_request(*[, sample_weight])	Configure whether metadata should be requested to be passed to the fit method.
set_params(**params)	Set the parameters of this estimator.
set_predict_proba_request(*[, extra_outputs])	Configure whether metadata should be requested to be passed to the predict_proba method.
set_predict_request(*[, extra_outputs])	Configure whether metadata should be requested to be passed to the predict method.
set_score_request(*[, sample_weight])	Configure whether metadata should be requested to be passed to the score method.

fit(X, y, **fit_params)

Initialize and fit the module.

If the module was already initialized, by calling fit, the module will be re-initialized (unless warm_start is True).

Parameters:

Xmatrix-like, shape (n_samples, …)

Training data set, usually complete, i.e., including the labeled and unlabeled samples

yarray-like of shape (n_samples, n_annotators)

It contains the class labels of the training samples, where missing labels are represented via missing_label. Specifically, label y[n, m] refers to the label of sample X[n] from annotator m.

fit_paramsdict-like

Further parameters as input to the ‘fit’ method of the skorch.net.NeuralNet.

Returns:

self: _SkorchMultiAnnotatorClassifier,

_SkorchMultiAnnotatorClassifier object fitted on the training data.

get_metadata_routing()

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:

routingMetadataRequest

A MetadataRequest encapsulating routing information.

get_params(deep=True)

Get parameters for this estimator.

Parameters:

deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

paramsdict

Parameter names mapped to their values.

initialize(X=None, y=None, enforce_check_X_y=False)

Initialize the wrapper and (optionally) validate inputs.

If any data is provided or enforce_check_X_y is True, inputs are validated via _validate_data. A new skorch.NeuralNet is then created and assigned to self.neural_net_.

Parameters:

Xarray-like of shape (n_samples, …), default=None

Input samples for optional validation.

yarray-like of shape (n_samples, …), default=None

Target values for optional validation.

enforce_check_X_ybool, default=False

Whether to validate even if both X and y are None.

Returns:

selfSkorchMixin

Returned when no input data was supplied (both X and y are None).

X_out, y_outtuple of nd.array, optional

Validated X and y as a tuple, returned when enforce_check_X_y=True.

partial_fit(X, y, **fit_params)

Fit the module without re-initialization.

If the module was already initialized, by calling partial_fit, the module will not be re-initialized again.

Parameters:

Xmatrix-like, shape (n_samples, …)

Training data set, usually complete, i.e., including the labeled and unlabeled samples

yarray-like of shape (n_samples, n_annotators)

It contains the class labels of the training samples, where missing labels are represented via missing_label. Specifically, label y[n, m] refers to the label of sample X[n] from annotator m.

fit_paramsdict-like

Further parameters as input to the ‘partial_fit’ method of the skorch.net.NeuralNet.

Returns:

self: _SkorchMultiAnnotatorClassifier,

_SkorchMultiAnnotatorClassifier object fitted on the training data.

predict(X, extra_outputs=None)

Return class predictions for the test samples X.

By default, this method returns only the class predictions y_pred. If extra_outputs is provided, a tuple is returned whose first element is y_pred and whose remaining elements are the requested additional forward outputs, in the order specified by extra_outputs.

Parameters:

Xarray-like of shape (n_samples, …)

Test samples.

extra_outputsNone or str or sequence of str, default=None

Names of additional outputs to return next to y_pred. The names must be a subset of the following keys:

• “logits” : Additionally return the class-membership logits L_class for the samples in X.

• “embeddings” : Additionally return the learned embeddings X_embed for the samples in X.

• “annotator_perf” : additionally return the estimated annotator performance probabilities P_perf for each sample–annotator pair.

• “annotator_class” : Additionally return the annotator–class probability estimates P_annot for each sample, class, and annotator.

Returns:

y_prednumpy.ndarray of shape (n_samples,)

Class predictions of the test samples.

*extrasnumpy.ndarray, optional

Only returned if extra_outputs is not None. In that case, the method returns a tuple whose first element is y_pred and whose remaining elements correspond to the requested forward outputs in the order given by extra_outputs. Potential outputs are:

• L_class : np.ndarray of shape (n_samples, n_classes), where L_class[n, c] is the logit for the class classes_[c] of sample X[n].

• X_embed : np.ndarray of shape (n_samples, …), where X_embed[n] refers to the learned embedding for sample X[n].

• P_perf : np.ndarray of shape (n_samples, n_annotators), where P_perf[n, m] refers to the estimated label correctness probability (performance) of annotator m when labeling sample X[n].

• P_annot : np.ndarray of shape (n_samples, n_annotators, n_classes), where P_annot[n, m, c] refers to the probability that annotator m provides the class label c for sample X[n].

predict_proba(X, extra_outputs=None)

Return class probability estimates for the test samples X.

By default, this method returns only the class probabilities P. If extra_outputs is provided, a tuple is returned whose first element is P and whose remaining elements are the requested additional forward outputs, in the order specified by extra_outputs.

Parameters:

Xarray-like of shape (n_samples, …)

Test samples.

extra_outputsNone or str or sequence of str, default=None

Names of additional outputs to return next to P. The names must be a subset of the following keys:

• “logits” : Additionally return the class-membership logits L_class for the samples in X.

• “embeddings” : Additionally return the learned embeddings X_embed for the samples in X.

• “annotator_perf” : additionally return the estimated annotator performance probabilities P_perf for each sample–annotator pair.

• “annotator_class” : Additionally return the annotator–class probability estimates P_annot for each sample, class, and annotator.

Returns:

Pnumpy.ndarray of shape (n_samples, n_classes)

Class probabilities of the test samples. Classes are ordered according to self.classes_.

*extrasnumpy.ndarray, optional

Only returned if extra_outputs is not None. In that case, the method returns a tuple whose first element is P and whose remaining elements correspond to the requested forward outputs in the order given by extra_outputs. Potential outputs are:

• L_class : np.ndarray of shape (n_samples, n_classes), where L_class[n, c] is the logit for the class classes_[c] of sample X[n].

• X_embed : np.ndarray of shape (n_samples, …), where X_embed[n] refers to the learned embedding for sample X[n].

• P_perf : np.ndarray of shape (n_samples, n_annotators), where P_perf[n, m] refers to the estimated label correctness probability (performance) of annotator m when labeling sample X[n].

• P_annot : np.ndarray of shape (n_samples, n_annotators, n_classes), where P_annot[n, m, c] refers to the probability that annotator m provides the class label c for sample X[n].

score(X, y, sample_weight=None)

Return the mean accuracy on the given test data and labels.

Parameters:

Xarray-like of shape (n_samples, n_features)

Test samples.

yarray-like of shape (n_samples,)

True labels for X.

sample_weightarray-like of shape (n_samples,), default=None

Sample weights.

Returns:

scorefloat

Mean accuracy of self.predict(X) regarding y.

set_fit_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → CrowdLayerClassifier

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to fit.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns:

selfobject

The updated object.

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:

**paramsdict

Estimator parameters.

Returns:

selfestimator instance

Estimator instance.

set_predict_proba_request(*, extra_outputs: bool | None | str = '$UNCHANGED$') → CrowdLayerClassifier

Configure whether metadata should be requested to be passed to the predict_proba method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to predict_proba if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to predict_proba.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

extra_outputsstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for extra_outputs parameter in predict_proba.

Returns:

selfobject

The updated object.

set_predict_request(*, extra_outputs: bool | None | str = '$UNCHANGED$') → CrowdLayerClassifier

Configure whether metadata should be requested to be passed to the predict method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to predict if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to predict.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

extra_outputsstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for extra_outputs parameter in predict.

Returns:

selfobject

The updated object.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → CrowdLayerClassifier

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to score.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns:

selfobject

The updated object.