Badge#

class skactiveml.pool.Badge(clf_embedding_flag_name=None, missing_label=nan, random_state=None, target_type='auto')[source]#

Bases: SingleAnnotatorPoolQueryStrategy

Batch Active Learning by Diverse Gradient Embedding (BADGE)

This class implements the BADGE algorithm [1], which selects a batch by running k-means++ on per-sample gradient embeddings, which combine uncertainty and diversity. For each unlabeled sample, it forms the gradient of the cross-entropy loss with respect to the last linear layer using the model’s pseudo-label. Large gradient norms indicate uncertainty, while k-means++ spreads selections to avoid redundancy.

The gradient embedding of a sample is the Kronecker product g = kron(q, v) of its probability residual q and its (learned) sample representation v. Since inner products factorize as <g_i, g_j> = <q_i, q_j> * <v_i, v_j> [2], the (n_samples, n_classes * n_features) embedding matrix is never materialized. Each k-means++ round only requires two matrix-vector products, which reduces the space complexity from O(n_samples * n_classes * n_features) to O(n_samples * (n_classes + n_features)).

The original BADGE method was proposed for multiclass classification. The multi-label support in this implementation is an extension and not part of the original proposal in [1]. For resolved multi-label targets, BADGE assumes independent sigmoid outputs per label and forms a multi-label gradient embedding from the binary-cross-entropy-style last-layer gradients, i.e., the residual of the label output j against the model’s own pseudo-label is p_j - 1[p_j >= 0.5], and the per-output last-layer gradients obtained by multiplying these residuals with the sample representation are concatenated into one gradient embedding. This per-output decomposition ignores correlations between label outputs. The factorization above applies unchanged, since the multi-label residual is also a per-output vector q.

Parameters:
clf_embedding_flag_namedict or str or None, default=None

Flag, which is passed to the predict_proba method for getting the (learned) sample representations.

  • If clf_embedding_flag_name is None and predict_proba returns only one output, the input samples X are used.

  • If clf_embedding_flag_name is None and predict_proba returns two outputs, (proba, embeddings) are expected as outputs.

  • If isinstance(clf_embedding_name, str), we call:

    clf.predict_proba(X, **{clf_embedding_flag_name: True})

    and expect (proba, embeddings) as output.

  • If isinstance(clf_embedding_name, dict), we call:

    clf.predict_proba(X, **clf_embedding_flag_name)

    and expect (proba, embeddings) as output.

missing_labelscalar or string or np.nan or None, default=np.nan

Value to represent a missing label.

random_stateNone or int or np.random.RandomState, default=None

The random state to use.

target_type“auto” or “single-output” or “multi-label”, default=”auto”

Declared target type. The strategy supports single-output and multi-label classification. A fitted classifier’s target specification is authoritative when available.

References

[1] (1,2)

J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. In Int. Conf. Learn. Represent., 2020.

[2]

J. Zhang, Y. Chen, G. Canal, S. Mussmann, A. M. Das, G. Bhatt, Y. Zhu, J. Bilmes, S. S. Du, K. Jamieson, and R. D. Nowak. LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning. J. Data-centric Mach. Learn. Res., 2024.

Methods

query(X, y, clf[, fit_clf, sample_weight, ...])

Determines for which candidate samples labels are to be queried.

get_metadata_routing()

Get metadata routing of this object.

get_params([deep])

Get parameters for this estimator.

set_params(**params)

Set the parameters of this estimator.
Badge.query(X, y, clf, fit_clf=True, sample_weight=None, candidates=None, batch_size=1, return_utilities=False)[source]#

Determines for which candidate samples labels are to be queried.

Parameters:
Xarray-like of shape (n_samples, n_features)

Training data set, usually complete, i.e., including the labeled and unlabeled samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs)

Labels of the training data set (possibly including unlabeled ones indicated by self.missing_label). For multi-label targets, a row y[i] must either contain only observed labels or only missing_label values, i.e., no mixing within a row. In this case, BADGE uses the multi-label extension described in the class docstring, i.e., independent sigmoid outputs per label. predict_proba must then return either one positive-class probability per label with shape (n_samples, n_outputs) or a list of binary probability matrices with shape (n_samples, 2) per output.

clfskactiveml.base.SkactivemlClassifier

Classifier implementing the methods fit and predict_proba.

fit_clfbool, default=True

Defines whether the classifier clf should be fitted on X, y, and sample_weight.

sample_weight: array-like of shape (n_samples,) or (n_samples, n_outputs), default=None

Weights of training samples in X. For two-dimensional y, one weight per sample is supported. Per-target weights are forwarded to clf.fit without additional validation and require estimator support.

candidatesNone or array-like of shape (n_candidates,), dtype=int or array-like of shape (n_candidates, n_features), default=None

  • If candidates is None, the unlabeled samples from (X,y) are considered as candidates.

  • If candidates is of shape (n_candidates,) and of type int, candidates is considered as the indices of the samples in (X,y).

  • If candidates is of shape (n_candidates, …), the candidate samples are directly given in candidates (not necessarily contained in X).

A given candidates is authoritative, i.e., an index array is taken as given, such that labeled samples remain candidates, e.g., to relabel them or to recompute their utilities.

batch_sizeint, default=1

The number of samples to be selected in one AL cycle. If it exceeds the number of candidates, it is reduced to that number and a warning is raised.

return_utilitiesbool, default=False

If True, also return the utilities based on the query strategy.

Returns:
query_indicesnumpy.ndarray of shape (batch_size,)

The query indices indicate for which candidate sample a label is to be queried, e.g., query_indices[0] indicates the first selected sample. A sample is selected at most once per batch.

  • If candidates is None or of shape (n_candidates,), the indexing refers to the samples in X.

  • If candidates is of shape (n_candidates, n_features), the indexing refers to the samples in candidates.

utilitiesnumpy.ndarray of shape (batch_size, n_samples) or numpy.ndarray of shape (batch_size, n_candidates)

The utilities of samples after each selected sample of the batch, e.g., utilities[0] indicates the utilities used for selecting the first sample (with index query_indices[0]) of the batch. Each row is the k-means++ sampling distribution of the respective round, i.e., its nansum is one. Utilities for samples that are no candidates and for candidates that have already been selected in an earlier round will be set to np.nan.

  • If candidates is None or of shape (n_candidates,), the indexing refers to the samples in X.

  • If candidates is of shape (n_candidates, n_features), the indexing refers to the samples in candidates.

Badge.get_metadata_routing()#

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:
routingMetadataRequest

A MetadataRequest encapsulating routing information.

Badge.get_params(deep=True)#

Get parameters for this estimator.

Parameters:
deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:
paramsdict

Parameter names mapped to their values.

Badge.set_params(**params)#

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:
**paramsdict

Estimator parameters.

Returns:
selfestimator instance

Estimator instance.

Examples using skactiveml.pool.Badge#

Batch Active Learning by Diverse Gradient Embedding (BADGE)

Batch Active Learning by Diverse Gradient Embedding (BADGE)