UncertaintySampling#

class skactiveml.pool.UncertaintySampling(method='least_confident', cost_matrix=None, missing_label=nan, random_state=None, multilabel_aggregation_fn=<function mean>, target_type='auto')[source]#

Bases: SingleAnnotatorPoolQueryStrategy

Uncertainty Sampling (US)

This class implement various uncertainty based query strategies, i.e., the standard uncertainty measures [1], cost-sensitive ones [2], and one optimizing expected average precision [3]. Concretely, four uncertainty score definitions are available:

least confident [1] selecting samples whose predicted top class has the lowest confidence,
margin sampling [1] selecting samples where the gap between the two most probable classes is smallest,
entropy-based uncertainty [1] selecting samples with the highest overall predictive uncertainty across classes,
and expected average precision [3] selecting samples with the highest expected improvement in average precision under the model’s current predictions.

For the least confident and margin sampling cost-sensitive variants are implemented considering a user-defined cost matrix [2]. Finally, the class can also be leverage to implement variants of density-weighted uncertainty sampling (DWUS) and the dual strategy for active learning (DUAL) [4].

The uncertainty measures were proposed for single-output classification. Multi-label support in this implementation is an extension and not part of the original proposal in [1]. For resolved multi-label targets, the per-label score of the label output j is computed from its positive-class probability p_j alone, i.e., min(p_j, 1 - p_j) for ‘least_confident’, 1 - |2 * p_j - 1| for ‘margin_sampling’, and the binary entropy for ‘entropy’ (cf. uncertainty_scores). multilabel_aggregation_fn reduces these per-label scores along the label axis to the utility of one sample. This per-output decomposition ignores correlations between label outputs.

Parameters:

method‘least_confident’ or ‘margin_sampling’ or ‘entropy’ or ‘expected_average_precision’, default=’least_confident’: The method to calculate the uncertainty. For multilabel targets (y.ndim == 2), only ‘least_confident’, ‘margin_sampling’, and ‘entropy’ are supported.
cost_matrixarray-like of shape (n_classes, n_classes): Cost matrix with cost_matrix[i,j] defining the cost of predicting class j for a sample with the actual class i. Only supported for least_confident and margin_sampling variant.
missing_labelscalar or string or np.nan or None, default=np.nan: Value to represent a missing label.
random_stateint or np.random.RandomState: The random state to use.
multilabel_aggregation_fncallable, default=np.mean: Callable reducing the per-label uncertainty scores of one sample to one utility. It is only used for resolved multi-label targets and method in [‘least_confident’, ‘margin_sampling’, ‘entropy’]. It is called with the per-label scores of shape (n_samples, n_outputs) and the label axis passed as the axis keyword argument, and must return one score per sample within the range of that sample’s per-label scores, e.g. np.mean, np.average, np.median, np.min, np.max, or a quantile. np.sum is not supported, because its result grows with the number of label outputs. Only the callability of the reduction is validated at runtime, so a violating reduction silently changes the acquisition scale.
target_type“auto” or “single-output” or “multi-label”, default=”auto”: Declared target type. Single-output classification is always supported. Multi-label classification is supported only when cost_matrix=None and method is “least_confident”, “margin_sampling”, or “entropy”. A fitted classifier’s target specification is authoritative when available.

References

[1] (1,2,3,4,5)

Settles, Burr. Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences, 2009.

[2] (1,2)

P.-L. Chen and H.-T. Lin. Active Learning for Multiclass Cost-Sensitive Classification Using Probabilistic Models. In Conf. Technol. Appl. Artif. Intell., pages 13–18, 2013.

[3]

H. Wang, X. Chang, L. Shi, Y. Yang, and Y.-D. Shen. Uncertainty Sampling for Action Recognition via Maximizing Expected Average Precision. In Int. Jt. Conf. Artif. Intell., pages 964–970, 2018.

[4]

P. Donmez, J. G. Carbonell, and P. N. Bennett. Dual Strategy Active Learning. In Eur. Conf. Mach. Learn, pages 116-127, 2007.

Methods

`query`(X, y, clf[, fit_clf, sample_weight, ...])	Determines for which candidate samples labels are to be queried.
`get_metadata_routing`()	Get metadata routing of this object.
`get_params`([deep])	Get parameters for this estimator.
`set_params`(**params)	Set the parameters of this estimator.

UncertaintySampling.query(X, y, clf, fit_clf=True, sample_weight=None, utility_weight=None, candidates=None, batch_size=1, return_utilities=False)[source]#

Determines for which candidate samples labels are to be queried.

Parameters:

Xarray-like of shape (n_samples, n_features)

Training data set, usually complete, i.e., including the labeled and unlabeled samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs)

Labels of the training data set (possibly including unlabeled ones indicated by self.missing_label). If y is two-dimensional, a row y[i] must either contain only observed labels or only missing_label values, i.e., no mixing within a row.

clfskactiveml.base.SkactivemlClassifier

Model implementing the methods fit and predict_proba. For multi-label classification, predict_proba may return either positive-class probabilities of shape (n_samples, n_outputs) or a list of binary probability matrices with shape (n_samples, 2) per label output.

fit_clfbool, default=True

Defines whether the classifier should be fitted on X, y, and sample_weight.

sample_weightarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None

Weights of training samples in X. For two-dimensional y, one weight per sample is supported. Per-target weights are forwarded to clf.fit without additional validation and require estimator support.

utility_weightarray-like, default=None

Weight for each candidate (multiplied with utilities). Usually, this is to be the density of a candidate. The length of utility_weight is usually n_samples, except for the case when candidates contains samples (ndim >= 2). Then the length is n_candidates.

candidatesNone or array-like of shape (n_candidates), dtype=int or array-like of shape (n_candidates, n_features), default=None

If candidates is None, the unlabeled samples from (X,y) are considered as candidates.
If candidates is of shape (n_candidates,) and of type int, candidates is considered as the indices of the samples in (X,y).
If candidates is of shape (n_candidates, …), the candidate samples are directly given in candidates (not necessarily contained in X).

batch_sizeint, default=1

The number of samples to be selected in one AL cycle.

return_utilitiesbool, default=False

If true, also return the utilities based on the query strategy.

Returns:

query_indicesnumpy.ndarray of shape (batch_size)

The query indices indicate for which candidate sample a label is to be queried, e.g., query_indices[0] indicates the first selected sample.

If candidates is None or of shape (n_candidates,), the indexing refers to the samples in X.
If candidates is of shape (n_candidates, n_features), the indexing refers to the samples in candidates.

utilitiesnumpy.ndarray of shape (batch_size, n_samples)

The utilities of samples after each selected sample of the batch, e.g., utilities[0] indicates the utilities used for selecting the first sample (with index query_indices[0]) of the batch. Utilities for labeled samples will be set to np.nan.

If candidates is None, the indexing refers to the samples in X.
If candidates is of shape (n_candidates,) and of type int, utilities refers to the samples in X.
If candidates is of shape (n_candidates, …), utilities refers to the indexing in candidates.

UncertaintySampling.get_metadata_routing()#

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:

routingMetadataRequest: A MetadataRequest encapsulating routing information.

UncertaintySampling.get_params(deep=True)#

Get parameters for this estimator.

Parameters:

deepbool, default=True: If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

paramsdict: Parameter names mapped to their values.

UncertaintySampling.set_params(**params)#

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters: