Strategy Overview

This is an overview of all implemented pool- and stream-based active learning strategies, which are often divided into threemain categories based on the utilities they compute for sample selection:

1. Informativeness-based strategies mostly select samples for which the model is most uncertain (e.g., via information-theoretic measures).

2. Representativeness-based strategies select samples that capture the overall data distribution (e.g., via clustering ordensity estimation).

3. Hybrid strategies combine both criteria to select samples that are informative and representative.

Furthermore, we distinguish between regression and classification as supervised learning tasks, where labels canbe provided by a single annotator or multiple annotators. Finally, a strategy builds a batch of samples by either including the samples with the top-k utilities or by including diverse samples with high utility scores.

You can use the checkboxes below to filter the query strategies based on these distinctions.

Regression Classification Top-k-Batch Diverse-Batch Single-Annotator Multi-Annotator

Pool-based AL Strategies

Baseline

Method	Base Class	Tags	Reference
Random Sampling	RandomSampling	pool regression classification single-annotator diverse-batch

Hybrid

Method	Base Class	Tags	Reference
Batch Active Learning by Diverse Gradient Embedding (BADGE)	Badge	pool classification single-annotator diverse-batch	Ash et al.[1]
Clustering Uncertainty-weighted Embeddings (CLUE)	Clue	pool classification regression single-annotator diverse-batch	Prabhu et al.[2]
Contrastive Active Learning (CAL)	ContrastiveAL	pool classification single-annotator top-k-batch	Margatina et al.[3]
Dropout Query (DropQuery)	DropQuery	pool classification single-annotator diverse-batch	Gupte et al.[4]
Fast Active Learning by Contrastive UNcertainty (FALCUN)	Falcun	pool classification single-annotator diverse-batch	Gilhuber et al.[5]
Density-Diversity-Distribution-Distance Sampling (4DS)	FourDs	pool classification single-annotator diverse-batch	Reitmaier and Sick[6]
Batch Density-Diversity-Distribution-Distance Sampling (Batch4DS)	FourDs	pool classification single-annotator diverse-batch	Reitmaier and Sick[6]
Multi-class Probabilistic Active Learning (McPAL)	ProbabilisticAL	pool classification single-annotator top-k-batch	Kottke et al.[7]
Querying Informative and Representative Examples (QUIRE)	Quire	pool classification single-annotator top-k-batch	Huang et al.[8], Huang et al.[9]
Density-weighted Uncertainty Sampling (DWUS)	UncertaintySampling	pool classification single-annotator top-k-batch	Donmez et al.[10]
Dual Strategy for Active Learning (DUAL)	UncertaintySampling	pool classification single-annotator	Donmez et al.[10]

Informativeness

Method	Base Class	Tags	Reference
Batch Bayesian Active Learning by Disagreement (BatchBALD)	BatchBALD	pool classification single-annotator diverse-batch	Houlsby et al.[11], Kirsch et al.[12]
Active Learning with Cost Embedding (ALCE)	CostEmbeddingAL	pool classification single-annotator top-k-batch	Huang and Lin[13]
Epistemic Uncertainty Sampling (EpisUS)	EpistemicUncertaintySampling	pool classification single-annotator top-k-batch	Nguyen et al.[14]
Bayesian Active Learning by Disagreement (BALD)	GreedyBALD	pool classification single-annotator top-k-batch	Houlsby et al.[11]
Monte-Carlo Expected Error Reduction (EER) with Log-Loss	MonteCarloEER	pool classification single-annotator top-k-batch	Roy and McCallum[15]
Monte-Carlo Expected Error Reduction (EER) with Misclassification-Loss	MonteCarloEER	pool classification single-annotator top-k-batch	Roy and McCallum[15]
Query-by-Committee (QBC) with Kullback-Leibler Divergence	QueryByCommittee	pool classification single-annotator top-k-batch	Seung et al.[16], McCallum and Nigamy[17]
Query-by-Committee (QBC) with Vote Entropy	QueryByCommittee	pool classification single-annotator top-k-batch	Seung et al.[16], Engelson and Dagan[18]
Query-by-Committee (QBC) with Variation Ratios	QueryByCommittee	pool classification single-annotator top-k-batch	Seung et al.[16], Beluch et al.[19]
Uncertainty Sampling (US) with Margin	UncertaintySampling	pool classification single-annotator top-k-batch	Settles[20]
Uncertainty Sampling (US) with Least-Confidence	UncertaintySampling	pool classification single-annotator top-k-batch	Settles[20]
Uncertainty Sampling (US) with Entropy	UncertaintySampling	pool classification single-annotator top-k-batch	Settles[20]
Uncertainty Sampling with Expected Average Precision (USAP)	UncertaintySampling	pool classification single-annotator top-k-batch	Wang et al.[21]
Value of Information (VOI) on Unlabeled Samples	ValueOfInformationEER	pool classification single-annotator top-k-batch	Joshi et al.[22]
Value of Information (VOI) on Labeled Samples	ValueOfInformationEER	pool classification single-annotator top-k-batch	Margineantu[23]
Value of Information (VOI)	ValueOfInformationEER	pool classification single-annotator top-k-batch	Kapoor et al.[24]

Representativeness

Method	Base Class	Tags	Reference
Core Set	CoreSet	pool regression classification single-annotator diverse-batch	Sener and Savarese[25]
Discriminative Active Learning (DAL)	DiscriminativeAL	pool classification regression single-annotator diverse-batch	Gissin and Shalev-Shwartz[26]
MaxHerding	MaxHerding	pool regression classification single-annotator diverse-batch	Bae et al.[27]
Probability Coverage (ProbCover)	ProbCover	pool classification single-annotator diverse-batch	Yehuda et al.[28]
Typical Clustering (TypiClust)	TypiClust	pool regression classification single-annotator diverse-batch	Hacohen et al.[29]

Wrapper

Method	Base Class	Tags	Reference
Parallel Utility Estimation Wrapper	ParallelUtilityEstimationWrapper	pool regression classification single-annotator top-k-batch
Sub-sampling Wrapper	SubSamplingWrapper	pool regression classification single-annotator top-k-batch diverse-batch

Pool-based AL Strategies for Regression

Hybrid

Method	Base Class	Tags	Reference
Regression Tree Based Active Learning (RT-AL) with Random Selection	RegressionTreeBasedAL	pool regression single-annotator diverse-batch	Jose et al.[30]
Regression Tree Based Active Learning (RT-AL) with Diversity Selection	RegressionTreeBasedAL	pool regression single-annotator diverse-batch	Jose et al.[30]
Regression Tree Based Active Learning (RT-AL) with Representativity Selection	RegressionTreeBasedAL	pool regression single-annotator diverse-batch	Jose et al.[30]

Informativeness

Method	Base Class	Tags	Reference
Expected Model Change	ExpectedModelChangeMaximization	pool regression single-annotator top-k-batch	Cai et al.[31]
Expected Model Output Change	ExpectedModelOutputChange	pool regression single-annotator top-k-batch	Käding et al.[32]
Expected Model Variance Reduction	ExpectedModelVarianceReduction	pool regression single-annotator top-k-batch	Cohn et al.[33]
Regression based Kullback Leibler Divergence Maximization	KLDivergenceMaximization	pool regression single-annotator top-k-batch	Elreedy et al.[34]
Query-by-Committee (QBC) with Empirical Variance	QueryByCommittee	pool regression single-annotator	Seung et al.[16], Burbidge et al.[35]

Representativeness

Method	Base Class	Tags	Reference
Greedy Sampling on the Target Space (GSy)	GreedySamplingTarget	pool regression single-annotator diverse-batch	Wu et al.[36]
Improved Greedy Sampling (GSi)	GreedySamplingTarget	pool regression single-annotator diverse-batch	Wu et al.[36]
Greedy Sampling on the Feature Space (GSx)	GreedySamplingX	pool regression classification single-annotator diverse-batch	Wu et al.[36]

Pool-based AL Strategies for Mulitple Annotators

Baseline

Method	Base Class	Tags	Reference
Interval Estimation Threshold	IntervalEstimationThreshold	pool classification multi-annotator
Random Sampling	SingleAnnotatorWrapper	pool classification multi-annotator
Core Set + Greedy Selection	SingleAnnotatorWrapper	pool classification multi-annotator

Stream-based AL Strategies

Baseline

Method	Base Class	Tags	Reference
Periodic Sampling	PeriodicSampling	stream classification single-annotator top-k-batch
Stream Random Sampling	StreamRandomSampling	stream classification single-annotator top-k-batch

Hybrid

Method	Base Class	Tags	Reference
Cognitive Dual-Query Strategy with Fixed-Uncertainty	CognitiveDualQueryStrategyFixUn	stream classification single-annotator top-k-batch	Liu et al.[37]
Cognitive Dual-Query Strategy with Random Sampling	CognitiveDualQueryStrategyRan	stream classification single-annotator top-k-batch	Liu et al.[37]
Cognitive Dual-Query Strategy with Randomized-Variable-Uncertainty	CognitiveDualQueryStrategyRanVarUn	stream classification single-annotator top-k-batch	Liu et al.[37]
Cognitive Dual-Query Strategy with Variable-Uncertainty	CognitiveDualQueryStrategyVarUn	stream classification single-annotator top-k-batch	Liu et al.[37]
Split	Split	stream classification single-annotator top-k-batch	Žliobaitė et al.[38]
Density Based Active Learning for Data Streams	StreamDensityBasedAL	stream classification single-annotator top-k-batch	Ienco et al.[39]
Probabilistic Active Learning in Datastreams	StreamProbabilisticAL	stream classification single-annotator top-k-batch	Kottke et al.[40]

Informativeness

Method	Base Class	Tags	Reference
Fixed-Uncertainty	FixedUncertainty	stream classification single-annotator top-k-batch	Žliobaitė et al.[38]
Randomized-Variable-Uncertainty	RandomVariableUncertainty	stream classification single-annotator top-k-batch	Žliobaitė et al.[38]
Variable-Uncertainty	VariableUncertainty	stream classification single-annotator top-k-batch	Žliobaitė et al.[38]

References

[1]

Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. In Int. Conf. Learn. Represent. 2020.

[2]

Viraj Prabhu, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman. Active domain adaptation via clustering uncertainty-weighted embeddings. In IEEE/CVF Int. Conf. Comput. Vis., 8505–8514. 2021.

[3]

Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. Active Learning by Acquiring Contrastive Examples. In Conf. Empir. Methods Nat. Lang. Process., 650–663. 2021.

[4]

Sanket Rajan Gupte, Josiah Aklilu, Jeffrey J Nirschl, and Serena Yeung-Levy. Revisiting Active Learning in the Era of Vision Foundation Models. Trans. Mach. Learn. Res., 2024.

[5]

Sandra Gilhuber, Anna Beer, Yunpu Ma, and Thomas Seidl. FALCUN: A Simple and Efficient Deep Active Learning Strategy. In Jt. Eur. Conf. Mach. Learn. Knowl. Discov. Databases, 421–439. 2024.

[6] (1,2)

Tobias Reitmaier and Bernhard Sick. Let us know your decision: Pool-based active training of a generative classifier with the selection strategy 4DS. Inf. Sci., 230:106–131, 2013.

[7]

Daniel Kottke, Georg Krempl, Dominik Lang, Johannes Teschner, and Myra Spiliopoulou. Multi-class Probabilistic Active Learning. In Eur. Conf. Artif. Intell., 586–594. 2016.

[8]

Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. Active Learning by Querying Informative and Representative Examples. In Adv. Neural Inf. Process. Syst. 2010.

[9]

Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. Active Learning by Querying Informative and Representative Examples. IEEE Trans. Pattern Anal. Mach. Intell., 36(10):1936–1949, 2014.

[10] (1,2)

Pinar Donmez, Jaime G Carbonell, and Paul N Bennett. Dual Strategy Active Learning. In Eur. Conf. Mach. Learn., 116–127. 2007.

[11] (1,2)

Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classification and Preference Learning. arXiv:1112.5745, 2011.

[12]

Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning. In Adv. Neural Inf. Process. Syst. 2019.

[13]

Kuan-hao Huang and Hsuan-tien Lin. A Novel Uncertainty Sampling Algorithm for Cost-Sensitive Multiclass Active Learning. In IEEE Int. Conf. Data Min., 925–930. 2016.

[14]

Vu-Linh Nguyen, Sébastien Destercke, and Eyke Hüllermeier. Epistemic Uncertainty Sampling. In Int. Conf. Discov. Sci., 72–86. 2019.

[15] (1,2)

Nicholas Roy and Andrew McCallum. Toward Optimal Active Learning through Monte Carlo Estimation of Error Reduction. In Int. Conf. Mach. Learn., 441–448. 2001.

[16] (1,2,3,4)

H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by Committee. In Annu. Workshop Comput. Learn. Theory., 287––294. 1992.

[17]

Andrew Kachites McCallum and Kamal Nigamy. Employing EM and Pool-Based Active Learning for Text Classification. In Int. Conf. Mach. Learn., 359–367. 1998.

[18]

Sean P Engelson and Ido Dagan. Minimizing Manual Annotation Cost in Supervised Training from Corpora. In Annu. Meet. Assoc. Comput. Linguist., 319–326. 1996.

[19]

William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The Power of Ensembles for Active Learning in Image Classification. In IEEE Conf. Comput. Vis. Pattern Recognit., 9368–9377. 2018.

[20] (1,2,3)

Burr Settles. Active learning literature survey. Technical Report 1648, University of Wisconsin, Department of Computer Science, 2009.

[21]

Hanmo Wang, Xiaojun Chang, Lei Shi, Yi Yang, and Yi-Dong Shen. Uncertainty Sampling for Action Recognition via Maximizing Expected Average Precision. In Int. Jt. Conf. Artif. Intell., 964–970. 2018.

[22]

Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class Active Learning for Image Classification. In IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2372–2379. 2009.

[23]

Dragos D Margineantu. Active Cost-Sensitive Learning. In Int. Jt. Conf. Artif. Intell., 1622–1623. 2005.

[24]

Ashish Kapoor, Eric Horvitz, and Sumit Basu. Selective Supervision: Guiding Supervised Learning with Decision-Theoretic Active Learning. In Int. Jt. Conf. Artif. Intell., 877–882. 2007.

[25]

Ozan Sener and Silvio Savarese. Active Learning for Convolutional Neural Networks: A Core-Set Approach. In Int. Conf. Learn. Represent. 2018.

[26]

Daniel Gissin and Shai Shalev-Shwartz. Discriminative Active Learning. arXiv:1907.06347, 2019.

[27]

Wonho Bae, Junhyug Noh, and Danica J Sutherland. Generalized Coverage for More Robust Low-Budget Active Learning. In Eur. Conf. Comput. Vis., 318–334. 2024.

[28]

Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active Learning Through a Covering Lens. In Adv. Neural Inf. Process. Syst. 2022.

[29]

Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets. In Int. Conf. Mach. Learn., 8175–8195. 2022.

[30] (1,2,3)

Ashna Jose, João Paulo Almeida de Mendonça, Emilie Devijver, Noël Jakse, Valérie Monbet, and Roberta Poloni. Regression Tree-based Active Learning. Data Min. Knowl. Discov., pages 420–460, 2023.

[31]

Wenbin Cai, Ya Zhang, and Jun Zhou. Maximizing Expected Model Change for Active Learning in Regression. In IEEE Int. Conf. Data Min., 51–60. 2013.

[32]

Christoph Käding, Erik Rodner, Alexander Freytag, Oliver Mothes, Björn Barz, and Joachim Denzler. Active Learning for Regression Tasks with Expected Model Output Changes. In Br. Mach. Vis. Conf. 2018.

[33]

David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active Learning with Statistical Models. J. Artif. Intell. Res., 4:129–145, 1996.

[34]

Dina Elreedy, Amir F. Atiya, and Samir I. Shaheen. A Novel Active Learning Regression Framework for Balancing the Exploration-Exploitation Trade-Off. Entropy, 21(7):651, 2019.

[35]

Robert Burbidge, Jem J Rowland, and Ross D King. Active Learning for Regression Based on Query by Committee. In Intell. Data Eng. Autom. Learn., 209–218. 2007.

[36] (1,2,3)

Dongrui Wu, Chin-Teng Lin, and Jian Huang. Active Learning for Regression using Greedy Sampling. Inf. Sci., 474:90–105, 2019.

[37] (1,2,3,4)

Sanmin Liu, Shan Xue, Jia Wu, Chuan Zhou, Jian Yang, Zhao Li, and Jie Cao. Online Active Learning for Drifting Data Streams. IEEE Trans. Neural Netw. Learn. Syst., 34(1):186–200, 2023.

[38] (1,2,3,4)

Indrė Žliobaitė, Albert Bifet, Bernhard Pfahringer, and Geoffrey Holmes. Active Learning With Drifting Streaming Data. IEEE Trans. Neural Netw. Learn. Syst., 25(1):27–39, 2014.

[39]

Dino Ienco, Indrė Žliobaitė, and Bernhard Pfahringer. High density-focused uncertainty sampling for active learning over evolving stream data. In Int. Workshop Big Data Streams Heterog. Source Min. Algorithms Syst. Program. Models Appl., 133–148. 2014.

[40]

Daniel Kottke, Georg Krempl, and Myra Spiliopoulou. Probabilistic Active Learning in Datastreams. In Adv. Intell. Data Anal., 145–157. 2015.