Strategy Overview

This is an overview of all implemented active learning strategies.

You can use the following checkboxes to filter the tables below.

Regression Classification Multi-Annotator Single-Annotator

Pool

Expected Error Reduction

Method	Base Class	Tags	Reference
Expected Model Output Change	ExpectedModelOutputChange	pool regression single-annotator	Käding et al.1
Expected Model Variance Reduction	ExpectedModelVarianceReduction	pool regression single-annotator	Cohn et al.2
Regression based Kullback Leibler Divergence Maximization	KLDivergenceMaximization	pool regression single-annotator	Elreedy et al.3
Monte-Carlo EER with Log-Loss	MonteCarloEER	pool classification single-annotator	Roy and McCallum4
Monte-Carlo EER with Misclassification-Loss	MonteCarloEER	pool classification single-annotator	Roy and McCallum4
Value of Information on Unlabeled Samples	ValueOfInformationEER	pool classification single-annotator	Joshi et al.5
Value of Information on Labeled Samples	ValueOfInformationEER	pool classification single-annotator	Margineantu6
Value of Information (VOI)	ValueOfInformationEER	pool classification single-annotator	Kapoor et al.7

Meta

Method	Base Class	Tags	Reference
Parallel Utility Estimation Wrapper	ParallelUtilityEstimationWrapper	pool regression classification single-annotator
Sub-sampling Wrapper	SubSamplingWrapper	pool regression classification single-annotator

Model Change

Method	Base Class	Tags	Reference
Expected Model Change	ExpectedModelChangeMaximization	pool regression single-annotator	Cai et al.8

Query-by-Committee

Method	Base Class	Tags	Reference
Batch Bayesian Active Learning by Disagreement (BatchBALD)	BatchBALD	pool classification single-annotator	Houlsby et al.9, Kirsch et al.10
Bayesian Active Learning by Disagreement (BALD)	GreedyBALD	pool classification single-annotator	Houlsby et al.9
Query-by-Committee (QBC) with Kullback-Leibler Divergence	QueryByCommittee	pool classification single-annotator	Seung et al.11, McCallum and Nigamy12
Query-by-Committee (QBC) with Vote Entropy	QueryByCommittee	pool classification single-annotator	Seung et al.11, Engelson and Dagan13
Query-by-Committee (QBC) with Variation Ratios	QueryByCommittee	pool classification single-annotator	Seung et al.11, Beluch et al.14
Query-by-Committee (QBC) with Empirical Variance	QueryByCommittee	pool regression single-annotator	Seung et al.11, Burbidge et al.15

Random Sampling

Method	Base Class	Tags	Reference
Random Sampling	RandomSampling	pool regression classification single-annotator

Uncertainty Sampling

Method	Base Class	Tags	Reference
Clustering Uncertainty-weighted Embeddings (CLUE)	Clue	pool classification single-annotator	Prabhu et al.16
Contrastive Active Learning (CAL)	ContrastiveAL	pool classification single-annotator	Margatina et al.17
Epistemic Uncertainty Sampling	EpistemicUncertaintySampling	pool classification single-annotator	Nguyen et al.18
Density-Diversity-Distribution-Distance Sampling (4DS)	FourDs	pool classification single-annotator	Reitmaier and Sick19
Batch Density-Diversity-Distribution-Distance Sampling (Batch4DS)	FourDs	pool classification single-annotator	Reitmaier and Sick19
Uncertainty Sampling with Margin	UncertaintySampling	pool classification single-annotator	Settles20
Uncertainty Sampling with Least-Confidence	UncertaintySampling	pool classification single-annotator	Settles20
Uncertainty Sampling with Entropy	UncertaintySampling	pool classification single-annotator	Settles20
Expected Average Precision	UncertaintySampling	pool classification single-annotator	Wang et al.21
Density-weighted Uncertainty Sampling	UncertaintySampling	pool classification single-annotator	Donmez et al.22, Nguyen and Smeulders23
Dual strategy for Active Learning	UncertaintySampling	pool classification single-annotator	Donmez et al.22, Nguyen and Smeulders23

Others

Method	Base Class	Tags	Reference
Batch Active Learning by Diverse Gradient Embedding (BADGE)	Badge	pool classification single-annotator	Ash et al.24
Core Set	CoreSet	pool regression classification single-annotator	Sener and Savarese25
Active Learning with Cost Embedding (ALCE)	CostEmbeddingAL	pool classification single-annotator	Huang and Lin26
Discriminative Active Learning (DAL)	DiscriminativeAL	pool classification regression single-annotator	Gissin and Shalev-Shwartz27
Greedy Sampling on the Target Space (GSy)	GreedySamplingTarget	pool regression single-annotator	Wu et al.28
Improved Greedy Sampling (GSi)	GreedySamplingTarget	pool regression single-annotator	Wu et al.28
Greedy Sampling on the Feature Space (GSx)	GreedySamplingX	pool regression classification single-annotator	Wu et al.28
Probability Coverage (ProbCover)	ProbCover	pool classification single-annotator	Yehuda et al.29
Multi-class Probabilistic Active Learning (McPAL)	ProbabilisticAL	pool classification single-annotator	Kottke et al.30
Querying Informative and Representative Examples (QUIRE)	Quire	pool classification single-annotator	Huang et al.31
Regression Tree Based Active Learning (RT-AL) with Random Selection	RegressionTreeBasedAL	pool regression single-annotator	Jose et al.32
Regression Tree Based Active Learning (RT-AL) with Diversity Selection	RegressionTreeBasedAL	pool regression single-annotator	Jose et al.32
Regression Tree Based Active Learning (RT-AL) with Representativity Selection	RegressionTreeBasedAL	pool regression single-annotator	Jose et al.32
Typical Clustering (TypiClust)	TypiClust	pool regression classification single-annotator	Hacohen et al.33

Stream

Density Sampling

Method	Base Class	Tags	Reference
Cognitive Dual-Query Strategy with Fixed-Uncertainty	CognitiveDualQueryStrategyFixUn	stream classification single-annotator	Liu et al.34
Cognitive Dual-Query Strategy with Random Sampling	CognitiveDualQueryStrategyRan	stream classification single-annotator	Liu et al.34
Cognitive Dual-Query Strategy with Randomized-Variable-Uncertainty	CognitiveDualQueryStrategyRanVarUn	stream classification single-annotator	Liu et al.34
Cognitive Dual-Query Strategy with Variable-Uncertainty	CognitiveDualQueryStrategyVarUn	stream classification single-annotator	Liu et al.34
Density Based Active Learning for Data Streams	StreamDensityBasedAL	stream classification single-annotator	Ienco et al.35

Probalistic Sampling

Method	Base Class	Tags	Reference
Probabilistic Active Learning in Datastreams	StreamProbabilisticAL	stream classification single-annotator	Kottke et al.36

Random Sampling

Method	Base Class	Tags	Reference
Periodic Sampling	PeriodicSampling	stream classification single-annotator
Stream Random Sampling	StreamRandomSampling	stream classification single-annotator

Uncertainty Sampling

Method	Base Class	Tags	Reference
Fixed-Uncertainty	FixedUncertainty	stream classification single-annotator	Žliobaitė et al.37
Randomized-Variable-Uncertainty	RandomVariableUncertainty	stream classification single-annotator	Žliobaitė et al.37
Split	Split	stream classification single-annotator	Žliobaitė et al.37
Variable-Uncertainty	VariableUncertainty	stream classification single-annotator	Žliobaitė et al.37

References

1

Christoph Käding, Erik Rodner, Alexander Freytag, Oliver Mothes, Björn Barz, Joachim Denzler, and Carl Zeiss AG. Active learning for regression tasks with expected model output changes. In BMVC, 103. 2018.

2

David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129–145, 1996.

3

Dina Elreedy, Amir F. Atiya, and Samir I. Shaheen. A novel active learning regression framework for balancing the exploration-exploitation trade-off. Entropy, 21(7):651, 2019.

4(1,2)

Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estimation of error reduction. In Proceedings of the International Conference on Machine Learning, 441–448. 2001.

5

Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In Proceedings of the Conference on Computer Vision and Pattern Recognition, 2372–2379. IEEE, 2009.

6

Dragos D Margineantu. Active cost-sensitive learning. In Proceedings of the International Joint Conference on Artificial Intelligence, volume 5, 1622–1623. 2005.

7

Ashish Kapoor, Eric Horvitz, and Sumit Basu. Selective supervision: guiding supervised learning with decision-theoretic active learning. In IJCAI, volume 7, 877–882. 2007.

8

Wenbin Cai, Ya Zhang, and Jun Zhou. Maximizing expected model change for active learning in regression. In 2013 IEEE 13th international conference on data mining, 51–60. IEEE, 2013.

9(1,2)

Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011.

10

Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: efficient and diverse batch acquisition for deep bayesian active learning. Advances in Neural Information Processing Systems, 2019.

11(1,2,3,4)

H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the Annual Workshop on Computational Learning Theory, 287–294. ACM, 1992.

12

Andrew Kachites McCallum and Kamal Nigamy. Employing em and pool-based active learning for text classification. In Proceedings of the International Conference on Machine Learning, 359–367. 1998.

13

Sean P Engelson and Ido Dagan. Minimizing Manual Annotation Cost in Supervised Training from Corpora. In Annual Meeting of the Association for Computational Linguistics, 319–326. 1996.

14

William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The Power of Ensembles for Active Learning in Image Classification. In IEEE Conference on Computer Vision and Pattern Recognition, 9368–9377. 2018.

15

Robert Burbidge, Jem J Rowland, and Ross D King. Active learning for regression based on query by committee. In Internation Conference on Intelligent Data Engineering and Automated Learning, 209–218. 2007.

16

Viraj Prabhu, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman. Active Domain Adaptation via Clustering Uncertainty-weighted Embeddings. In ICCV, 8505–8514. 2021.

17

Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. Active Learning by Acquiring Contrastive Examples. In Conference on Empirical Methods in Natural Language Processing, 650–663. 2021.

18

Vu-Linh Nguyen, Sébastien Destercke, and Eyke Hüllermeier. Epistemic uncertainty sampling. In Proceedings of the International Conference on Discovery Science, 72–86. Springer, 2019.

19(1,2)

Tobias Reitmaier and Bernhard Sick. Let us know your decision: pool-based active training of a generative classifier with the selection strategy 4ds. Information Sciences, 230:106–131, 2013.

20(1,2,3)

Burr Settles. Active learning literature survey. Technical Report 1648, University of Wisconsin, Department of Computer Science, 2009.

21

Hanmo Wang, Xiaojun Chang, Lei Shi, Yi Yang, and Yi-Dong Shen. Uncertainty sampling for action recognition via maximizing expected average precision. In Proceedings of the International Joint Conference on Artificial Intelligence. 2018.

22(1,2)

Pinar Donmez, Jaime G Carbonell, and Paul N Bennett. Dual strategy active learning. In European Conference on Machine Learning, 116–127. Springer, 2007.

23(1,2)

Hieu T Nguyen and Arnold Smeulders. Active learning using pre-clustering. In International Conference on Machine Learning, 79. 2004.

24

Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. In International Conference on Learning Representations. 2020.

25

Ozan Sener and Silvio Savarese. Active Learning for Convolutional Neural Networks: A Core-set Approach. In Internation Conference on Learning Representations. 2017.

26

Kuan-hao Huang and Hsuan-tien Lin. A novel uncertainty sampling algorithm for cost-sensitive multiclass active learning. In Proceedings of the International Conference on Data Mining, 925–930. IEEE, 2016.

27

Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv, 2019.

28(1,2,3)

Dongrui Wu, Chin-Teng Lin, and Jian Huang. Active learning for regression using greedy sampling. Information Sciences, 474:90–105, 2019.

29

Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. Active Learning Through a Covering Lens. In Advances in Neural Information Processing Systems. 2022.

30

Daniel Kottke, Georg Krempl, Dominik Lang, Johannes Teschner, and Myra Spiliopoulou. Multi-class probabilistic active learning. In Proceedings of the European Conference on Artificial Intelligence, volume 285, 586–594. IOS Press, 2016.

31

Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. Active learning by querying informative and representative examples. Advances in neural information processing systems, 2010.

32(1,2,3)

Ashna Jose, João Paulo Almeida de Mendonça, Emilie Devijver, Noël Jakse, Valérie Monbet, and Roberta Poloni. Regression tree-based active learning. Data Mining and Knowledge Discovery, pages 420–460, 2023.

33

Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets. In International Conference on Machine Learning. 2022.

34(1,2,3,4)

Sanmin Liu, Shan Xue, Jia Wu, Chuan Zhou, Jian Yang, Zhao Li, and Jie Cao. Online active learning for drifting data streams. IEEE Transactions on Neural Networks and Learning Systems, 34(1):186–200, 2023.

35

Dino Ienco, Indrė Žliobaitė, and Bernhard Pfahringer. High density-focused uncertainty sampling for active learning over evolving stream data. In Proceedings of the 3rd International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications, volume 36 of Proceedings of Machine Learning Research, 133–148. New York, New York, USA, 24 Aug 2014. PMLR.

36

Daniel Kottke, Georg Krempl, and Myra Spiliopoulou. Probabilistic active learning in datastreams. In Advances in Intelligent Data Analysis XIV, 145–157. Cham, 2015. Springer International Publishing.

37(1,2,3,4)

Indrė Žliobaitė, Albert Bifet, Bernhard Pfahringer, and Geoffrey Holmes. Active learning with drifting streaming data. IEEE Transactions on Neural Networks and Learning Systems, 25(1):27–39, 2014.