Strategy Overview

This is an overview of all implemented active learning strategies.

You can use the following checkboxes to filter the tables below.

Regression Classification Multi-Annotator Single-Annotator

Pool

Expected Error Reduction

Method	Base Class	Tags	Reference
Expected Model Output Change	ExpectedModelOutputChange	pool regression single-annotator	Käding et al.1
Expected Model Variance Reduction	ExpectedModelVarianceReduction	pool regression single-annotator	Cohn et al.2
Regression based Kullback Leibler Divergence Maximization	KLDivergenceMaximization	pool regression single-annotator	Elreedy et al.3
Monte-Carlo EER with Log-Loss	MonteCarloEER	pool classification single-annotator	Roy and McCallum4
Monte-Carlo EER with Misclassification-Loss	MonteCarloEER	pool classification single-annotator	Roy and McCallum4
Value of Information on Unlabeled Samples	ValueOfInformationEER	pool classification single-annotator	Joshi et al.5
Value of Information on Labeled Samples	ValueOfInformationEER	pool classification single-annotator	Margineantu6
Value of Information (VOI)	ValueOfInformationEER	pool classification single-annotator	Kapoor et al.7

Model Change

Method	Base Class	Tags	Reference
Expected Model Change	ExpectedModelChangeMaximization	pool regression single-annotator	Cai et al.8

Query-by-Committee

Method	Base Class	Tags	Reference
Bayesian Active Learning by Disagreement (BALD)	BatchBALD	pool classification single-annotator	Houlsby et al.9
Batch Bayesian Active Learning by Disagreement	BatchBALD	pool classification single-annotator	Houlsby et al.9, Kirsch et al.10
Query-by-Committee with Kullback-Leibler Divergence	QueryByCommittee	pool classification regression single-annotator	Seung et al.11, McCallum and Nigamy12

Random Sampling

Method	Base Class	Tags	Reference
Random Sampling	RandomSampling	pool regression classification single-annotator

Uncertainty Sampling

Method	Base Class	Tags	Reference
Epistemic Uncertainty Sampling	EpistemicUncertaintySampling	pool classification single-annotator	Nguyen et al.13
Density-Diversity-Distribution-Distance Sampling (4DS)	FourDs	pool classification single-annotator	Reitmaier and Sick14
Batch Density-Diversity-Distribution-Distance Sampling (Batch4DS)	FourDs	pool classification single-annotator	Reitmaier and Sick14
Uncertainty Sampling with Margin	UncertaintySampling	pool classification single-annotator	Settles15
Uncertainty Sampling with Least-Confidence	UncertaintySampling	pool classification single-annotator	Settles15
Uncertainty Sampling with Entropy	UncertaintySampling	pool classification single-annotator	Settles15
Expected Average Precision	UncertaintySampling	pool classification single-annotator	Wang et al.16
Density-weighted Uncertainty Sampling	UncertaintySampling	pool classification single-annotator	Donmez et al.17, Nguyen and Smeulders18
Dual strategy for Active Learning	UncertaintySampling	pool classification single-annotator	Donmez et al.17, Nguyen and Smeulders18

Others

Method	Base Class	Tags	Reference
Active Learning with Cost Embedding (ALCE)	CostEmbeddingAL	pool classification single-annotator	Huang and Lin19
Discriminative Active Learning (DAL)	DiscriminativeAL	pool classification regression single-annotator	Gissin and Shalev-Shwartz20
Greedy Sampling on the Target Space (GSy)	GreedySamplingTarget	pool regression single-annotator	Wu et al.21
Improved Greedy Sampling (GSi)	GreedySamplingTarget	pool regression single-annotator	Wu et al.21
Greedy Sampling on the Feature Space (GSx)	GreedySamplingX	pool regression single-annotator	Wu et al.21
Multi-class Probabilistic Active Learning (McPAL)	ProbabilisticAL	pool classification single-annotator	Kottke et al.22
Query-by-Committee with Vote Entropy	QueryByCommittee	pool classification single-annotator	Seung et al.11, Engelson and Dagan23
Querying Informative and Representative Examples (QUIRE)	Quire	pool classification single-annotator	Huang et al.24

Stream

Density Sampling

Method	Base Class	Tags	Reference
Cognitive Dual-Query Strategy with Fixed-Uncertainty	CognitiveDualQueryStrategyFixUn	stream classification single-annotator	Liu et al.25
Cognitive Dual-Query Strategy with Random Sampling	CognitiveDualQueryStrategyRan	stream classification single-annotator	Liu et al.25
Cognitive Dual-Query Strategy with Randomized-Variable-Uncertainty	CognitiveDualQueryStrategyRanVarUn	stream classification single-annotator	Liu et al.25
Cognitive Dual-Query Strategy with Variable-Uncertainty	CognitiveDualQueryStrategyVarUn	stream classification single-annotator	Liu et al.25
Density Based Active Learning for Data Streams	StreamDensityBasedAL	stream classification single-annotator	Ienco et al.26

Probalistic Sampling

Method	Base Class	Tags	Reference
Probabilistic Active Learning in Datastreams	StreamProbabilisticAL	stream classification single-annotator	Kottke et al.27

Random Sampling

Method	Base Class	Tags	Reference
Periodic Sampling	PeriodicSampling	stream classification single-annotator
Stream Random Sampling	StreamRandomSampling	stream classification single-annotator

Uncertainty Sampling

Method	Base Class	Tags	Reference
Fixed-Uncertainty	FixedUncertainty	stream classification single-annotator	Žliobaitė et al.28
Randomized-Variable-Uncertainty	RandomVariableUncertainty	stream classification single-annotator	Žliobaitė et al.28
Split	Split	stream classification single-annotator	Žliobaitė et al.28
Variable-Uncertainty	VariableUncertainty	stream classification single-annotator	Žliobaitė et al.28

References

1

Christoph Käding, Erik Rodner, Alexander Freytag, Oliver Mothes, Björn Barz, Joachim Denzler, and Carl Zeiss AG. Active learning for regression tasks with expected model output changes. In BMVC, 103. 2018.

2

David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129–145, 1996.

3

Dina Elreedy, Amir F. Atiya, and Samir I. Shaheen. A novel active learning regression framework for balancing the exploration-exploitation trade-off. Entropy, 21(7):651, 2019.

4(1,2)

Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estimation of error reduction. In Proceedings of the International Conference on Machine Learning, 441–448. 2001.

5

Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In Proceedings of the Conference on Computer Vision and Pattern Recognition, 2372–2379. IEEE, 2009.

6

Dragos D Margineantu. Active cost-sensitive learning. In Proceedings of the International Joint Conference on Artificial Intelligence, volume 5, 1622–1623. 2005.

7

Ashish Kapoor, Eric Horvitz, and Sumit Basu. Selective supervision: guiding supervised learning with decision-theoretic active learning. In IJCAI, volume 7, 877–882. 2007.

8

Wenbin Cai, Ya Zhang, and Jun Zhou. Maximizing expected model change for active learning in regression. In 2013 IEEE 13th international conference on data mining, 51–60. IEEE, 2013.

9(1,2)

Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011.

10

Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: efficient and diverse batch acquisition for deep bayesian active learning. Advances in Neural Information Processing Systems, 2019.

11(1,2)

H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the Annual Workshop on Computational Learning Theory, 287–294. ACM, 1992.

12

Andrew Kachites McCallum and Kamal Nigamy. Employing em and pool-based active learning for text classification. In Proceedings of the International Conference on Machine Learning, 359–367. 1998.

13

Vu-Linh Nguyen, Sébastien Destercke, and Eyke Hüllermeier. Epistemic uncertainty sampling. In Proceedings of the International Conference on Discovery Science, 72–86. Springer, 2019.

14(1,2)

Tobias Reitmaier and Bernhard Sick. Let us know your decision: pool-based active training of a generative classifier with the selection strategy 4ds. Information Sciences, 230:106–131, 2013.

15(1,2,3)

Burr Settles. Active learning literature survey. Technical Report 1648, University of Wisconsin, Department of Computer Science, 2009.

16

Hanmo Wang, Xiaojun Chang, Lei Shi, Yi Yang, and Yi-Dong Shen. Uncertainty sampling for action recognition via maximizing expected average precision. In Proceedings of the International Joint Conference on Artificial Intelligence. 2018.

17(1,2)

Pinar Donmez, Jaime G Carbonell, and Paul N Bennett. Dual strategy active learning. In European Conference on Machine Learning, 116–127. Springer, 2007.

18(1,2)

Hieu T Nguyen and Arnold Smeulders. Active learning using pre-clustering. In International Conference on Machine Learning, 79. 2004.

19

Kuan-hao Huang and Hsuan-tien Lin. A novel uncertainty sampling algorithm for cost-sensitive multiclass active learning. In Proceedings of the International Conference on Data Mining, 925–930. IEEE, 2016.

20

Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv, 2019.

21(1,2,3)

Dongrui Wu, Chin-Teng Lin, and Jian Huang. Active learning for regression using greedy sampling. Information Sciences, 474:90–105, 2019.

22

Daniel Kottke, Georg Krempl, Dominik Lang, Johannes Teschner, and Myra Spiliopoulou. Multi-class probabilistic active learning. In Proceedings of the European Conference on Artificial Intelligence, volume 285, 586–594. IOS Press, 2016.

23

Sean P Engelson and Ido Dagan. Minimizing manual annotation cost in supervised training from corpora. arXiv, 1996.

24

Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. Active learning by querying informative and representative examples. Advances in neural information processing systems, 2010.

25(1,2,3,4)

Sanmin Liu, Shan Xue, Jia Wu, Chuan Zhou, Jian Yang, Zhao Li, and Jie Cao. Online active learning for drifting data streams. IEEE Transactions on Neural Networks and Learning Systems, 34(1):186–200, 2023.

26

Dino Ienco, Indrė Žliobaitė, and Bernhard Pfahringer. High density-focused uncertainty sampling for active learning over evolving stream data. In Proceedings of the 3rd International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications, volume 36 of Proceedings of Machine Learning Research, 133–148. New York, New York, USA, 24 Aug 2014. PMLR.

27

Daniel Kottke, Georg Krempl, and Myra Spiliopoulou. Probabilistic active learning in datastreams. In Advances in Intelligent Data Analysis XIV, 145–157. Cham, 2015. Springer International Publishing.

28(1,2,3,4)

Indrė Žliobaitė, Albert Bifet, Bernhard Pfahringer, and Geoffrey Holmes. Active learning with drifting streaming data. IEEE Transactions on Neural Networks and Learning Systems, 25(1):27–39, 2014.