Foundations and Trends® in Computer Graphics and Vision > Vol 10 > Issue 3

Crowdsourcing in Computer Vision

By Adriana Kovashka, University of Pittsburgh, USA, kovashka@cs.pitt.edu | Olga Russakovsky, Carnegie Mellon University, USA, olgarus@cmu.edu | Li Fei-Fei, Stanford University, USA, feifeili@cs.stanford.edu | Kristen Grauman, University of Texas at Austin, USA, grauman@cs.utexas.edu

 
Suggested Citation
Adriana Kovashka, Olga Russakovsky, Li Fei-Fei and Kristen Grauman (2016), "Crowdsourcing in Computer Vision", Foundations and TrendsĀ® in Computer Graphics and Vision: Vol. 10: No. 3, pp 177-243. http://dx.doi.org/10.1561/0600000071

Publication Date: 30 Nov 2016
© 2016 A. Kovashka, O. Russakovsky, L. Fei-Fei and K. Grauman
 
Subjects
Interdisciplinary influence: Artificial intelligence and the user interface,  Object and scene recognition
 

Free Preview:

Download extract

Share

Download article
In this article:
1. Introduction
2. What annotations to collect
3. How to collect annotations
4. Which data to annotate
5. Conclusions
References

Abstract

Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.

DOI:10.1561/0600000071
ISBN: 978-1-68083-212-9
80 pp. $65.00
Buy book (pb)
 
ISBN: 978-1-68083-213-6
80 pp. $140.00
Buy E-book (.pdf)
Table of contents:
1. Introduction
2. What annotations to collect
3. How to collect annotations
4. Which data to annotate
5. Conclusions
References

Crowdsourcing in Computer Vision

Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks.

Crowdsourcing in Computer Vision describes the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. It begins by discussing data collection on both classic vision tasks, such as object recognition, and recent vision tasks, such as visual story-telling. It then summarizes key design decisions for creating effective data collection interfaces and workflows, and presents strategies for intelligently selecting the most important data instances to annotate. It concludes with some thoughts on the future of crowdsourcing in computer vision.

Crowdsourcing in Computer Vision provides an overview of how crowdsourcing has been used in computer vision, enabling a computer vision researcher who has previously not collected non-expert data to devise a data collection strategy. It will also be of help to researchers who focus broadly on crowdsourcing to examine how the latter has been applied in computer vision, and to improve the methods that can be employed to ensure the quality and expedience of data collection.

 
CGV-071