Frequently, Machine Learning (ML) algorithms are trained on human-labeled data. Although often seen as a “gold standard,” human labeling is all but error free. Decisions in the design of labeling tasks can lead to distortions of the resulting labeled data and impact predictions. Building on insights from survey methodology, a field that studies the impact of instrument design on survey data and estimates, we examine how the structure of a hate speech labeling task affects which labels are assigned. We also examine what effect task ordering has on the perception of hate speech and what role background characteristics of annotators have on classifications provided by annotators. The study demonstrates the importance of applying design thinking at the earliest steps of ML product development. Design principles such as quick prototyping and critically assessing user interfaces are not only important in interaction with end users of an artificial intelligence (AI)-driven products, but are crucial early in development, prior to training AI algorithms.
Improving labeling through social science insights
Results and research agenda
Beck, J., Eckman, S. A., Chew, R., & Kreuter, F. (2022). Improving labeling through social science insights: Results and research agenda. In J. Y. C. Chen, G. Fragomeni, H. Degen, & S. Ntoa (Eds.), HCI International 2022 – Late Breaking Papers: Interacting with eXtended Reality and Artificial Intelligence (Vol. 13518, pp. 233-244). Springer, Cham. Lecture Notes in Computer Science Vol. 13518 https://doi.org/10.1007/978-3-031-21707-4_18
To contact an RTI author, request a report, or for additional information about publications by our experts, send us your request.
Multifaceted risk for non-suicidal self-injury only versus suicide attempt in a population-based cohort of adults
Long-term effects of a diet supplement containing Cannabis sativa oil and Boswellia serrata in dogs with osteoarthritis following physiotherapy treatments
Epigenetic biomarkers for smoking cessation