The browser you are using is not supported by this website. All versions of Internet Explorer are no longer supported, either by us or Microsoft (read more here: https://www.microsoft.com/en-us/microsoft-365/windows/end-of-ie-support).

Please use a modern browser to fully experience our website, such as the newest versions of Edge, Chrome, Firefox or Safari etc.

Portrait of Isak Engdahl. Photo: Jakob Roséen.

Isak Engdahl

Doctoral student

Portrait of Isak Engdahl. Photo: Jakob Roséen.

Agreements ‘in the wild’ : Standards and alignment in machine learning benchmark dataset construction

Author

  • Isak Engdahl

Summary, in English

This article presents an ethnographic case study of a corporate-academic group constructing a benchmark dataset of daily activities for a variety of machine learning and computer vision tasks. Using a socio-technical perspective, the article conceptualizes the dataset as a knowledge object that is stabilized by both practical standards (for daily activities, datafication, annotation and benchmarks) and alignment work – that is, efforts including forging agreements to make these standards effective in practice. By attending to alignment work, the article highlights the informal, communicative and supportive efforts that underlie the success of standards and the smoothing of tensions between actors and factors. Emphasizing these efforts constitutes a contribution in several ways. This article's ethnographic mode of analysis challenges and supplements quantitative metrics on datasets. It advances the field of dataset analysis by offering a detailed empirical examination of the development of a new benchmark dataset as a collective accomplishment. By showing the importance of alignment efforts and their close ties to standards and their limitations, it adds to our understanding of how machine learning datasets are built. And, most importantly, it calls into question a key characterization of the dataset: that it captures unscripted activities occurring naturally ‘in the wild’, as alignment work bleeds into moments of data capture.

Department/s

  • Sociology

Publishing year

2024-04-01

Language

English

Publication/Series

Big Data and Society

Volume

11

Issue

2

Document type

Journal article

Publisher

SAGE Publications

Topic

  • Information Systems, Social aspects

Keywords

  • alignment work
  • benchmark
  • dataset analysis
  • Ethnography of machine learning
  • in-the-wild
  • standards

Status

Published

ISBN/ISSN/Other

  • ISSN: 2053-9517