The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

Paradigm framing

The preprint focuses on the paradigm shift in data labeling from the traditional “ground truth” paradigm to the “perspectivist” paradigm, particularly within the field of Natural Language Processing.

Highlights

The preprint argues for a “perspectivist turn” in machine learning, challenging the established paradigm that treats annotator disagreement as noise. This new perspective views disagreement as a valuable source of information, reflecting diverse viewpoints and lived experiences. While this represents a potential paradigm shift, the field hasn’t fully transitioned. There’s ongoing debate and exploration of how to best incorporate this new perspective into data collection, model training, and evaluation methods. Therefore, the current state appears to be a drift from the existing model, potentially leading towards a crisis if the challenges and limitations outlined in the preprint are not addressed. It might be too early to definitively declare a “model revolution” or “paradigm change” as the field is still grappling with the practical and normative implications of this shift.