No Agreement Without Loss: Learning and Social Choice in Peer Review

Simple item page
dc.catalogadorjwg
dc.contributor.authorBarcelo Baeza, Pablo
dc.contributor.authorDuarte, Mauricio
dc.contributor.authorRojas González, Luis Cristóbal
dc.contributor.authorSteifer, Tomasz
dc.date.accessioned2023-10-12T13:24:12Z
dc.date.available2023-10-12T13:24:12Z
dc.date.issued2023
dc.description.abstractIn peer review systems, reviewers are often asked toevaluate various features of submissions, such as technical qualityor novelty. A score is given to each of the predefined features andbased on these the reviewer has to provide an overall quantitativerecommendation. It may be assumed that each reviewer has her ownmapping from the set of features to a recommendation, and thatdifferent reviewers have different mappings in mind. This introducesan element of arbitrariness known as commensuration bias. In thispaper we discuss a framework, introduced by Noothigattu, Shah andProcaccia, and then applied by the organizers of the AAAI 2022conference. Noothigattu, Shah and Procaccia proposed to aggregatereviewer’s mapping by minimizing certain loss functions, and studiedaxiomatic properties of this approach, in the sense of social choicetheory. We challenge several of the results and assumptions used intheir work and report a number of negative results. On the one hand,we study a trade-off between some of the axioms proposed and theability of the method to properly capture agreements of the majorityof reviewers. On the other hand, we show that dropping a certainunrealistic assumption has dramatic effects, including causing themethod to be discontinuous.
dc.fechaingreso.objetodigital2023-10-12
dc.fuente.origenORCID
dc.identifier.doi10.3233/FAIA230270
dc.identifier.urihttps://doi.org/10.3233/FAIA230270
dc.identifier.urihttps://repositorio.uc.cl/handle/11534/75098
dc.information.autorucInstituto de Ingeniería Matemática y Computacional; Barcelo Baeza Pablo; ; 13516
dc.information.autorucInstituto de Ingeniería Matemática y Computacional; Rojas Gonzalez Luis Cristobal; ; 1182858
dc.language.isoen
dc.nota.accesoContenido completo
dc.pagina.final197
dc.pagina.inicio190
dc.rightsacceso abierto
dc.subject.ddc001.42
dc.subject.deweyCiencias de la computaciónes_ES
dc.titleNo Agreement Without Loss: Learning and Social Choice in Peer Review
dc.typecapítulo de libro
sipa.codpersvinculados13516
sipa.codpersvinculados1182858
sipa.trazabilidadORCID;2023-10-09
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
FAIA-372-FAIA230270.pdf
Size:
303.34 KB
Format:
Adobe Portable Document Format
Description:
Download
Collections
Capítulos de libros