About The Forum
Calendar 2017 Poster Sessions : Pairwise Comparisons for Online Reputation Systems
Abstract:
The modern online marketplace relies on reputation systems to identify poor performers and reward excellent ones. However, these systems are often plagued with problems stemming from pressure and various cognitive biases, leading to reputation inflation and noisy ratings. In this work, we argue that marketplaces should adopt a pairwise-comparison based reputation system. Such systems better extract marginal information from each comparison and may be more resistant to cognitive biases. We show that pairwise aggregation methods perform well under the characteristics of a standard online platform: a dynamic market composition; a feedback loop in which ratings impact future opportunities and comparisons; and biased ratings, such as recency bias or other biases common in reputation systems. After characterizing the conditions under which the joint distribution of true and estimated ratings converges to a steady state, we show that rankings from pairwise comparison systems correlate better with true rankings. Finally, we present simulation results and leverage implicit comparisons in three real world platforms: movie ratings in the MovieLens dataset, course decisions made by undergraduates at a large research institution, and interview and hiring decisions by employers on a popular online freelancing platform. We show that the ratings distribution converge, that pairwise comparisons are more predictive of future ratings, and that various comparisons aggregation methods produce similar rankings.
Bio:
Nikhil Garg is a 2nd year PhD student in Electrical Engineering at Stanford University, working with Ashish Goel and Ramesh Johari. He received a B.S. and B.A. from The University of Texas at Austin.
The modern online marketplace relies on reputation systems to identify poor performers and reward excellent ones. However, these systems are often plagued with problems stemming from pressure and various cognitive biases, leading to reputation inflation and noisy ratings. In this work, we argue that marketplaces should adopt a pairwise-comparison based reputation system. Such systems better extract marginal information from each comparison and may be more resistant to cognitive biases. We show that pairwise aggregation methods perform well under the characteristics of a standard online platform: a dynamic market composition; a feedback loop in which ratings impact future opportunities and comparisons; and biased ratings, such as recency bias or other biases common in reputation systems. After characterizing the conditions under which the joint distribution of true and estimated ratings converges to a steady state, we show that rankings from pairwise comparison systems correlate better with true rankings. Finally, we present simulation results and leverage implicit comparisons in three real world platforms: movie ratings in the MovieLens dataset, course decisions made by undergraduates at a large research institution, and interview and hiring decisions by employers on a popular online freelancing platform. We show that the ratings distribution converge, that pairwise comparisons are more predictive of future ratings, and that various comparisons aggregation methods produce similar rankings.
Bio:
Nikhil Garg is a 2nd year PhD student in Electrical Engineering at Stanford University, working with Ashish Goel and Ramesh Johari. He received a B.S. and B.A. from The University of Texas at Austin.