A Human-in-the-loop Workflow for Multi-Factorial Sensitivity Analysis of Algorithmic Rankers

Document Type

Conference Proceeding

Publication Date

6-18-2023

Abstract

Algorithmic rankers are ubiquitously applied in automated decision systems such as hiring, admission, and loan-approval systems. Without appropriate explanations, decision-makers often cannot audit or trust algorithmic rankers' outcomes. In recent years, XAI (explainable AI) methods have focused on classification models, but there for algorithmic rankers, we are yet to develop state-of-the-art explanation methods. Moreover, explanations are also sensitive to changes in data and ranker properties, and decision-makers need transparent model diagnostics for calibrating the degree and impact of ranker sensitivity. To fulfill these needs, we take a dual approach of: i) designing explanations by transforming Shapley values for the simple form of a ranker based on linear weighted summation and ii) designing a human-in-the-loop sensitivity analysis workflow by simulating data whose attributes follow user-specified statistical distributions and correlations. We leverage a visualization interface to validate the transformed Shapley values and draw inferences from them by leveraging multi-factorial simulations, including data distributions, ranker parameters, and rank ranges.

Identifier

85167945983 (Scopus)

ISBN

[9798400702167]

Publication Title

Hilda 2023 Workshop on Human in the Loop Data Analytics Co Located with SIGMOD 2023

External Full Text Location

https://doi.org/10.1145/3597465.3605221

Grant

1928627

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS