Contestable AI: designing responsible decision-making systems

'Contestable AI' explores algorithmic decision-making systems that are open and responsive to dispute.

Artificial intelligence is increasingly used to automate and support public sector decision-making. A known risk of public AI systems is that they can violate people’s fundamental human rights to autonomy and dignity. To address this issue, TU Delft researcher Kars Alfrink, with support from Ianus Keller, Neelke Doorn, and Gerd Kortuem, is researching contestable AI.

Framework

Contestable AI can be described as algorithmic decision-making systems that are open and responsive to dispute throughout their entire lifecycle. Recently, Kars has developed a provisional design framework for contestable AI. This was published at Minds & Machines. The next step was to create a compelling example of contestable AI and to evaluate the result with people who work with AI in the public sector.

Concept video

With the help of Responsible Sensing Lab and boutique animation studio Trim Trab, Kars has created a speculative, provocative concept video that applies the Contestable AI by Design framework to the phenomenon of cities using “scan cars” for various forms of urban sensing.

Video Contestable Camera Cars

Concept design by Kars Alfrink / TU Delft contestable.ai

Animation by Trim Tab trimtab.studio

Music and sound by Six Feet High sixfeethigh.nl

Produced by Responsible Sensing Lab

The primary purpose of the concept video is to make concrete some of the contestability mechanisms that Kars proposes in his framework. It shows ways for citizens to understand a system, channels for them to voice their opinion, and arenas for dialogue and debate between citizens and the city. The video also shows some potential harms of camera car system and how contestations from citizens may lead to system improvements.

The video serves as a source of inspiration for designers, engineers, and policy-makers who are working to make public AI more human-centered.

Research

Kars has used the video to interview employees of the City of Amsterdam who work with AI about the main challenges facing the implementation of contestability in current and future municipal AI systems. Based on an analysis of the interviews, Kars claims that: (1) current forms of citizen participation struggle to be inclusive, (2) public AI systems should be better embedded in local representative democracies, and (3) cities need more internal capacities to be appropriately accountable to citizens.

The resulting paper titled "Contestable Camera Cars: A Speculative Design Exploration of Public AI That Is Open and Responsive to Dispute" will be presented at CHI 2023. A preprint is available at arXiv.

Learn more about Kars' PhD research at www.contestable.ai.

Disclaimer: The camera car application the video depicts is a speculation on the part of Kars and the research team and does not reflect actual plans of the city.