Open the black box: data-driven explanation of black box decision making

---

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Luca Pappalardo, Salvatore Ruggieri and Franco Turini

University of Pisa and Italian National Research Council, Italy

: J Comput Eng Inf Technol

Abstract

---

The last decade has witnessed the rise of a black box society, obscure or secret algorithms often based on sophisticated machine learning models that predict behavioural traits of individuals, such as credit score, insurance risk, health status, interest categories–on the basis of personal data disseminated in the digital ecosystem. Black box map users feature into a class or a score without exposing the reasons why. This is worrisome, not only for lack of transparency, but also for possible biases hidden in the algorithms. Machine learning constructs predictive models and decision making systems based on (big) data, i.e., the digital traces of human activities–opinions, movements, lifestyles, etc. that obviously reflect human biases and prejudices. Therefore, the models learnt from training data may inherit such biases, possibly leading to unfair or wrong decisions. Many controversial cases warn us that delegating decision-making to black box algorithms is critical in many sensitive domains, including crime prediction, personality scoring, image recognition, etc. (not to mention the possibility of fraudulent rules maliciously introduced by humans.) Recently, the European Parliament adopted the general data protection regulation (GDPR) to become the law for all member states in May 2018. An innovative, sharply debated aspect of the GDPR is the provision on automated (algorithmic) individual decision-making, including profiling, which for the first time, introduces to some extent, a right of explanation for all individuals to obtain “meaningful explanations of the logic involved” when automated decision making takes place. Despite the doubts among law scholars on the real scope of this provision, the need for such a principle is becoming clearer, together with the awareness that its implementation represents a big scientific challenge. Without an enabling technology, capable of explaining the logic of black boxes, the right of explanation will remain “dead letter”, and we risk creating and using automated decision systems we don’t really understand. This talk provides a perspective on the many open questions and the relatively little advances achieved so far along the quest to providing human-comprehensible explanations of black box decision models. To this purpose, we leverage our experience on discrimination-aware data mining. Our insight was to adopt data mining itself to discover the discriminatory rules hidden in the historical records of human or algorithmic decision-making. More ambitiously, today the challenge is to reveal the rules and biases hidden into a black box, allowing humans understand and validate the logic of the associated decision model.

Biography

---