Mauricio Mazuecos author Alberto Testoni author Raffaella Bernardi author Luciana Benotti author 2020-jul text Association for Computational Linguistics Online conference publication Task success is the standard metric used to evaluate referential visual dialogue systems. In this paper we propose two new metrics that evaluate how each question contributes to the goal. First, we measure how effective each question is by evaluating whether the question discards objects that are not the referent. Second, we define referring questions as those that univocally identify one object in the image. We report the new metrics for human dialogues and for state of the art publicly available models on GuessWhat?!. Regarding our first metric, we find that successful dialogues do not have a higher percentage of effective questions for most models. With respect to the second metric, humans make questions at the end of the dialogue that are referring, confirming their guess before guessing. Human dialogues that use this strategy have a higher task success but models do not seem to learn it. mazuecos-etal-2020-role https://www.aclweb.org/anthology/2020.alvr-1.4 2020-jul 19 25