Hanoz Bhathena author Angelica Willis author Nathan Dass author 2020-jul text Association for Computational Linguistics Online conference publication We evaluate the compositionality of general-purpose sentence encoders by proposing two different metrics to quantify compositional understanding capability of sentence encoders. We introduce a novel metric, Polarity Sensitivity Scoring (PSS), which utilizes sentiment perturbations as a proxy for measuring compositionality. We then compare results from PSS with those obtained via our proposed extension of a metric called Tree Reconstruction Error (TRE) (CITATION) where compositionality is evaluated by measuring how well a true representation producing model can be approximated by a model that explicitly combines representations of its primitives. bhathena-etal-2020-evaluating https://www.aclweb.org/anthology/2020.repl4nlp-1.22 2020-jul 185 193