Things are just ramping up at this week’s ACL conference (Association for Computational Linguistics) in Dublin, so we wanted to highlight five intriguing papers on responsible AI that quickly became our favourites.

Responsible AI continues to be top-of-mind for researchers in the NLP field. As much as ground may have been gained in this area, there is still a lot of consideration that needs to occur for researchers to address this burgeoning challenge.

As Nick Frosst, co-founder of Cohere, told us in a recent blog post: “There’s no silver bullet for this, even though there’s a lot of people working on it. Languages continuously change – they’re living things – so there will never be a complete lock on this.”

The pursuit of responsible AI will consist of cross-functional team collaboration, new tools and processes, and continuous support from key stakeholders. At Borealis AI, we regard frontline research reports to be vital in creating a forward-thinking healthy ecosystem for both creators and consumers of AI technology.

Our Top 5 Responsible AI ACL 2022 Papers:

Mitigating Gender Bias in Machine Translation through Adversarial Learning

Anonymous ACL submission

Improving language translation is at the heart of this report, which presented an adversarial learning framework to address challenges related to biases of gender and race that perpetuate stereotypes. The paper’s approach improved the disparity in translation quality for sentences with male vs. female entities by 86% for English-German translation and by 91% for English-French translation.

On The Robustness of Offensive Language Classifiers

Jonathan Rusert, Zubair Shafiq, Padmini Srinivasan

As more social media platforms unleash ML-based offensive language classification systems to combat hateful and racist speech at scale, how robust can they be against adversarial and intentional attacks (as opposed to misspellings resulting in offensive terms)? This paper’s authors analyzed the efficacy of these state-of-the-art offensive language classifiers against more crafty adversarial attacks and found that these barrages can weaken the accuracy of offensive language classifiers by more than 50% while also being able to preserve the readability and meaning of the modified text.

Bias Mitigation in Machine Translation Quality Estimation

Hanna Behnke, Marina Fomicheva, Lucia Specia

Input bias is in the crosshairs of this paper on how machine translation quality estimation (QE) builds predictive models. While high-quality QE models have been shown to achieve impressive results, they often over-rely on features that don’t directly impact translation efficiency, such as a tendency to give high scores to translations that are grammatically correct even if they don’t preserve the meaning of the source. The researchers found that training a multitask architecture with an auxiliary binary classification task that employs additional augmented data best achieves the desired effects and scored well with different languages and quality metrics.

Ethics Sheets for AI Tasks

Saif M. Mohammad

In this position paper, the researcher contends how important it is for AI practitioners to recognize the ethical considerations of their AI systems not just at the level of individual models and datasets, but also at the level of AI tasks. He laid out a new form of such an effort, Ethics Sheets for AI Tasks, committed to fleshing out the assumptions and ethical considerations hidden in how a task is commonly framed and in the choices made regarding the data, method, and evaluation. He also presented a template for ethics sheets with 50 ethical considerations, using the task of emotion recognition as an example. Ethics sheets act as a means to engage with and document ethical considerations before building datasets and systems.

FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing

Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Felix Schwemer, Anders Søgaard

Covering four jurisdictions (European Council, USA, Switzerland, and China) and five languages (English, German, French, Italian and Chinese), researchers presented a benchmark suite of four datasets for assessing the fairness of pre-trained language models and the techniques used to fine-tune them for downstream tasks. In these experiments, they evaluated pre-trained language models using several group-robust fine-tuning techniques and reveal that performance group disparities are active in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Also, they offer a quantitative and qualitative analysis of their results, highlighting open challenges in the development of robustness methods in legal NLP.