Syllabus
WS 22/23
Instructor: Mareike Hartmann (mareikeh@lst.uni-saarland.de)
Meetings: Thursdays 14:15 - 15:45, C7.1 Room U15
Office Hours: by appointment
Please sign up for the MS team
Cross-lingual learning is a form of transfer learning, and refers to methods which learn from a task in one language and transfer this knowledge to a task in another language, which is particularly helpful to solve tasks in languages with few training resources. In this seminar, we will discuss different methods for cross-lingual learning, focusing on transfer via multilingual word embeddings and pre-trained multilingual language models, and cover recent applications for cross-lingual transfer.
Prerequisites: Background in natural language processing and deep learning is required.
Topics and papers to be discussed:\
| Topic | Readings |
| Multilingual word embeddings | Mikolov et al. (2013b): Exploiting Similarities among Languages for Machine Translation Faruqui and Dyer (2014b): Improving Vector Space Word Representations Using Multilingual Correlation Mrkšić et al. (2017): Semantic Specialization of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints Additional background material: Ruder et al. (2019): A Survey of Cross-lingual Word Embedding Models |
| Bilingual dictionary induction: Inducing word-to-word translations from comparable data | Bergsma and van Durme (2011): Learning Bilingual Lexicons Using the Visual Similarity of Labeled Web Images Kiela et al. (2015): Visual Bilingual Lexicon Induction with Transferred ConvNet Features Rapp (1999): Automatic Identification of Word Translations from Unrelated English and German Irvine and Callison-Birch (2017): A Comprehensive Analysis of Bilingual Lexicon Induction |
| Multilingual word embeddings: Unsupervised approache | Conneau et al. (2018): Word translation without parallel data Artetxe et al. (2018): A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings Søgaard et al. (2019): On the Limitations of Unsupervised Bilingual Dictionary Induction |
| Evaluating multilingual word embeddings | Upadhyay et al. (2016): Cross-lingual Models of Word Embeddings: An Empirical Comparison Glavaš et al. (2019): How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions](https://aclanthology.org/P19-1070.pdf) |
| Transformer-based multilingual language models | Conneau and Lample (2019): Cross-lingual Language Model Pretraining Conneau et al. (2020): Unsupervised Cross-lingual Representation Learning at Scale |
| Cross-lingual effectiveness of multilingual language models | Wu and Dredze (2019): Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT Pires et al. (2019): How Multilingual is Multilingual BERT? |
| Essential elements for multilinguality | Karthikeyan et al. (2020): Cross-Lingual Ability of Multilingual BERT: An Empirical Study Dufter and Schütze (2020): Identifying Elements Essential for BERT’s Multilinguality Conneau et al. (2020): Emerging Cross-lingual Structure in Pretrained Language Models Muller et al. (2021): First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT Lin et al. (2019): Choosing Transfer Languages for Cross-Lingual Learning |
| Seq2seq multilingual language models | Xue et al. (2021): mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer Chi et al. (2021): mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs |
| Limitations of multilingual language models | Lauscher et al. (2020): From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers Wu and Dredze (2020): Are All Languages Created Equal in Multilingual BERT? |
| Benchmarks for evaluating cross-lingual transfer | Hu et al. (2020): XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation Liang et al. (2020): XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation Ruder et al. (2021): XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation |
| Applications for cross-lingual transfer learning | de Vries et al (2022): Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages Riabi et al. (2021): Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering Perez-Beltrachini and Lapata (2021): Models and Datasets for Cross-Lingual Summarisation Kondratyuk and Straka (2019): 75 Languages, 1 Model: Parsing Universal Dependencies Universally Müller-Eberstein et al. (2021): Genre as Weak Supervision for Cross-lingual Dependency Parsing Zhang et al. (2021): On the Benefit of Syntactic Supervision for Cross-lingual Transfer in Semantic Role Labeling Chen et al. (2021): Zero-Shot Cross-Lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders |