Making NLP multilingual.

Given limited budget in low-resource settings, which samples should we curate for a training or evaluation dataset? Is it beneficial to filter out unhelpful or noisy samples before multilingual training, and if so, what should be filtered out?

• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection

Language identification is important: we need it to know (among other things) what went in our web-scale data. But LID for anything beyond a few hundred of the highest-resourced languages can be ill-defined and painful, because languages are closely related, lack data, orthographic standardization, and because web data is noisy, non-standard, and / or language-mixed. How can we conceptualize meaningful taxonomies and build reliable systems for LID?

• LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages

  Introduce hierarchical model for resolving confusing cases, contribute LID corpus for 350+ languages.

Given imbalanced data sizes for various languages or tasks, how can we construct batches or manipulate training data constitution in order to achieve some goal over language performance (e.g. max-min, mean, minimum, equitable performance)?

• Balancing Training for Multilingual Neural Machine Translation

  Introduces differentiable data selection (DDS) for training which learns an optimal data sampling strategy.

These days, we’ve taken to translating English resources into other languages: low-resource problem solved. Unfortunately, life is not so easy. Even human-translated data has translation artifacts or “translationese”, and machine translated data has a host of other problems. But translation as a strategy is tempting and useful. Can we understand and try to quantify what can and can’t be achieved with translation-assisted resource building, the nature and distribution of problems, and alternate or composite strategies?

• On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?

  Discusses problems with using machine translated data for evaluation. Compares model results on human translated benchmarks versus MT.

Tokenization quality is an important factor in multilingual performance. However, it’s very expensive to experiment with tokenization, because you have to pretrain your entire model on top of your new tokenizer to see if it worked better. It would be nice if we understood properties of good tokenizer and had quickly measurable or intrinsic qualities of a tokenizer that correlated with final performance to help us iterate quickly.

• Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks

  Present tokenization parity and information parity potentially as intrinsic measures of tokenizer goodness, and measure impact on downsteam performance.

• Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings

  Look at language identity encoding, monolingual and cross-lingual semantics, and token embedding geometries.

Natural training corpora are extremely language-imbalanced. Tokenizers have a fixed vocabulary. How can we better construct the training corpus and vocabulary constitution of tokenizers?

• Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training

  While building the training corpus of the tokenizer, iteratively use average log probability over subwords of a monolingual corpus to choose a language for allocating the next k vocabulary budget in the shared multilingual tokenizer vocabulary.

Tokenizers are becoming uncool. Can we learn how to segment our text end-to-end?

• MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling

  Differentiable learned tokenizer which creates “blocks” of characters with some probability, with embeddings pooled over a block with CNNs.

• Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer

  Image-based tokenization!

How can we use linguistic knowledge to inspire tokenization? Should we try? Can we develop tokenization strategies that efficiently address the varying morphological typologies of different languages?

• KinyaBERT: a Morphology-aware Kinyarwanda Language Model

  Uses morphanalyzer and tagger and embeds morphemes. Not scalable, but fun to read.

Tokenization is unfair to low-resource languages. It chops them up, and that means that inference costs more, and that performance is worse. How can we address this problem? Can we in parallel develop fairer pricing systems, perhaps based on information content rather than number of tokens?

• Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages

  Using Huffman encoding to map a script into a different script in a lossless manner to enhance cross-lingual transfer. This idea has potential for an alternative to Unicode-based byte-level tokenization.

Can we introduce lightweight language modules to “adapt” a model to a particular language?

• Condensing Multilingual Knowledge with Lightweight Language-Specific Modules

  Low-rank approximations to language-specific modules to reduce the number of learned parameters, distillation method for condensing the above into a single shared module.

• MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer

  Generating language adapters for unseen languages based on their typology.

Mixture-of-experts models are useful for increasing modularity in LLMs and reducing inference costs while maintaining total capacity. How does this setup interact with multilingual settings and how can it be made to benefit these settings?

• Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering

  Analyze language routing behaviour in LLM mixture-of-experts models and propose routing-inspired steering method to boost multilingual performance.

Adding together the parameters of two fine-tuned models can give us a third model with the skills of both the original models. Crazy stuff. How can this idea be effectively leveraged for multilinguality, where we want to be good at many languages and many tasks?

• The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs

  Show that model parameters responsible for math and language skills are distinct, that these can be frozen while fine-tuning to facilitate compositionality, and that model merging is effective in this setting.

If LLMs are good at English, it might make sense to translate multilingual input to English, solve the task, and translate back. However, this is not optimal in other settings: e.g. inputs might require cultural knowledge or MT might introduce errors. Can we figure out when and how to effectively use translation in multilingual pipelines.

• Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning

  Learns when to invoke translation tool.

How do properties relevant to multilinguality evolve during pre-training? Can and should we intervene? If so,how?

• Tracing Multilingual Factual Knowledge Acquisition in Pretraining

  Look at checkpoints in OLMO, and analyze evolution of factual recall and cross-lingual consistency in 6 languages.

• Probing the Emergence of Cross-lingual Alignment during LLM Training

  Look at neuron overlap for processing different languages over pretraining.

• ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality

Good old next token prediction seems to be generally great, but it’s tempting to give models some bias for behaviours we would like, for example, cross-lingual alignment, language identity representation, and others. This used to be a lot more popular in the BERT era.

• Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment

  Multilingual contrastive learning.

Multilinguality can be framed as a multitask problem. We want to get better at many language-task combinations, and they may have good or bad synergy. How can better optimization algorithms help us understand or improve the multilingual performance landscape?

• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models

  Modify training gradients to boost positive synergy and reduce negative interference.

RL can help us when we know how to assign rewards to model outputs. Code is also a low-resource domain, but we are able to be good at it because we can construct verifiable rewards for it. Natural low-resource languages are harder - output quality is somewhat subjective, and it’s more difficult to build verifiable rewards for desired characteristics like fluency, naturalness, grammaticality, relevance, etc. Further, RL comes with its own problems, like reward hacking behaviour from models, and so we need to be careful in designing rewards for these output facets. But some of these are tractable, and there are huge rewards to be won in leveraging them.

• MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning

  Lexical and semantic rewards rewards for machine translation and multilinguality. Other work points out reward hacking with these kinds of ideas.

Instruction-tuned models are a slightly different beast from base pretrained models. Alignment or instruction-tuning datasets are expensive to collect and generally English-heavy. There’s evidence that this process actually makes naturally multilingual base models a little worse at multilinguality, or introduces potentially undesirable biases, for example, against dialectal output. It’s important to understand what happens when we align, and what we can do for preserving and enhancing multilingual performance after this step.

• Multilingual Instruction Tuning With Just a Pinch of Multilinguality

  Gains from a small set of multilingual instruction-responses.

As we try to “fit” more languages into a single model, performance on individual languages starts degrading. What drives this phenomenon, is massively multilinguality doomed, and what can we do about it?

• Causes and Cures for Interference in Multilingual Translation

  Shows the effect of data proportion on negative interference/

• There is No Theoretical Curse of Multilinguality for Embedding Space Structure

  Provides a theoretical and intrinsic perspective on the curse of multilinguality.

As for everything, data sparsity means bad quality, with the consequences that multilingual representation spaces are generally worse at everything. For example, they may be brittle, with worse semantic structure. How can we develop methods that reliably measure these flaws, and mitigate them? (Steering is a natural response here; discussed separately below.)

• The Geometry of Low-Resource Language Representations

  CPT method for measuring and mitigating geometric degeneration.

We may want certain structure from representation spaces in general, such as good semantic structure (similar things closer together), cross-lingual structure which helps transfer across languages, or interpretable encoding of some aspects of the text so that we can fiddle with them. What is the geometry of trained multilingual representation spaces? What are the consequences and what can we do better? How can we better conceptualize what we want from representations spaces, study these properties effectively, and develop strategies to achieve them in our systems?

• IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces

  Improving geometric similarity across languages in word embedding spaces using modified training objective.

• Exploring Alignment in Shared Cross-lingual Spaces

  Studying alignment and overlap of concept clusters across languages.

• Shared Global and Local Geometry of Language Model Embeddings

  Methods to characterize and analyse local and global geometry. Not specific to multilinguality but with potential applications.

How is language identity and semantic information represented in representation spaces? How can we improve representation structures to benefit applications?

• Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations

  Find that language identity is encoded in a low-rank space and introduce methods to project into a shared semantic space. Several other works are interested in the orthogonality of language and semantic information and using this fact for manipulation of such spaces.

• Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs

  Facts across languages may not be “stored” in the same “place”.

The world’s knowledge resides in various languages, and we want to retrieve the relevant information regardless of the language of the document or query. Language preference is an undesirable bias based on the document language during RAG. Systems general favor English or high-resource language documents over others. It’s a big problem, with mitigating ideas such as language-based reranking. Neater solutions pending.

• Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG

  Studies language preference and show that it comes at a quality cost.

We want monolingual RAG to be effective for all languages. Today, retrieval is largely performed by embedding-based retrieval, which depends on the quality of the representation space in any language. How can we perform effective retrieval for a low-resource language when the underlying representation space may be poor quality? Translation-based solutions swerve around this problem but introduce their own problems. Can we find fundamental solutions here?

• Multilingual Retrieval-Augmented Generation for Knowledge-Intensive Question Answering Task

  Compare native multilingual versus translation-based approaches.

Depending on the scenario, we also want cross-lingual RAG - information present in some language document should be available regardless of the language of the query. Further, not all information is present in all languages, either due to missing documents in low-resource languages, or naturally varying information focus in various languages or cultures. How can we construct appropriately cross-lingual representation spaces or develop strategies to achieve this?

• Learning Cross-Lingual IR from an English Retriever

  Use knowledge distillation from a translate-retrieve pipeline to a single cross-lingual retrieval step, and show that it outperforms the former.

Once we’ve retrieved the documents, how do we better incorporate the information into the final output? This is here for completeness; I haven’t engaged much with the literature and don’t know of much that is specifically directed at multilingual settings apart from language preference issues.

• BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval Augmented Generation

  When presented with conflicting sources of information, how should we resolve or weight them?

LLMs show the ability to “learn” from provided context in the prompt, and this can be used to boost multilingual performance. Recent work looks at teaching LLMs new languages using information in the context. Other work looks at careful selection of examples to boost desired multilingual and cross-lingual task performance. Note that this works in the comprehension direction; generation is still abysmal. Can we systematically understand what context is useful? Can we understand why it is useful, interplay with existing parameteric knowledge and underlying mechanisms? How much and what information is required to acquire basic generation fluency in a new language? Can we develop reliable and robust methods to leverage in-context learning for multilinguality in both directions?

• Hire a Linguist!: Learning Endangered Languages in LLMs with In-Context Linguistic Descriptions

  Use rich linguistic descriptions to teach LLMs new languages.

• Rashid: A Cipher-Based Framework for Exploring In-Context Language Learning

  Studies in-context language learning in a controlled setting.

Asking an LLM to think more about something often helps. A chain of thought is a search through a solution space, and we would like this search to be effective in various languages. It may also, ideally, help us catch up in worse-performing languages. Similarly, phrasing the ask differently or putting multiple models together in a “multi-agent debate”, or various other techniques may also help in getting more out of trained models. Do multilingual settings also show gains from more reasoning, and do those gains come from more language processing or something else? Why do we observe language drift in the reasoning traces, and what are fundamental solutions to this problem? Can we diagnose and mitigate language performance gaps using inference scaling, and how can we do it in a reliable and generalizable manner?

• Multilingual Test-Time Scaling via Initial Thought Transfer

  Study of test time scaling behaviour in multilingual settings.

• Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs

  Diagnose chain-of-thought reasoning failures within a graph framing of reasoning.

Steering is a cheap way to “steer” LLMs towards a particular behaviour: first we compute a steering vector in its representation space that represents that behaviour, and then we induce it by adding that vector to computed representations at inference time. Neat. How generalizable and robust is steering in multilingual settings, how does it compare to other inference time methods, and what can we do with it?

• Rethinking Cross-Lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs

  Uses steering to simultaneously improve factuality transfer and cultural grounding.

Can we use what we know something about LLM representation spaces to do some hacky shenanigans at inference time and make multilinguality work?

• Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention

  Train alignment matrices for low-resource languages and then align representations at inference-time to boost LRL performance.

We can give LLMs access to “tools”, which could be any function API, and they can now learn to integrate them usefully into their process. These tools may have multilingual interfaces; can LLMs use those as effectively as English-based tools? Another natural line of investigation: can we design tools and tool use to help LLMs avoid pitfalls and performance degradation in multilinguality? This last thing has surprisingly little work. Someone should work on it.

• PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflows

Famously, LLMs have been found to “think” in English, meaning that regardless of the input or output language, internal processing is performed in a “language-agnostic” (probably English-centric) manner. This has been studied in the past using logit lens, a technique to observe the semantics of internal layer representations. Follow-up work complicates and adds nuance to this idea, and a definitive understanding and theory remains to be found. Can we develop reliable techniques to study internal workings of LLMs? Can we consolidate and resolve current hypotheses in the field regarding this question? What are the implications of internal workings of LLMs for multilingual performance, including other aspects of interest such as cross-lingual sharing and language grounding?

• Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs

  Better techniques for finding and understanding the “lingua franca” of internal representation spaces.

• Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models

  Study which neurons in LLMs contribute to language generation and show that intervening on them alters performance.

• The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failure

  Errors arise in the transition from a language-agnostic task-solving phase to a language-specific realization phase.

• Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models

  Diagnoses factual inconsistency from an interpretability perspective.

Once upon a time, we cared mostly about classification tasks and some generation tasks like machine translation and summarization, that were measured with string overlap. Today, we care about subjective and open-ended generation tasks too. These are difficult to evaluate. Often, we use trained systems to do this, like xCOMET, Metric-X, or LLM-as-judge. But these are bad for low-resource languages or domains. When is it safe to use a particular metric? Can we provide confidence guarantees or reliability estimates or develop metrics with transparent blind spots? Can we build more reliable metrics?

• Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages

  Important paper!

• Last Translation Benchmark

  Introduces the verification rule paradigm which mitigates some of the pitfalls of LLM judges.

Benchmarks are large and costly to evaluate on. Recent work looks at selecting a few well-chosen examples which give us a reliable idea of scores on the entire benchmark. The multilingual setting: we have several languages and several tasks and want to know how good a model is for each combination. Can we do better than evaluating on everything? Can we gain meaningful insights on language abilities and task abilities? Can we generalize to new combinations?

• Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation

  Multidimensional IRT for the multilingual setting.

Languages and dialects differ from each other in systematic ways. How much do LLMs care about this? What is the relationship between language ties and performance? Although this field can feel (in my opinion) somewhat old and tired, fresh takes still pop up now and then.

• How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning

  Cross-lingual transfer from the lens of training data attribution.

• Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages

  Meta-learning approach for cross-task and cross-language generalization. Not really a cross-lingual transfer paper, but maybe a source of inspiration.

Dialects are funky and non-standardized, full of quirks and one-off things that weren’t in your training data. How robust are our models to these, how can we be better? Can we teach models to generate dialectal text in a data-efficient manner? What are the mechanisms of dialectal generalization in models and can we insert our knowledge of dialects in useful ways? And other fascinating questions.

• Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization

  Quantifying model degradation as dialectal distance grows.

• DialUp! Modeling the Language Continuum by Adapting Models to Dialects and Dialects to Models

  Synthetic training data for inducing dialectal robustness.

Bilingual and multilingual people mix languages when they speak all the time without even noticing. It’s not arbitrary: there are syntactic and other rules as for any language, and pragmatic and social cues. How good are current models at ingesting and producing naturalistic code-switching? How does it depend on the resourcedness of the languages being code-mixed and of the code-mixed variety itself in the training data? Can we build tools to evaluate naturalness (once we understand it ourselves)?

Fun fact: code-switching was the thing that got me into NLP. I don’t keep up anymore with the literature, though, so here’s a cool-looking survey-ish paper.

• Code Mixologist : A Practitioner’s Guide to Building Code-Mixed LLMs

  Broad discussion.

• Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data

  Talks about code-switching syntax and synthetic data generation. One of the first NLP papers I ever read.

An embarrassing issue for NLP: sometimes, outputs aren’t even in the right language. What are fundamental causes, and fundamental solutions?

• Understanding and Mitigating Language Confusion in LLMs

  Develops evaluation metrics and shows progress made by inference-time methods.

Machine translation is multilinguality-complete right? In general, MT (and summarization) is a test of how well we can understand any content in a language, and articulate any thought in that language. This includes the capability to understand and produce fluent, grammatical, naturalistic, and idiomatic utterances. If we can do this perfectly, we can do anything that we can do for English.

If we can add 2+2 in English, we should be able to do it in any language. Unfortunately, we can’t. Another issue is that we also often want the reasoning traces of the model to be in the right language so that (for example) the user can go through it. Is multilingual reasoning decomposable to translation and high-resource language reasoning, even internally to a model? What is this field up to other than using translation in various ways? Multilingual reasoning (in my opinion) has the potential to produce much more interesting papers than it currently does :eyes:

• Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning

Does a model “know” the same things when asked in different languages? Where is parametric knowledge stored, and is it reachable independent of input or output language?

• Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics

  Look at dynamics of factual representation formation during training. Very cool.

• How Do Multilingual Language Models Remember Facts?

  Investigate multilingual factual recall mechanisms using causal tracing and other things.

Can a model comply with user constraints and instructions in all languages? Respond in pirate-speak, use 5 sentences only, give me a bulleted list, etc.

LLMs are less safe and more toxic in other languages. We need to fix this.

We want models to be cross-lingually consistent and show cross-lingual transfer. That is, they should know the same things and exhibit the same behaviour across languages. We also want them to be localized and culturally grounded. Sometimes, asking in a different language warrants different cultural assumptions and results in different expected behaviours. These two objectives can be somewhat at odds with each other in practice - one seeks to homogenize, the other to diversify. How can we make or bake in systematic decisions about which is to be prioritized for a given input? How can we achieve both in suitable contexts without compromising the other?

• Rethinking Cross-lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs

  Shows that methods target transfer and culturally specificity are at odds.

What do language communities want from multilingual NLP?

• Decolonising Speech and Language Technology

What effect does NLP and AI have on language use and vice versa?

• Losing our Tail, Again: (Un)Natural Selection and Multilingual LLMs

  Fun paper!