EMNLP ’26 Multilinguality Digest
EMNLP was very fun. Great conversations and ideas for multilinguality. Here are 22 papers thinking about various directions in the field that I enjoyed, ordered roughly by topic area. Some particularly brain-tickling ones are marked **.
Obviously, there were other great works in multilinguality at EMNLP that are not on this list. This is my personal shortlist, etc.
-
** ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
Think of multilingual pretraining as a game. The result is the aggregate loss over all languages. Each language contributes something to each other language (cross-lingual transfer). But how much is this effect, given the presence of other languages? They use Shapley Value to understand this (in practice, faster approximation using IRDS).
Now, they frame a multilingual scaling law based on this for a particular target language.
They just scale normal scaling law which predicts loss given a particular model size and tokens (fitted with some params) with a term that represents cross-lingual transfer.They can then find the optimal mixture of data by miminizing this loss (brute force search over various distributions?). They show that this performs best (although the difference seems small and is measured by perplexity). Very cool idea.
-
** Localizing Cross-Lingual Generalization in Parameter Space
Says that if there was cross-lingual generation, then the parameters of the model with just an English adapter must be in the same loss basin as the model parameters with target-language task adapters. (Same model; finetuned with the different adapters).
Uses linear mode connectivity to check whether this is the case.
Finds that it is task dependent.
Then, they ask the question: which capability is it that the English adapters are missing when they don’t generalize?
They look at finetuning only initial/middle/final layers as per the multilingual workflow hypothesis (comp, task-solving, gen) and find that it’s mostly comprehension and task solving (unexpected?). Interesting to try to reconcile this with the intuition that fine-tuning the last few layers should help most for language-specific stuff… -
Plasticity without Collapse: Plasticity-Inducing Control Optimizer for Cross-Lingual Continual Pre-Training
General method to reduce catastrophic forgetting. Metrics for CF.
-
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
Previous work finds that multilingual MoE models do diffuse language agnostic routing in middle layers but specialise in later layers.
This work makes that choice explicit saving a bunch of parameters and maintaining performance. -
** ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs
Propose a way to evaluate crosslinguality of embeddings. The idea is to take bitext, then get the semantic parse of the English side, then make one semantic edit. Then check whether the perturbed version is farther away than original. Get % of triplets where this is the case. Neat!
-
Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops
Datasets: KLAR, mParaRel - cross lingual consistency datasets.
Increase CLC by training the model to have semantic alignment in encoder, with language specific decoders.
This can be done by first training an autoencoder only with a cross-lingual loss and then some other objectives to prevent collapse (self-reconstruction, variance maintaining), and then freezing that and training language specific decoders to decode from that space and reconstruct the original language hidden states.
Note that this takes hidden states as input and produces hidden states as output.
Then at inference time with the multilingual LLM, we can at any layer take the hidden states, project them into the encoder, make some modifications, decode and get back some hidden states, and then give them back to the multilingual LLM.
They try three things on CLC datasets, given many language variants of the same question: a baseline (no modification), PCA (project away some inconsistent subspace in the shared latent space), and mean-shift (pull all language representations towards the mean in the latent space).
Gains are small. No analysis on what examples give gains. Cool idea. -
Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
Three things:
1. Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases.
2. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models.
3. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze task flips the prediction to the English model’s answer in most cases.CKA to measure alignment across models
-
Target-Language Generation in Multilingual Models: Activation Steering and Optimal Control
Good intro for steering.
They want to steer for on-targetness.
Rubric for evaluation: language adherence, semantic coherence, linguistic coherence (right language, high quality content, linguistically well formed).
Have binary metric for each of these based on a threshold (somewhat arbitrary): lang adherence (lang id says that lang); semantic coherence (is embedding similarity better than that between clearly non-equivalent sentences?); linguistic coherence (log perplexity with an ngram LM to avoid confounds from other LLMs).
Then they have three criteria for steering: 1. Minimal disturbance 2. Guaranteed output language correct 3. Respect geometric structure i.e. don’t mess up quality.Method: We can compute the desired language region as the pre-image of the language outputs (good idea!). Seems like they use a linear classifier on the space to decide what is the language region. Not sure how details work.
Then we find the minimal intervention that takes us there
Apparently this transformation also doesn’t disturb semantics - didn’t understand why not - only claimed in conclusion never mentioned before. -
Cross-lingual Representation Learning via Centroid Intervention Fusion
Improvement on INCLINE (inference time representation mapping) : we’re no longer mapping to English but to a shared cross-lingual target
-
** Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space
Wants to make hidden layer representations closer to English.
Only do this in middle layers.
Separate out language-specific part.
For the language agnostic part - take the hidden state for each token.
Assign uniform probability to each token.
Do optimal transport to move the target language distribution towards English using cosine distance as the cost. -
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
We want to compare language models across languages in a fair way. This should also be doable across tokenisers.
The paper compares several metrics like perplexity, etc., and shows that sentence level NLL is the way to go.They show that all metrics other than Sent NLL have a bunch of confounders, shown by correlations with these variables.
For ranking: they assume that true rank is influenced by training data composition. They show that only Sent NLL reflects expected rank. (Idk if I believe this because can you really say that that is expected rank? But well. The figure is neat.) -
Cross-lingual Self-Consistency for Multilingual Reasoning with Language Models
In tasks where we have some chain of reasoning and then a final answer, previous method does RL to incentivise the consistency of the final answer marginalised over many chains of thought (self-consistency). In this work, they extend this idea to encourage consistency across languages. The reward is framed as low cross-entropy with some target distribution (could be English, could be the average over all languages).
The method only needs English questions and uses the model itself to translate thus avoiding any influence of other models and also not requiring gold answers.
This shows some gains but very narrowly beats S1: SFT on English data (Read: Crosslingual reasoning through test-time scaling.) -
Reinforcement Learning with Robust Rubric Rewards
LLM writes checklist style rubric, either verifiable or fuzzy.
LLM extracts text and passes it to relevant reward-er.
We do RL. -
One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography
Try pretraining with romanization, IPA. Then fine-tune. Then see how well it does on 8 languages consisting of four pairs: Hindi/Urdu and others, where script may impede transfer.
-
Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models
Compares a bunch of tokenisers for multilingual equity: parity aware BPE takes into account the fact that different scripts have diff bytes per character.
This is the best on equity-efficiency frontier. -
No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
Look at 200 models and their tokenisers - keeping everything identical except tokenisation. Shows (lack of) relationship between linguistic tokenizer metrics and downstream performance. Tokenization continues to keep life interesting.
-
Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfer
Find which neurons activate for reasoning in HRLs. Activate those with steering in LRLs. Performance improves.
-
Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs
Breaking down translation stage in model-internal comprehension —> task-solving —> translation stages. Show that choice of word order, language, lexical content happen in this order.
-
Translating the Untranslatable: An Operationalizable Ontology for Untranslatability
Makes an ontology for the types of untranslatable things. And strategies that people prefer models to use when dealing with them. Scope for LTB thoughts.
-
Register-based data selection for multilingual LLMs
Studies multilingual data sampling based on register labels in HPLT3. Strategies: exclude low performing registers entirely, or subsample, etc.
-
Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment – Is English Enough?
Find correlation between cross-lingual alignment metrics and final performance.
They also do this for translation: they use PMI between source and the target as a metric of translation that is comparable across languages for a given model. This is nice.