Prelude-base 9B
This repository contains checkpoints of OpenEuroLLM Prelude-base, a 9B dense base model which has been trained on Leonardo (CINECA). See the OpenEuroLLM progress tracker for live status across all OpenEuroLLM training runs.
It has been trained on 10T tokens (~80% English + code + math, ~20% multilingual across the 36 initial OpenEuroLLM target languages, represented as 41 language/script codes in the data mix, since some languages are split into macrolanguage + specific-variant codes, e.g. Latvian/Standard Latvian/Latgalian, Estonian/Standard Estonian, Albanian/Tosk Albanian, Norwegian Bokmål/Nynorsk). Full dataset composition, sources, and the per-language token breakdown are documented in the training-data-collection repo.
Following standard for model releases, each branch (e.g. iter_0124800) is a complete, self-contained HuggingFace checkpoint and a validation.json with example generations (see Sanity-check generations below).
The main branch itself contains the final checkpoint of the base model after annealing (stage2-step_1192093). Branches prefixed with iter_ are checkpoints obtained before the annealing and annealing checkpoints are prefixed with anneal_.
How to Get Started
Load a specific training checkpoint via the revision argument (branch name = Megatron
iteration, e.g. iter_0124800):
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "openeurollm/prelude-base"
revision = "stage2-step_1192093" # default to "stage2-step_1192093" if empty
model = AutoModelForCausalLM.from_pretrained(
repo, revision=revision, dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
prompts = [
"Hola, ¿qué tal?",
"Bonjour, comment ça va ?",
"Guten Tag, wie geht es dir?",
"Ciao, come stai?",
"Hei, mitä kuuluu?",
]
generate_kwargs = dict(max_new_tokens=50, do_sample=True, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.3, no_repeat_ngram_size=3)
for prompt in prompts:
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, **generate_kwargs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Prelude-base was trained without a <bos> token: documents are packed back to back, each ending in <eos>.
The tokenizers on all branches therefore do not add <bos> (add_bos_token: false).
See the repository's branch list for all available checkpoints.
Sanity-check generations
Every checkpoint's branch includes a validation.json with greedy-decoded completions for a
fixed battery of prompts, generated at conversion time as a quick "does this checkpoint produce
coherent text" spot-check:
- 4 canonical incomplete-sentence prompts, covering general knowledge ("The capital of France is..."), common sense ("The opposite of hot is..."), science ("Water boils at a temperature of..."), and creative/narrative ("Once upon a time...")
- Each of the 4 is localized into all 36 target languages (144 generations total), using each language's actual native conventions where relevant (e.g. its own fairy-tale opening phrase for the narrative prompt) rather than literal word-for-word translation.
These completions were generated with <bos> in front of each prompt. A rerun on the final
checkpoint without <bos> gave completions of the same quality.
Model Details
Dense, untied input/output embeddings, closely follows the Qwen3 architecture.
| Layers | 36 |
| Hidden size | 4096 |
| FFN hidden size | 12288 (dense) |
| Attention heads | 32 |
| KV groups (GQA) | 8 |
Head dim (kv_channels) |
128 |
| Position embedding | RoPE, base 100000 |
| Normalization | RMSNorm, eps 1e-5, with QK-norm |
| Activation | SwiGLU |
| Sequence length | 4096 |
| Vocab size | 262,145 (256k tokenizer, padded to 262,272 for TP efficiency) |
| Dtype | bf16 |
num tensors : 399
dtypes : ['BF16']
embed_tokens params: 1,074,266,112
lm_head params : 1,074,266,112 (untied from embeddings)
transformer body : 6,946,075,648
TOTAL params : 9,094,607,872 (~9.095 B)
weight bytes (hdr) : 18,189,215,744 (~18.19 GB)
- Developed by: OpenEuroLLM
- Model type: dense, autoregressive decoder-only Transformer
- Languages: 36 initial OpenEuroLLM target languages (41 language/script codes in the data mix, see note above); per-language token breakdown
- License: Apache 2.0
- Training repository: oellm-autoexp,
baby_9b_denseexperiment - Architecture config:
qwen3_dense_9b.yaml
Training Details
| Global batch size | 2048 |
| Micro batch size | 4 |
| Sequence length | 4096 |
| Tokens / iteration | ~8.39M |
| Total training iterations | 1,192,093 (~10T tokens) |
| LR schedule | WSD (warmup-stable-decay), linear decay |
| Peak LR | 3.5e-4 |
| Warmup iterations | 4,000 |
| Cooldown iterations | 238,476 (final 20% of training) |
| Optimizer | Adam (β1=0.9, β2=0.95, eps=1e-8), distributed optimizer |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Parallelism | TP=4, PP=1, CP=1, sequence parallel |
| Checkpoint format | Megatron torch_dist, saved every 2,400 iterations |
| Hardware | 256 nodes / 1024 GPUs, Leonardo (CINECA) |
Training logs (loss curves, throughput, etc.) are tracked on Weights & Biases.
Tokens trained at a given checkpoint
Each iteration consumes exactly one global batch, so the number of tokens seen at a given branch's iteration can be computed directly from its name:
tokens(iteration) = iteration × global_batch_size × sequence_length
= iteration × 2048 × 4096
= iteration × 8,388,608
For example, iter_0124800 corresponds to 124,800 × 8,388,608 ≈ 1.05T tokens.
Evaluation
We report evaluations in this model card on English and Multilingual benchmarks together with in-depth benchmarks developed for German, Norwegian and Finnish as part of the consortium.
English & Multilingual evaluations. We show here the performance on English & Multilingual evaluations. Note that we report the performance of models which are trained mostly in English (Olmo3 and Marin) to illustrate the benefit over English models for over languages even though those models are not expected to perform very well on non English benchmarks.
Those tasks were all decontaminated from our training data, we do not know whether they are part of the training set of other methods.
In-depth evaluations. We show the performance in German, Norwegian (NorEval) and Finnish. Those tasks were not explicitly decontaminated but we do not expect large contamination.
Other evaluations and details. Evaluations including on tasks which were not explicitly decontaminated are available here. This directory also generates all figures given evaluation stored on this HuggingFace repository.
Uses
These are intermediate checkpoints, intended for studying training dynamics (loss/capability progression, multilingual acquisition, etc.), not for production or downstream deployment as no safety alignment or instruction tuning has been applied.
- Downloads last month
- 3,605