Safetensors
qwen3

OpenEuroLLM logo

Prelude-base 9B

This repository contains checkpoints of OpenEuroLLM Prelude-base, a 9B dense base model which has been trained on Leonardo (CINECA). See the OpenEuroLLM progress tracker for live status across all OpenEuroLLM training runs.

It has been trained on 10T tokens (~80% English + code + math, ~20% multilingual across the 36 initial OpenEuroLLM target languages, represented as 41 language/script codes in the data mix, since some languages are split into macrolanguage + specific-variant codes, e.g. Latvian/Standard Latvian/Latgalian, Estonian/Standard Estonian, Albanian/Tosk Albanian, Norwegian Bokmål/Nynorsk). Full dataset composition, sources, and the per-language token breakdown are documented in the training-data-collection repo.

Following standard for model releases, each branch (e.g. iter_0124800) is a complete, self-contained HuggingFace checkpoint and a validation.json with example generations (see Sanity-check generations below). The main branch itself contains the final checkpoint of the base model after annealing (stage2-step_1192093). Branches prefixed with iter_ are checkpoints obtained before the annealing and annealing checkpoints are prefixed with anneal_.

How to Get Started

Load a specific training checkpoint via the revision argument (branch name = Megatron iteration, e.g. iter_0124800):

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "openeurollm/prelude-base"
revision = "stage2-step_1192093"  # default to "stage2-step_1192093" if empty

model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)

prompts = [
    "Hola, ¿qué tal?",
    "Bonjour, comment ça va ?",
    "Guten Tag, wie geht es dir?",
    "Ciao, come stai?",
    "Hei, mitä kuuluu?",
]
generate_kwargs = dict(max_new_tokens=50, do_sample=True, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.3, no_repeat_ngram_size=3)

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, **generate_kwargs)
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Prelude-base was trained without a <bos> token: documents are packed back to back, each ending in <eos>. The tokenizers on all branches therefore do not add <bos> (add_bos_token: false).

See the repository's branch list for all available checkpoints.

Sanity-check generations

Every checkpoint's branch includes a validation.json with greedy-decoded completions for a fixed battery of prompts, generated at conversion time as a quick "does this checkpoint produce coherent text" spot-check:

  • 4 canonical incomplete-sentence prompts, covering general knowledge ("The capital of France is..."), common sense ("The opposite of hot is..."), science ("Water boils at a temperature of..."), and creative/narrative ("Once upon a time...")
  • Each of the 4 is localized into all 36 target languages (144 generations total), using each language's actual native conventions where relevant (e.g. its own fairy-tale opening phrase for the narrative prompt) rather than literal word-for-word translation.

These completions were generated with <bos> in front of each prompt. A rerun on the final checkpoint without <bos> gave completions of the same quality.

Model Details

Dense, untied input/output embeddings, closely follows the Qwen3 architecture.

Layers 36
Hidden size 4096
FFN hidden size 12288 (dense)
Attention heads 32
KV groups (GQA) 8
Head dim (kv_channels) 128
Position embedding RoPE, base 100000
Normalization RMSNorm, eps 1e-5, with QK-norm
Activation SwiGLU
Sequence length 4096
Vocab size 262,145 (256k tokenizer, padded to 262,272 for TP efficiency)
Dtype bf16
num tensors        : 399
dtypes             : ['BF16']
embed_tokens params: 1,074,266,112
lm_head params     : 1,074,266,112   (untied from embeddings)
transformer body   : 6,946,075,648
TOTAL params       : 9,094,607,872   (~9.095 B)
weight bytes (hdr) : 18,189,215,744  (~18.19 GB)

Training Details

Global batch size 2048
Micro batch size 4
Sequence length 4096
Tokens / iteration ~8.39M
Total training iterations 1,192,093 (~10T tokens)
LR schedule WSD (warmup-stable-decay), linear decay
Peak LR 3.5e-4
Warmup iterations 4,000
Cooldown iterations 238,476 (final 20% of training)
Optimizer Adam (β1=0.9, β2=0.95, eps=1e-8), distributed optimizer
Weight decay 0.1
Gradient clipping 1.0
Parallelism TP=4, PP=1, CP=1, sequence parallel
Checkpoint format Megatron torch_dist, saved every 2,400 iterations
Hardware 256 nodes / 1024 GPUs, Leonardo (CINECA)

Training logs (loss curves, throughput, etc.) are tracked on Weights & Biases.

Tokens trained at a given checkpoint

Each iteration consumes exactly one global batch, so the number of tokens seen at a given branch's iteration can be computed directly from its name:

tokens(iteration) = iteration × global_batch_size × sequence_length
                   = iteration × 2048 × 4096
                   = iteration × 8,388,608

For example, iter_0124800 corresponds to 124,800 × 8,388,608 ≈ 1.05T tokens.

Evaluation

We report evaluations in this model card on English and Multilingual benchmarks together with in-depth benchmarks developed for German, Norwegian and Finnish as part of the consortium.

English & Multilingual evaluations. We show here the performance on English & Multilingual evaluations. Note that we report the performance of models which are trained mostly in English (Olmo3 and Marin) to illustrate the benefit over English models for over languages even though those models are not expected to perform very well on non English benchmarks.

English benchmarks (decontaminated tasks)

Multilingual benchmarks (decontaminated tasks)

Those tasks were all decontaminated from our training data, we do not know whether they are part of the training set of other methods.

In-depth evaluations. We show the performance in German, Norwegian (NorEval) and Finnish. Those tasks were not explicitly decontaminated but we do not expect large contamination.

German benchmarks

Norwegian benchmarks (NorEval)

Finnish benchmarks (FIN-bench)

Other evaluations and details. Evaluations including on tasks which were not explicitly decontaminated are available here. This directory also generates all figures given evaluation stored on this HuggingFace repository.

Uses

These are intermediate checkpoints, intended for studying training dynamics (loss/capability progression, multilingual acquisition, etc.), not for production or downstream deployment as no safety alignment or instruction tuning has been applied.

Downloads last month
3,605
Safetensors
Model size
9B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for openeurollm/prelude-base

Finetunes
8 models
Quantizations
2 models

Spaces using openeurollm/prelude-base 2

Collection including openeurollm/prelude-base