Data and evaluation
Every dataset and benchmark below exists publicly. We checked its licence before listing it. Each source's licence will be respected and stated in the model card.
Training data
| Dataset | Source | Licence | Use |
|---|---|---|---|
| Norwegian Colossal Corpus (NCC) | National Library of Norway | Mixed per sub-corpus (e.g. Wikipedia CC BY-SA 3.0). We use only the openly licensed parts. | Norwegian adaptation |
| norwegian-alpaca | National Library of Norway | CC BY 4.0 | Norwegian instructions |
| glaive-function-calling-v2 | Glaive AI | Apache-2.0 | Function calling (to be translated) |
| xlam-function-calling-60k | Salesforce | CC BY 4.0 (gated) | Function calling (to be translated) |
Benchmarks
| Benchmark | Licence | Measures |
|---|---|---|
| NorEval | UiO LTG, MIT | Norwegian understanding and generation, Bokmål and Nynorsk |
| EuroEval (formerly ScandEval) | MIT | Norwegian language tasks, comparable with other models |
| Berkeley Function Calling Leaderboard (BFCL) | Apache-2.0 | English tool calling, to check we don't lose ability |
| Majere NO-FC (new, ours) | To be released CC BY 4.0 | Norwegian tool calling: choosing the function, exact argument match, JSON validity, and correctly declining. Bokmål and Nynorsk, with a human-checked test split. |
Baseline on 40 seed tasks, no fine-tuning (12 Oct 2026)
Baseline on 40 seed tasks, no fine-tuning. NO-FC v0, run on one RTX 5060 Ti 16 GB through Ollama (Q4_K_M, temperature 0, native tool calling). 40 tasks is a small development set, so treat these as indicative numbers. Raw answers and error analysis.
| Model | Exact match (all) | Bokmål | Nynorsk | Function name | Arguments | No call | Clarify |
|---|---|---|---|---|---|---|---|
| qwen3:8b | 0.825 (33/40) | 0.85 | 0.80 | 0.92 | 0.82 | 1.00 | 1.00 |
| NorMistral-11B-thinking | Not scored: the GGUF has no tool-calling support in Ollama | ||||||
Main errors for qwen3:8b: wrong weekday counted from the reference date (3), "halv ni" read as 09:30 instead of 08:30 (1), a wrong phone number in Nynorsk (1), and unnecessary follow-up questions (2). Norwegian date and time expressions are exactly what the fine-tuning targets.
Metrics and baselines
Baselines are the untouched base models (Qwen3-8B, NorMistral-7B-warm instruct) under the same prompting and quantisation. Headline metrics: NorEval average (Bokmål and Nynorsk reported separately), BFCL accuracy, and NO-FC exact-match and JSON validity, all measured at 4-bit on a 16 GB GPU. Pre-registered criterion: the project counts as successful only if NO-FC improves clearly over both baselines without NorEval dropping beyond seed noise. Results are published either way.