Rose-1.5-Medium
~98M parameters · Rose X1.5 · 4096 context · ~100B training tokens
Intelligence Index: 21.07
ArithMark-3: 40.70%
A Break Note
Before anything else, I want to say something about this release.
Rose-1.5 will be my last model series for a while.
I've spent a lot of time on these models — training them, evaluating them, debugging them, breaking them, fixing them, and occasionally discovering that the problem was my own code all along.
It has been incredibly rewarding, but it has also started taking a lot more time than I can afford right now.
I have my exams coming up, and I need to give them the attention they deserve.
So after the Rose-1.5 series, I'm taking a long break from model development.
There will still be one larger Rose-1.5 model coming, but after that, I'm stepping away for a while.
This isn't really the ending I imagined when I started the Rose series.
But I'm glad I get to pause it here.
And somehow... this is the one that worked.
After Rose-Mini, Rose-Medium and Rose Pro, I wasn't expecting another model to suddenly make the leaderboard interesting again.
Then Rose-1.5-Medium happened.
It is based on the new Rose X1.5 architecture, trained for roughly 100B tokens, supports a native 4096-token context, and comes in at roughly 98M parameters.
And somehow, this smaller model managed to do something I wasn't expecting:
It surpassed Rose Pro on the Intelligence Index.
Rose Pro:
151M parameters — 20.97 Intelligence Index
Rose-1.5-Medium:
~98M parameters — 21.07 Intelligence Index
A difference of only 0.10, but enough to put Rose-1.5-Medium above the previous Rose flagship on the leaderboard.
I was not planning on writing that sentence.
Model Overview
| Property | Rose-1.5-Medium |
|---|---|
| Architecture | Rose X1.5 |
| Parameters | ~98M |
| Layers | 22 |
| Hidden Size | 576 |
| Attention Heads | 9 |
| KV Heads | 3 |
| Head Dimension | 64 |
| Intermediate Size | 1532 |
| Vocabulary | 32,768 |
| Context Length | 4096 |
| Training Tokens | ~100B |
| Attention | Rose X1.5 / XSA |
| QK Normalization | Yes |
| Refresh Gates | Layers 6, 13 |
| Embeddings | Tied |
| Activation | SiLU |
| Normalization | RMSNorm |
| Dropout | 0 |
The configuration is available in the repository's config.json.
The Rose Family
This is where things got interesting.
| Model | Parameters | Intelligence Index | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-3 |
|---|---|---|---|---|---|---|---|
| Rose-Mini | 49.4M | 10.85 | 28.95% | 36.62% | 24.32% | 58.54% | 36.50% |
| Rose-Medium | ~97.8M | 17.73 | 35.29% | 44.19% | 26.19% | 62.95% | 38.30% |
| Rose Pro | 151.3M | 20.97 | 38.24% | 48.65% | 26.96% | 65.18% | 38.20% |
| Rose-1.5-Medium | ~98M | 21.07 | 38.09% | 47.22% | 27.13% | 64.80% | 40.70% |
The progression is probably the most interesting part.
10.85 → 17.73 → 20.97 → 21.07
Rose-Mini started the family.
Rose-Medium showed that scaling could take things considerably further.
Rose Pro pushed the parameter count to 151M and became the strongest Rose model at the time.
Then Rose-1.5-Medium arrived with roughly the same parameter budget as Rose-Medium.
And somehow:
21.07 > 20.97
Apparently I did not need another 50 million parameters.
Intelligence Index
The Open SLM leaderboard uses the Intelligence Index as its main ranking metric.
Rose-1.5-Medium scored:
21.07
For comparison:
| Model | Intelligence Index |
|---|---|
| Rose-Mini | 10.85 |
| Rose-Medium | 17.73 |
| Rose Pro | 20.97 |
| Rose-1.5-Medium | 21.07 |
So Rose-1.5-Medium has officially moved above the entire previous Rose family on the leaderboard's overall Intelligence Index.
And this is probably the result I'm happiest about.
Not because 21.07 is some magical number.
But because this model is only around 98M parameters, while Rose Pro is around 151M.
The leaderboard apparently decided that Rose-1.5-Medium deserved the seat.
I am not going to argue with it.
The ArithMark Problem
There was a small problem with the Rose family.
ArithMark-3.
| Model | Parameters | ArithMark-3 |
|---|---|---|
| Rose-Mini | 49.4M | 36.50% |
| Rose-Medium | ~97.8M | 38.30% |
| Rose Pro | 151.3M | 38.20% |
| Rose-1.5-Medium | ~98M | 40.70% |
Rose-Medium reached 38.30%.
Then I made Rose Pro significantly larger.
ArithMark replied with:
38.20%.
So naturally, I went back to roughly 98M parameters.
And got:
40.70%.
At this point I'm not sure whether I improved the architecture or simply annoyed the benchmark enough.
Jokes aside, this is the first Rose model to break the 40% ArithMark-3 mark.
And it is currently the 4th-highest ArithMark-3 score across the entire AxiomicLabs Open SLM leaderboard, despite being only around 98M parameters.
That is probably the result I least expected from this model.
Standard Benchmarks
HellaSwag
38.09%
Very close to Rose Pro's 38.24%.
PIQA
64.80%
Again, extremely close to Rose Pro's 65.18%.
ARC-Easy
47.22%
Rose Pro remains ahead here at 48.65%.
ARC-Challenge
27.13%
A small but meaningful improvement over Rose Pro's 26.96%.
What Actually Changed?
Rose-1.5 is not meant to be a completely different philosophy from Rose X1.
It is an evolution.
The architecture keeps the ideas that made Rose X1 interesting while introducing several changes and refinements.
One of the major differences is the shift toward a slightly wider but shallower configuration:
Rose-Medium
- 24 layers
- 512 hidden size
Rose-1.5-Medium
- 22 layers
- 576 hidden size
The context length also moves from 2048 to:
4096 tokens.
Whether every improvement comes directly from these architectural changes is impossible to prove from a single training run.
But the result is certainly interesting.
And after enough experiments, I have learned not to confidently blame one variable for everything.
That lesson was expensive.
Training
Rose-1.5-Medium was trained for approximately:
100 billion tokens
The goal was not simply to make a larger model.
The goal was to see whether the new Rose X1.5 design could extract more capability from roughly the same parameter range.
The answer, at least from these evaluations, appears to be:
Yes.
And this time, I actually got some sweet fruit after all the hard work.
Rose-1.5 Is Not Finished Yet
Rose-1.5-Medium is only the first model of the Rose-1.5 family.
There is one more, larger model planned.
The next release will be in approximately the:
135M–150M parameter range
and will continue using the Rose X1.5 architecture.
So there is still one more Rose-1.5 model to come.
After that, however, I will be taking my break.
What About Rose X2?
Rose X2 exists.
The architecture has already been finalized and ablated.
So technically, the next generation is sitting there.
But for now:
Rose X2 will not be released.
I have no interest in promising a model that I may not have the time to properly train, evaluate and release.
Maybe Rose X2 will return someday.
Maybe it won't.
For now, it is finished on paper, tested through experiments, and waiting.
Why I'm Stopping
This is probably the hardest part of writing this model card.
I've spent a lot of time working on these models.
A lot of late hours.
A lot of training runs.
A lot of debugging.
A lot of benchmark evaluation.
And quite a few moments where I looked at my own code and wondered how I had managed to create the problem in the first place.
The Rose series has taught me an enormous amount.
But it has also started taking too much of my time.
I have exams.
I have other things I need to focus on.
And eventually, you have to know when to put the keyboard down.
So after the final Rose-1.5 release:
I'm taking a long break.
Not because I stopped caring about this.
Quite the opposite.
I think taking a step back is probably the best thing I can do.
One Last Rose
Rose started small.
Rose-Mini taught me how much could be squeezed out of a relatively tiny model.
Rose-Medium showed me what happened when I pushed the scale further.
Rose Pro taught me that simply making a model bigger does not automatically make it better.
And Rose-1.5-Medium somehow managed to give me the result I had been chasing for a while.
~98M parameters.
4096 context.
~100B tokens.
21.07 Intelligence Index.
40.70% ArithMark-3.
And above Rose Pro on the leaderboard.
After all the failed experiments, long training runs, debugging sessions and my occasionally impressive ability to introduce bugs with my own fingers...
This one finally feels like some sweet fruit.
And honestly, that makes stopping here a little bittersweet.
There is one more model coming.
Then I'll step away.
I don't know exactly when I'll come back to this.
But I know I need the break.
So, for now:
Thank you for following the Rose series.
See you after the break.
Credits
Built and trained by Mr Munk.
Rose X1.5 continues the architectural work started with the original Rose X1 series.
Thanks to everyone whose open-source work, datasets, benchmarks and research made these experiments possible.
And thank you to everyone who followed the Rose experiments from the beginning.
- Downloads last month
- 465