Question on the architecture ablation and 1M-context evaluation
Thank you for releasing Solar Open 2 and its technical report. I found the hybrid KDA–softmax architecture, NoPE design, and support for negative eigenvalues particularly interesting.
I have a question regarding the proxy architecture ablation.
The report uses validation loss, MMLU, and HellaSwag learning curves to show that the hybrid architecture reaches a given level of general capability with fewer training tokens. These results provide useful evidence of improved pre-training sample efficiency.
However, the report also states that the central architectural objective is to provide a usable context window beyond one million tokens and support long-horizon agentic tasks. The motivations for NoPE, hybrid attention, and negative eigenvalues are closely related to long-context extrapolation, information retention, and state tracking.
I recognize that the final model is evaluated on AA-LCR and that multiple metrics are tracked during the length-expansion stage. However, those results reflect the completed model after pre-training, length expansion, checkpoint merging, and post-training, rather than isolating the architectural contribution itself.
Were controlled proxy ablations conducted between the all-softmax baseline and the hybrid architecture on length-dependent long-context tasks, such as retrieval, aggregation, information overwrite, or state tracking? If such experiments were omitted only at the report level but actually conducted, would it be possible to share those results?
My concern is not that MMLU and HellaSwag are unsuitable as low-cost proxy signals. Rather, they demonstrate general sample efficiency but do not directly establish whether the hybrid architecture improves the long-context capabilities it was primarily designed to support. If such controlled long-context experiments were conducted, it would be valuable to see the results.
Hi, thank you for the thoughtful question.
We didn't run that experiment — a fully controlled all-softmax vs. hybrid ablation on length-dependent tasks. What we do have is a proxy-model result on length extrapolation.
During length extension we trained only up to 256k, and measured RULER at both lengths: 80.0 at 256k and 68.0 at 1M — the latter 4× beyond any length seen in training.
You may already be familiar with it, but Kimi Linear (arXiv:2510.26692) may be a useful reference here — it runs what looks like a comparison close to what you're asking about within the same architecture family, including a NoPE vs RoPE ablation.
Hope this is helpful.
Thank you, very helpful. The 256K-to-1M RULER extrapolation result is also interesting, and the reference to the Kimi Linear ablation is helpful for understanding the motivation behind the NoPE design.
I have one related follow-up question about Figure 4. How were the target scores of 0.55 for MMLU and 0.63 for HellaSwag selected? More broadly, what was the rationale for using MMLU and HellaSwag as reference metrics for capability acquisition in the architecture ablation?
Were these scores predefined as target capability thresholds before the experiments, or were they selected based on the scores reached by the hybrid proxy at the 210B-token checkpoint?
Additionally, did you examine whether the reported 3.2× and 1.8× differences in tokens-to-target remain comparable at other score thresholds along the learning curves? I ask because the estimated efficiency ratio may depend substantially on the selected operating point.