Recommended settings
Hi, what are the recommended llama.cpp settings? MTP seems not to be supported but it is fast enough on my Strix Halo.
Thank you so much for your great open models!
MTP is coming, don't worry :)
And what about the recommended llama.cpp settings? 😉
@LaurentPayot Sorry for the late reply. You can reuse the Step-3.5 settings:For general chat domain, we suggest: temperature=0.6, top_p=0.95, and for reasoning / agent scenario, we recommend temperature=1.0, top_p=0.95.
Hello, I am also using a Strix Halo device. I was wondering if you loaded the model under Windows 11 or Linux, and how you solved the issue of VRAM usage limitation! Under Windows 11, I still cannot overcome the ROCm limitation and can only use Vulkan for loading! I'm curious about how you handle the loading of models like Q4_K_S, which have weights exceeding 100G!
@X-AI-GT The benchmarks I posted in the model card were actually tested on Windows + Vulkan. I haven't really dug into ROCm yet. In your experience, is ROCm noticeably faster than Vulkan on Strix Halo?
Are you also using the combination of Windows 11 and Vulkan? Based on my experience, the advantage of ROCm is that its additional overhead is very low, requiring only slightly over 1GB of VRAM, whereas the "additional" overhead of using Vulkan can reach up to 6GB or more of VRAM, which is also a major obstacle that restricts our ability to load models! As for speed, in most cases, ROCm's speed is abysmal, completely uncomparable to Vulkan!
@X-AI-GT I’m using Linux (Ubuntu 26.04) with my EVO-X2 128GB so I can allow up to 120GB to be used by the iGPU, instead of the 96GB limit on Windows. See https://strixhalo.wiki/AI/AI_Capabilities_Overview#memory-limits
I’m using IQ4_XS quantization, so I have very little headroom for context. But usually quants lower than 4bits are not good quality. This is especially true for small models, maybe I should try with this bigger model.
By the way, I’m using Llama.cpp Vulkan.
Currently, under Windows 11, Vulkan can also use about 120GB of video memory, but due to its additional overhead, it is subject to many limitations! And I am still exploring ways to reduce this additional expense! But we haven't found the specific reason yet! I downloaded two versions of IQ4_XS quantization and Q4_K_S quantization for this model, and from programming testing, IQ4 is even better than Q4_K_S. I didn't have time to do more testing, so there may be uncertainty! However, I can only use a context length of 64K for Q4_K_S, while IQ4 can enable a context length of 128K (both using Q8 KV quantization). Due to heat dissipation issues, I did not conduct a full load speed test. In about 70% of the load tests, IQ4's speed peaked at 27tokens/s. I started using it from Step-3.5, but found that the low precision effect was very poor. Yesterday's Step-3.7-iq4 test, I found that it was too sluggish when executing tasks! The efficiency is very poor! If Deepseek v4 Flash is integrated into the mainline Vulkan in the later stage, it is worth trying. I have been using Deepseek v4 Flash recently, which is not very excellent, but the efficiency is still good! MiniMax M3.0 doesn't seem to be worth looking forward to, and according to our equipment, the accuracy that can be used will only be lower! This level of equipment is very awkward! A slightly better model cannot use the accuracy that can truly work. A model that can use sufficient accuracy is almost impossible to truly work with! Perhaps API is the ultimate destination ... Oh, I lifted some restrictions on Windows 11, so I can load this model, otherwise it would be a nightmare of 96GB!