Anyone tried this to run only on standard PC ram only?
Anyone tried this to run only on standard PC ram only? Like basic CPU workstation mobo and ram.
If yes how much tokens/s can this go during long context? Please please tell me!!!
Basic cpu and basic ram will freeze/OOM/fry if you try on a consumer grade machine.
The super bare minimal setup I theorically found is 2x3090 24Gb Vram + 256Gb ram to split the KV and be able to run at... 1-2tps (in the minimal compressed version).
Didn't test it yet because I only have 128Gb max on my biggest machine.
Your best option to have it unlimited at home is either to have a big cluster of gpus or a friend with big gpus, or use cloud providers. Enjoy !
Basic cpu and basic ram will freeze/OOM/fry if you try on a consumer grade machine.
The super bare minimal setup I theorically found is 2x3090 24Gb Vram + 256Gb ram to split the KV and be able to run at... 1-2tps (in the minimal compressed version).Didn't test it yet because I only have 128Gb max on my biggest machine.
Your best option to have it unlimited at home is either to have a big cluster of gpus or a friend with big gpus, or use cloud providers. Enjoy !
Hmm thanks! Okay then i suppose i will drop "consumer grade" hopes.. How do i know what item is "not consumer grade" based on price? must have "workstation or server word in it?"
Well, a 3090 is technically consumer grade, A 3060 12Gb too (avoid 4000 serie unless super incredible price), then there is the 5090.
Regarding prices, there exists three eras ^^
Pre crypto prices
Pre AI prices
Current prices, inheriting from the previous ones. Don't mistake me, rise in prices means the demand is super high (combined to scarcity of silicon-based components) and that people are actually willing to fiddle with AI at home. Check your local second hand market for 3090, they are sometimes cheaper wrapped in a fully working system 😎
Then there are some homelabs with 8x3090 which is still consumer grade !
To sumup the difference between consumer/cluster is indeed the price (juicy 8xMi300X or HB100 or HB200) but the physical hardware that is ahead of consumer one. Bandwith, memory correction, network, monitoring, cooling, power consumption... They are all on an other level.
(If you have brave browser, Ctrl + B then paste the technical stuff to the chat, or copy paste to your favorite chatbot and request more insight regarding consumer/cluster/datacenter based on my answer).
Enjoy the big numbers 🤓
Basic cpu and basic ram will freeze/OOM/fry if you try on a consumer grade machine.
Not true, people you need to stop being trapped in the "GPU thinking jail", you dont need GPUs to run primitive textual models (GPU usable only in video gen) you can run ANYTHING textual gen on ANCIENT hardware, because speed is NOTHING (moreover by running low quality fast on dozens GPUs = you're just getting tokens faster into trashbin, so = slop, analogy=its like producing trash or very fancy expensive food grinder under the kitchen sink), ONLY MEMORY needed to launch it (good quality=model size on storage disk=memory size to fit good quality model). Hardware is just enterprise motherboard which supports up to 24(!) Ram modules, Gigabyte selling such for 20 years+, my used board costs me $150 after old servers recycling(it fits in standard PC case).
I've tested it on my only CPU Xeon (from year 2016) with 768Gb RAM and its not their best release, i think because of it size (1,6 terabytes, their previous models was much smaller) its their slowest model ever (ive tested all their models) even in quantized.
I've tested Unsloth UD Q6-K-XLARGE it uses 690Gb Ram in 13k context size. For comparison Kimi K2,7 on same hardware in UD Q8-K-XLARGE are x2,5 faster and uses ~602Gb Ram in 32k context. (info: Q6 and lower can be sloppy, Q8 is super high quality generated content/code/etc)
You can try new method in Colibri, which everywhere populated. Its a program for this model on CPU which just using your SSD size mostly than Ram (some Ram needed anyway). I'm not sure what quality is there, kinda it's full model by parameters but in size its just ~400Gb. I haven't tested it yet, people who tried reporting that its really works but speed very slow and depends on bandwidth with your SSD. I want to try it with only Ram, which i have a plenty. There's not many models in Colibri, it will be great if Kimi latest models could be supported there to compare speed.
#My Hardware# Intel Xeon E5-2699v4 LGA2011-3 22 cores 44 threads (2016) $110 # Gigabyte C612 chipset 12 RAM slots VGA motherboard year 2016 $150 # Samsung-Hynix ECC RAM 12x64Gb=768Gb ~$900 # VGA monitor # IKEA chair # NO GPU # Run: Trillions Deepseeks, Kimis in Q5-Q6, 400-500billions in BF16, super high quality 1 token/sec | testai.neocities.org
P.S.: btw, i have 3090($550 last year), you need many Ram anyway. Only one 3090 usable in my setup with 768Gb Ram to run such models in ik_llama, its super uber puper quality near original model(people claim best perplexity tests), its there needed this little 24Gb VRAM to even launch model in ik_llama. I've tried to start only on Ram but in ik_llama it crashed because Kimi there went above Terabyte territory to fit the model. I declined this method because of inability to fit 3090 inside PC case with enterprise motherboard, current Q8 quality is enough for me. (llama.cpp are different from ik_llama)
Indeed, when you are closer to 1Tb ram than "consumer" grade 32-64Gb, you can run hughe models :D
I also agree that memory is key., Colibri got that, delocalised ram access over network is also a big topic.
Super noice setup btw "old" hardware has a lot to offer ! As long as it's DDR3+ imho
Basic cpu and basic ram will freeze/OOM/fry if you try on a consumer grade machine.
Not true, people you need to stop being trapped in the "GPU thinking jail", you dont need GPUs to run primitive textual models (GPU usable only in video gen) you can run ANYTHING textual gen on ANCIENT hardware, because speed is NOTHING (moreover by running low quality fast on dozens GPUs = you're just getting tokens faster into trashbin, so = slop, analogy=its like producing trash or very fancy expensive food grinder under the kitchen sink), ONLY MEMORY needed to launch it (good quality=model size on storage disk=memory size to fit good quality model). Hardware is just enterprise motherboard which supports up to 24(!) Ram modules, Gigabyte selling such for 20 years+, my used board costs me $150 after old servers recycling(it fits in standard PC case).
I've tested it on my only CPU Xeon (from year 2016) with 768Gb RAM and its not their best release, i think because of it size (1,6 terabytes, their previous models was much smaller) its their slowest model ever (ive tested all their models) even in quantized.
I've tested Unsloth UD Q6-K-XLARGE it uses 690Gb Ram in 13k context size. For comparison Kimi K2,7 on same hardware in UD Q8-K-XLARGE are x2,5 faster and uses ~602Gb Ram in 32k context. (info: Q6 and lower can be sloppy, Q8 is super high quality generated content/code/etc)You can try new method in Colibri, which everywhere populated. Its a program for this model on CPU which just using your SSD size mostly than Ram (some Ram needed anyway). I'm not sure what quality is there, kinda it's full model by parameters but in size its just ~400Gb. I haven't tested it yet, people who tried reporting that its really works but speed very slow and depends on bandwidth with your SSD. I want to try it with only Ram, which i have a plenty. There's not many models in Colibri, it will be great if Kimi latest models could be supported there to compare speed.
#My Hardware# Intel Xeon E5-2699v4 LGA2011-3 22 cores 44 threads (2016) $110 # Gigabyte C612 chipset 12 RAM slots VGA motherboard year 2016 $150 # Samsung-Hynix ECC RAM 12x64Gb=768Gb ~$900 # VGA monitor # IKEA chair # NO GPU # Run: Trillions Deepseeks, Kimis in Q5-Q6, 400-500billions in BF16, super high quality 1 token/sec | testai.neocities.org
P.S.: btw, i have 3090($550 last year), you need many Ram anyway. Only one 3090 usable in my setup with 768Gb Ram to run such models in ik_llama, its super uber puper quality near original model(people claim best perplexity tests), its there needed this little 24Gb VRAM to even launch model in ik_llama. I've tried to start only on Ram but in ik_llama it crashed because Kimi there went above Terabyte territory to fit the model. I declined this method because of inability to fit 3090 inside PC case with enterprise motherboard, current Q8 quality is enough for me. (llama.cpp are different from ik_llama)
Thanks, well i will keep my mind open for whatever i decided to upgrade.