Need gguf!

#1
by drmcbride - opened

@drmcbride just wait a few days till unsloth cook with their quants haha >.<

What can it run on? Will 64 DDR and 12 GPUs be enough?

64GB of what DDR?

and what are your 12 GPUs?

with fast DDR4 or decent DDR5, and 12 RTX 5070 (12 x 12GB = 144GB) = 64GB + 144GB = 2008GB of VRAM the model is absolutely runnable at max context at Q6 which would approximately total to about 172.3 GB total RAM needed.

hope that helps! :)

I have an RTX 3060 12 GB and 32 GB of RAM, and the 85 GB IQ2_S/IQ2_XXS file from AtomicChat/Hy4-flash-GGUF writes 4.3 to 5.1 tokens/s here. Slow, but the full model answers on a 12 GB card.
I ran it with RustingStrata, my own engine: https://github.com/RustingStudio/RustingStrata
Curious what 64 GB gives. I can't test that. Has anyone tried with llama.cpp or something like that?

are you running the model directly from the SSD (or at least partially)? @rustingstudio

@mithulaartigala if very shortly, yes I am, I hold most used experts in VRAM and RAM and leave most unused on SSD

ohh wow thats cool. I will definitely give it a try! <3

Sign up or log in to comment