0xSero ✓

[text] Breakthroughs: 1. Turboquant merged into vLLM 75% vram reduction for kvcache near losslsss 2. Someone merged M2.7 & M2.5 & got it to perform better than M2.7 3. 40% faster prefill on AMD strix halo (128gb for MoEs w 10B active params) 4. Megatrain 100B model trained on 1 GPU

[image] Locally Part 1 - Apple Silicon Macs give you large pools of memory to run big models, but the token generation speed will be lower than most are used to. Macs are best with large MoEs that have low ACTIVE params. Basically when you see a model like Qwen3.5-397B-A17B this

[text] Here’s what I’d recommend if you’re just getting started in AI, local or otherwise. 1. Work with the compute you have, even the dumbest LLMs can be useful if you treat them as a node in your system. Some basic problems of what could be useful to get you started - tag all

[image] Best harnesses for local models: 1. Droid: - Very good performance, forces the models to behave, you can wire in all your local LLMs very easily w BYOK - Allows you to use your local models as orchestrators/subagents so you can benefit from Cloud as models as well - Practically

[text] Help me spread this, I am on a role and need to squeeze every last bit out of it.

[image] Guide to running BIG B0Is on your small hardware. 1. Use REAPs: up to 50% savings 2. Use quantisations: 75% savings - AWQ / GPTQ / W4A16 / FP8 = FAST inference - GGUF / EXL3 = Slow but just works - MLX = Best for apple 3. Use 8bit KV cache: 50-75% savings

[text] Local AI is a human right, our children, families, neighbours, friends, and fellow humans deserve privacy and freedom.

[text] I meant to post this in here, I will be posting weekly models meant for limited hardware budgets, RN I am learning how to deal with the 30b~ class