Local AI

[video] Local AI is about to be competitive. I will do everything in my power to make it better than Claude desktop/claude code by end of year.

[image] Important: This is a summary of an amazing video by one of the best creators I know. Video in the first comment. If you've been struggling to setup a productive local environment I'll summarise, but you should watch the video. 1. Qwen3.6-27B with NO THINKING - 4bit - 16bit

[image] Weekly best models for your hardware: ~~ 8 to 16gb ~~ Granite models are amazing: [NEW] - Gemma-E4B is a good general QA model - Qwen3.5-9B is the best at this level imo - ~~ 16 to 64gb ~~ Another

[text] Breakthroughs: 1. Turboquant merged into vLLM 75% vram reduction for kvcache near losslsss 2. Someone merged M2.7 & M2.5 & got it to perform better than M2.7 3. 40% faster prefill on AMD strix halo (128gb for MoEs w 10B active params) 4. Megatrain 100B model trained on 1 GPU

[image] Best models for your hardware - 4gb to 12gb vram - VibeThinker-3B - smokes everything remotely close to its weight class. Challenging 30b models! Last version was also topping math benchmarks - 12gb to 24gb vram - Gemma-12B-coder Built on top of

[image] Locally Part 1 - Apple Silicon Macs give you large pools of memory to run big models, but the token generation speed will be lower than most are used to. Macs are best with large MoEs that have low ACTIVE params. Basically when you see a model like Qwen3.5-397B-A17B this

[video] To hell with Anthropic and their random policy changes. AI local is the future. This game was created using DeepSeek v4 entirely locally. The setup is not for the faint-hearted currently. DeepSeek v4 (came out a couple of days ago) GGUF 2-bit quant and llama.cpp patch from

[video] local LLMs on Mac are getting good. but the workflow around models is still messy. you have models in: •⁠ ⁠LM Studio •⁠ ⁠Ollama •⁠ ⁠Hugging Face cache •⁠ ⁠random folders •⁠ ⁠manual downloads and every runtime expects something slightly different. we built

[text] Here’s what I’d recommend if you’re just getting started in AI, local or otherwise. 1. Work with the compute you have, even the dumbest LLMs can be useful if you treat them as a node in your system. Some basic problems of what could be useful to get you started - tag all

[image] Cheapest competitive build on Nvidia 2 sparks = 8000$ Total specs - 256gb - 8tb - 546gb/s memory bandwidth - tons of flops --- models: - Deepseek-v4-flash - MiMo-v2.5-flash fp4 - MiniMax-M2.7 - Qwen3.5-397b-reap Flaws: - Low mem bandwidth - You need 2 for best perf

[image] Best harnesses for local models: 1. Droid: - Very good performance, forces the models to behave, you can wire in all your local LLMs very easily w BYOK - Allows you to use your local models as orchestrators/subagents so you can benefit from Cloud as models as well - Practically

[image] One thing I've done this year is: - Download all my X data from settings/account - Download all my youtube, gmaps, gmail, google from takout google com - Download all my personal data from Claude, ChatGPT - Export a copy of every AI session on Cursor Claude Code, Codex, Droid,

[image] Based on my knowledge of: - AI labs - hardware manufacturers - model size performance - training process - training economics If you want to buy hardware for local AI you need 500-750gb of 500gb/s+ memory bandwidth to make serious use of local hardware at frontier levels

[image] To my fellow homelab friends, I need some help. My room is starting to cook, so I need to setup air conditioning (I already have a portable one but need something more pro) Essentially the rooms humidity is quite low and given I have cats there’s a lot of dust I need to clean

[image] Guide to running BIG B0Is on your small hardware. 1. Use REAPs: up to 50% savings 2. Use quantisations: 75% savings - AWQ / GPTQ / W4A16 / FP8 = FAST inference - GGUF / EXL3 = Slow but just works - MLX = Best for apple 3. Use 8bit KV cache: 50-75% savings

[image] Top 5 builds for AI inference in 2025-2026 I have spent around 12 months researching, building, experimenting, and bench-marking AI models, tools, hardware and costs. Top 3 picks will be the safest, best cost to performance ratios. The last 2 will be more interesting

[image] I got 2 intel bad boys on Friday. At this point I’m struggling to find more power for all this. - 4x 6000s - 1x DGX Spark - AMD Strix - 4x 3090 - 2x intel arc b70 - Mac mini 16gb - MacBook Pro 32gb 544gb VRAM 300gb mixed Total = 844gb AI + 512gb ddr4 I need some guru help

[image] Why I built vllm-studio - Storing configurations for models and engines - Fully integrated agent with Pi - Model downloading, exploration, and config - Creating benchmark datasets Backend can be deployed to multiple servers, and the frontend can connect to all from anywhere.

[text] Local AI is a human right, our children, families, neighbours, friends, and fellow humans deserve privacy and freedom.