x/LocalLLaMA

[image] You don’t “run a model” You run Kernels The model is just a graph The Inference Engine is scheduler / optimizer / executor But the actual work? That happens in the Kernels - MatMul Kernels - Attention Kernels - RMSNorm Kernels - KV cache Kernels - Quantized linear Kernels -

[video] Qwen 3.8 Flash Next running on my Android phone (12GB RAM) Bigmoeonedge project enables running massive models on edge devices - such as a mid-range Android phone with 12GB of RAM. Qwen 3.8 Flash Next is now supported by Bigmoeonedge project, just hours after its launch. The

[text] 1/ 🧵 I just cracked open the Claude Code source — and what I found isn’t “just a smarter terminal chat.” It’s a full-blown behavioral observatory running in your machine. 1. Keyword sniffers. 2. Hesitation trackers. 3. Hidden trigger words. 4. Telemetry that fingerprints

[image] Just some numbers so you don’t get misled RTX 3090 (7 years old) > 24GB VRAM > Bandwidth: 936.2 GB/s > Bi-directional NVLink 112GB/s RTX PRO 4000 > 24GB VRAM > Bandwidth: 672 GB/s > No Bi-directional NVLink, > need 32 Gen. 5 PCIe Lanes to pool 2 at 64GB/s

[image] How am I connecting the DGX Spark cluster > Mikrotik CRS804-4DDQ 1.6Tbps switch > (4) 400G QSFP-DD to 2x 200G QSFP56 Each DGX Spark has a ConnectX-7 supporting 200Gbps Each cable out of the switch goes into 2 DGX Sparks This allows 8x DGX Sparks cluster at the full 1.6Tbps

[image] X is killing x/LocalLLaMA Community tonight We built the new permanent home - links DOT theahmadosman DOT com/discord-server That's where you go now

[image] okay let me say this out loud again. if you want to run local models on a single RTX 3090, your best option right now is qwen 3.5 27B dense Q4_K_M. 35 tok/s, flat from 4K to 300K+ context, zero speed degradation. thinking mode works. 262K native context on 24GB. slower than MoE

[text] Most people think VRAM = model size and that’s why their runs crash GPU memory math is complex and so are the implications Here’s how it actually works in a nutshell ↓

[image] ~2,000 miles from home - SSH into the mothership - WireGuard tunnel like I’m on LAN - Resolve everything over private DNS - Routes via reverse proxies - Dispatch jobs to GPUs - Sync state across agents - Search conversations, traces, artifacts, logs Read how below

[image] Which one of you is this hahaha

[text] What am I working on? Condensing everything I do into one place: - local AI / LLMs - inference + benchmarking - hardware + cluster builds - LLM research + notes - agent workflows - real-world perf (tokens/sec, concurrency, thermals) All into a single, searchable, indexable

[image] Bay Area folks, I’m putting together a casual get-together in San Francisco this Saturday, May 9th Pull up to talk local AI, GPUs, infra, opensource, agents, homelabs, and whatever else the GPU-poisoned brain wants to discuss Limited capacity, first come first served

[text] Help me spread this, I am on a role and need to squeeze every last bit out of it.

[image] which one of you is this?

[video] New quick video, more coming up Goal isn't just to show how large models perform but also to show how small and medium models scale up as you multiply # of nodes and how Tensor Parallelism performs on Unified Memory hardware

[image] RTX PRO 6000 (96GB VRAM, ~$15K) GIVEAWAY FAQ Q: Cost to enter? A: $0. Free. Q: Do I have to register for GTC? A: Yes, virtual attendance is COMPLETELY FREE Q: Where do I enter? A: Tap the link in my bio, there’s a clear button on the page Q: How do I increase my chances? A:

[text] Tip of the day Always tell your agent to think through the plan comprehensively covering any and all relevant things, and using best practices and up-to-date information as of April 13th, 2026 (replace with today's date)

[image] Introducing MOTIF-3-alis-mlx-4.5bpw. This build maximizes speed on MLX platforms with 256GB or more without significantly compromising quality. Based on M3 Ultra, it achieves a prefill speed of 790 tok/s and a decode speed (with MTP) of 50 tok/s. *MOTIF-3 is currently the

[image] Local AI was never this EASY > Install ODS > Let it detect your hardware > It will download the best model for your hardware > And then start local inference and Open WebUI for you With ODS, you can > Add voice, agents like Hermes, workflows, RAG, search, image generation, and

[image] Qwen3.8-Flash-Next on one RTX 5090 after FreeToken #257 decode 65.3 tok/s · prefill 3,580 tok/s @ 37K same harness: 39.6→65.3 (1.65×), 2,191→3,580 (1.63×) 512 experts, ~92GB in host RAM over PCIe 196GB = memory assigned to WSL2, not the whole box

[text] I meant to post this in here, I will be posting weekly models meant for limited hardware budgets, RN I am learning how to deal with the 30b~ class

[video] local ai crowd would appreciate it more than most. two of the best open models that fit one dgx spark, qwen 3.5 122b vs laguna s 2.1, both nvfp4, same prompt at the same second. both one shotted a playable game, but in completely opposite ways, one planned and verified every

[image] it's a good day sir! 23.5 TB of VRAM. 288 GPUs. 117.3 kW. does this still count as local AI or is local just a state of mind now? lol

[text] On an 8GB Jetson, SmolLM2-135M came out around 165 tok/s and 29.6 tok/J at 25W. That’s the kind of number I wanted a public board for small open models, same GGUFs, llama.cpp vs Ollama, power modes locked, heat logged. Writeup on the problem, the protocol, and what’s live and

[text] Looking for gpu owners and gamers Fren built @openmayhem_ai Local First + Decentralized ethos at the center— Become a provider and earn - compete with data centers