Every GPU in the room.
One inference engine.
Community GPU Pool turns a team's gaming laptops, MacBooks, and Linux workstations into a private, hop-minimal, pipeline-parallel LLM cluster — on the network you already have. No per-token API bills. No new hardware.
PyTorch · llama.cpp · MLX — wired Ethernet only
The obstacles are on the network side, not the compute side.
Four machines on one desk already hold more VRAM than the model needs. Getting them to cooperate is the hard part — and it has nothing to do with FLOPs.
Every layer boundary is a hop
Every token traverses every hop. Naive layer splitting on a LAN destroys interactive latency before it ever helps throughput.
VRAM varies 4–10×
A 4090, an M3 Max, and a 1660 Ti are not peers. Even splitting is the worst possible packing for a heterogeneous pool.
Adding nodes makes it slower
In a naive pipeline, every new node is a permanent extra hop. The cluster gets bigger and every request gets slower, forever.
Laptops leave the building
A lid closes, a Wi-Fi lease expires, a student walks out. The system must survive losing any node without losing a single committed token.
Treat round-trip cost as the scarce resource.
Two nodes used well (1 hop) beats four nodes evenly split (3 hops). Every design decision below follows from that single line.
Hop-minimal packing
Greedily fill the strongest node first, spill the remainder to the next strongest. If it fits on one node, hops = 0 and the scheduler serves locally.
Speculative decoding, locally
A tiny 1B draft model on the coordinator proposes k = 4–6 tokens at zero network cost. The full pipeline verifies all of them in a single traversal.
Scale by replication, not depth
More nodes become more independent pipelines behind the router — not a longer chain. Depth is reserved for models that literally cannot fit any other way.
One committed token log
The coordinator holds every committed token for every sequence. Recovery and migration are just a re-prefill of that log — never a restart.
Play with the planner.
This is a real port of
plan(nodes, model)
— the same pure, strongest-first greedy packer that runs on the
coordinator. Toggle nodes, change the model, watch hops happen.
Target model
Node pool
Two axes, opposite trades.
The first question on node join is never "how do we reshard?" It is: do we need capacity, or throughput?
Split layers across more nodes
You get more VRAM for a bigger model. You pay for it on every single token, for every single request, forever.
- +1 hop per node, permanently
- Bigger model fits
- Iteration time = slowest stage
- Only when the model fits no other way
Duplicate the whole pipeline
You get near-linear throughput for the cost of one more copy of the weights. The router sends each request to the shorter queue.
- Zero additional hops
- ~Linear throughput gain
- Isolated failure domains
- The default outcome of plan()
Small enough to read. Careful enough to trust.
Every subsystem is deliberately the cheapest thing that is still correct at team scale.
Continuous batching
Requests from the whole team are queued and processed concurrently. While node B computes user X's token, node A computes user Y's. Admission is bounded by the tightest stage's KV headroom.
Speculative decoding
A local 1B draft proposes k tokens with zero network cost. Verification is memory-bandwidth-bound, so checking 6 tokens costs ~1.3× checking 1 — one round-trip, 4–6 committed tokens.
Argmax-only return
The last stage returns argmax, not logits. For a 128k vocab that's 224 bytes instead of 14 MB per iteration — a ~60,000× reduction on the return path. Greedy verification needs nothing more.
Token-log recovery
No committed token ever lives only on a worker. When a node dies, survivors evict and reload their new ranges, then re-prefill each sequence's committed log. Seconds at 8k context.
A pure planner
plan()
is a side-effect-free function. Rollback, dry-run, and join
policy all fall out of that one property — there is no state to
unwind.
Drain-then-form
A LIVE pipeline is never mutated. Reconfiguration means drain, wait for zero in-flight, plan over freed nodes, then warm. Live pipelines stay live the whole time.
Twelve weeks to a working pool.
The pipeline abstraction lands at week 9 — after speculative decoding already works on one pipeline. Then the refactor wraps working code instead of debugging a design.
Transport + registry
gRPC forward(), zero-copy tensors, invite-token enrollment, heartbeat.
Single pipeline
greedy_pack, layer sharding, KV buffers, truncate/evict, per-stage timing logs.
Speculative decoding
Local draft, k-curve measurement, acceptance rate, lm_head profiling.
Replication + migration
Router, planner, drain-then-form, cross-pipeline recovery, admin API.
Your team already owns the hardware.
Join the private beta and stop paying per token for models your own machines could serve.