Inference of LLM on a Cluster of Old Laptops

An experiment was conducted to test distributed LLM inference on a cluster of three old laptops with Intel CPUs from different generations: Core i7-4700MQ (Haswell, 2013), Core i5-8265U (Whiskey Lake, 2018), and Core i3-1115G4 (Tiger Lake, 2020), totaling 52 GB of RAM. The software used was llama.cpp (build b10069) with a RPC server on each node, and the nodes were connected via gigabit Ethernet. The models tested were Llama-3.2-1B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-32B-Instruct (all Q4_K_M), and gpt-oss-20b (MoE). For the 1B model, the cluster (i3+i5) gave 22.5 t/s token generation vs 29.2 t/s on the best solo node (i3), a 23% slowdown. For the 8B model, the cluster was nearly at parity with solo (4.6 vs 4.9 t/s), but prompt processing was almost 3x faster (10.9 vs 3.9 t/s). For the 32B model, which did not fit in any single node, the cluster achieved 1.1 t/s versus 0.3 t/s on a node with swapping, a 268% improvement. With the MoE model gpt-oss-20b, which also did not fit on a single node, the cluster gave stable 8.0-8.1 t/s generation and prompt eval of 15.4-15.9 t/s, compared to unstable solo performance. The conclusion is that distributed inference is only justified when the model does not fit into one node; otherwise, a single fast node is better. Also, prompt processing benefits from parallelism, while token generation does not, and any weak node can bottleneck the cluster.