Bonsai-27B on recursive llama.cpp fork: full setup and real benchmark on RTX 3050
cubetitled-ui
This third article about a recursive llama.cpp fork covers running the hybrid Bonsai-27B model (Delta-Net linear architecture) on a weak GPU, showing a real benchmark of depth D=0 vs D=12 on a cryptarithmetic task. With recurrence enabled (D=12), the model solved SEND+MORE=MONEY, while the base engine failed. Setup commands and recurrence environment variables are provided.
The author presents the third part of their series on a recursive fork of llama.cpp, focusing on Bonsai-27B, a hybrid model with linear Delta-Net layers and full-attention layers. Recurrence is applied only to full-attention layers. The model in Q1_0 quantization is about 3.5 GB, fitting into 6 GB VRAM. The article provides build instructions for CUDA and CPU, and usage commands with environment variables RECURRENT_D (depth: 0, 12, 24), RECURRENT_LAYERS_COUNT, and RECURRENT_KV. A real benchmark on an RTX 3050 Mobile 6 GB with seed 42 and temperature 0.3 was run on the cryptarithmetic puzzle SEND+MORE=MONEY, with a 7000-token limit. D=0 generated 7000 tokens without solving, hitting the length limit (finish_reason: length). D=12 generated 6594 tokens and correctly solved it with S=9, E=5, N=6, D=7, M=1, O=0, R=8, Y=2, verifying 9567+1085=10652, stopping properly (finish_reason: stop). Generation time was about 5.5 minutes for both, with speeds of 21.1 t/s (D=0) and 19.8 t/s (D=12), a ~6% slowdown. The article notes that recurrence does not change weights, only the inference computation graph, and lists compatible models including Qwen3.5, Qwen2, Gemma 2, Mistral 3, and Delta-Net (Bonsai).
- Abbreviations
- GPU = Graphics Processing Unit — графический процессор
- CUDA = Compute Unified Device Architecture — архитектура параллельных вычислений NVIDIA
- CPU = Central Processing Unit — центральный процессор
- VRAM = Video Random Access Memory — видеопамять
- CLI = Command-Line Interface — интерфейс командной строки
- GGUF = GPT-Generated Unified Format — формат файлов моделей llama.cpp
Source: Habr — хаб ИИ —
original