Prism has shattered previous on-device limitations with Bonsai, a massive 27-billion-parameter language model designed to run efficiently on smartphones. This breakthrough, detailed in a company announcement, uses a novel hybrid architecture and aggressive quantization to deliver near-70B model performance right in your pocket.
A New Architecture for the Edge
Unlike traditional monolithic models, Bonsai employs a unique 'Compound Mixture-of-Experts' (CompoundMoE) architecture. This design combines a smaller, dense "trunk" model (3B parameters) with larger, specialized "branch" models. The trunk provides a foundation of general knowledge, while the branches, which are Mixture-of-Experts (MoE) layers, handle more specialized reasoning tasks.
This hybrid approach, as explained by Prism, allows Bonsai to activate only the necessary "experts" for a given query. This significantly reduces the computational load compared to running a full 27B dense model, making it feasible for the limited processing power and memory of a mobile device.
Performance that Punches Up
Bonsai's efficiency doesn't come at the cost of performance. Prism's internal benchmarks show the model not only surpasses leading 8B models but also competes directly with behemoths like Llama 3 70B on several reasoning and knowledge tasks. This achievement is made possible through advanced compression techniques.
Key performance metrics include:
- Model Size: The entire model is quantized down to 2.6 bits, resulting in a compact file size of just 3.7GB.
- On-Device Speed: It runs at an impressive 20 tokens per second on a Google Pixel 8 Pro using the popular llama.cpp inference engine.
- Benchmark Strength: It reportedly outperforms Llama 3 8B and is competitive with Llama 3 70B on evaluations like MMLU and GSM8K.
Staying on top of developments in model optimization and on-device deployment is critical for developers and AI professionals. For weekly analysis of breakthroughs like Bonsai, consider subscribing to the AI Breaking Wire newsletter to get expert insights delivered to your inbox.
The Technology Behind the Breakthrough
The magic behind fitting such a large model onto a phone lies in aggressive post-training quantization. Quantization is a process that reduces the precision of a model's numerical weights, drastically shrinking its memory footprint and speeding up computation. Bonsai's 2.6-bit quantization is a significant step forward, enabling its deployment without requiring specialized hardware.
By building on the open-source llama.cpp framework, Prism ensures that Bonsai can be run and tested by a wider community of developers. This accessibility is key to fostering innovation and exploring new use cases for powerful, private, on-device AI.