A 12GB iPhone 17 Pro Max running a 30B large language model and a 16GB iPad Pro running a 120B model: the developers of Inpherio Live report throughput of 10.9 and 4.5 tokens per second, respectively. Our judgment is that the significance of this approach lies not in speed, but in an engineering technique—streaming weights from flash memory—that works around the roughly 6GB per-app memory ceiling on iOS. This is not about shrinking cloud capabilities; it is about bringing the model back onto the device itself.
What this is
Mixture-of-Experts (MoE) is a model architecture with a large total parameter count in which only a small subset of expert networks is activated for each token. On the iPhone, the developers modified llama.cpp, the open-source inference engine, to read weights directly from flash memory on demand while keeping only active experts in memory. This is the key to fitting large models onto a phone: the same method does not work for conventional dense models, which use every parameter for each token.
The product takes the form of an AI server on a local network, exposing OpenAI-compatible endpoints (/v1/chat/completions, /v1/responses, /v1/embeddings, and /v1/audio/*) as well as Ollama-compatible interfaces. Existing tools can connect simply by changing the base URL. It also supports local vision models, including Qwen2.5-VL, Gemma 3, and SmolVLM2, along with Whisper speech-to-text and Kokoro speech synthesis. Its memory manager loads models on demand and keeps up to three models warm, reducing the risk of the system terminating the process. There are no accounts, no telemetry, and zero bytes uploaded to the cloud.
Industry view
Notably, the interface layer is OpenAI-compatible rather than a proprietary protocol. Compatibility layers are a rare pragmatic choice in the local AI ecosystem: they mean that LangChain, IDE plugins, and scripting tools can connect with little or no modification, lowering the barrier to entry from configuring a CUDA environment to entering a URL. Speed is not the decisive concern. A rate of 4.5 tokens per second for the 120B model is usable but not particularly responsive, while 58.3 tokens per second for a 1.5B model indicates that the experience is sufficient for everyday small tasks.
The risks are real. First, the app is still awaiting Apple review, and its fate remains uncertain. App Store policy on applications that bundle their own large-model runtime has historically fluctuated, with comparable apps being rejected or repeatedly delayed. Second, sustained flash-memory reads raise power-consumption and thermal concerns. A phone operating as a long-running online server faces hard constraints in battery life and thermal management. The official benchmarks explicitly note that reads come from storage, which points to I/O rather than compute as the bottleneck; there is not yet enough data to establish stability under prolonged heavy loads. Third, the business model is awkward: small models that fit comfortably in memory, embeddings, and speech features are all free, while streamed large-model access requires a one-time payment. Whether users will pay to turn a phone into a server is itself an unproven proposition. The developers date their benchmarks to November 2026. Because these are self-reported results on a single device model, without independent reproduction, we view them as evidence of engineering feasibility rather than a performance verdict.
Impact on regular people
For enterprise IT: Compliance-sensitive material no longer has to leave the internal network. What previously required purchasing servers and deploying operational staff may now require little more than an old iPad. Real-world deployment will require evaluating device wear, concurrent capacity, and operational standardization.
For individual professionals: Document processing, meeting transcription, translation, and similar tasks can be completed locally on a phone without uploading sensitive content to the cloud. That depends on the app clearing review and on users tolerating its battery drain and heat generation.
For the consumer market: AI features that work offline are shifting from a selling point to a default expectation. If the next major smartphone launch emphasizes local inference rather than benchmark scores, this news may become a reference point before that product even arrives.