Edge0: Open-Source Framework Running 35B LLMs On-Device with 1–2.5GB RAM

A notable open-source project has emerged in the on-device AI ecosystem, offering developers the ability to run large language models locally without relying on remote cloud infrastructure or massive desktop GPU rigs. Developed by Samuel Zeng and the Edge0 team, the newly released Edge0 framework enables local inference of models scaling up to 35 billion parameters while maintaining a peak memory footprint of just 1 to 2.5 gigabytes on personal hardware.

Image source: Tony Simons (@tonysimons_) via X

For most developers and machine learning practitioners, running a 30B+ parameter model has historically demanded substantial dedicated VRAM or high-capacity unified memory configurations. Edge0 aims to lower this steep barrier to entry, eliminating recurring cloud API expenses and mitigating sensitive data privacy risks by running large models entirely within resource-constrained local environments.

Ultra-Low Memory Architecture for 35B Models

The central technical milestone introduced by the Edge0 framework is its ability to reliably execute inference on a 35B parameter model within a tight peak memory boundary of 1 to 2.5GB.

Typically, even heavily quantized 4-bit or 8-bit variants of 30B-class models require tens of gigabytes of RAM to reside in memory. Rather than loading an entire model weight matrix into system memory all at once, Edge0 focuses on a lightweight execution and memory-streaming pipeline engineered for extreme memory conservation.

• Minimal Memory Footprint: By actively managing memory usage and capping peak allocation at 1 to 2.5GB during 35B model execution, the runtime prevents Out-Of-Memory (OOM) failures on devices with limited available RAM.
• Data Privacy and Zero Cloud Overhead: Since all computation executes self-contained on the local hardware, teams can deploy models without transmitting proprietary prompts over the network or incurring per-token API costs from commercial providers.

This lightweight approach offers an attractive alternative for developers seeking to build fully private, offline-first local intelligence workflows without enterprise-scale compute clusters.

Apple Silicon Integration and MLX Backend Support

The initial open-source release of Edge0 has been tailored primarily for Apple Silicon personal computing hardware.

According to its official GitHub repository (Edge0-AI/edge0), Edge0 provides native support for Apple's MLX machine learning framework. This allows Mac systems powered by M1, M2, M3, and M4 processors to leverage their unified memory architecture directly for local inference workloads.

• Optimized for macOS and MLX: By interfacing with the MLX framework, Edge0 efficiently passes and processes model weights within Apple Silicon's shared memory architecture.
• Accessible Local Experimentation: Developers can test, iterate, and integrate large model pipelines directly on standard Apple Silicon laptops and workstations without configuring external GPU accelerators.

While public demonstration videos and social announcements showcased on-device execution on mobile form factors such as the iPhone, the currently published repository codebase explicitly prioritizes the macOS environment and its MLX backend.

Practical Trade-Offs, Performance Latency, and Current Constraints

Before integrating Edge0 into production engineering or research pipelines, teams should evaluate several essential technical trade-offs inherent to its architectural approach.

• Inference Latency and Throughput (TPS): Running a 35B model within 1 to 2.5GB of peak memory requires aggressive layer-wise weight streaming, aggressive quantization, or dynamic memory swapping. Consequently, token generation speed (Tokens Per Second) and initial time-to-first-token will naturally experience noticeable overhead compared to high-end systems where the entire model resides unconstrained in high-bandwidth VRAM.
• Platform Scope: The initial release is specifically designed for Apple Silicon macOS via MLX. Standard x86_64 Linux distributions, Windows systems, and Android mobile devices are not currently supported in this release.
• Mobile Deployment Readiness: Deploying Edge0 to iOS mobile targets—as previewed in early demonstrations—will require monitoring subsequent releases and verifying additional mobile build procedures as they become officially available.

By demonstrating that large 35B models can be brought down to a 1–2.5GB memory envelope, Edge0 marks a significant proof-of-concept for accessible, local-first artificial intelligence on personal hardware.

Sources

• GitHub Repository: Edge0-AI/edge0 — Official open-source repository for the Edge0 on-device AI inference framework
• Social Announcement: Tony Simons (@tonysimons_) via X — Announcement and demonstration of Edge0 running a 35B model with 1–2.5GB peak memory