AirLLM: Lossless FP16 70B Inference on 4GB VRAM and Mac mini via NVMe Streaming
An overview of AirLLM, an open-source tool that streams 70B LLMs layer-by-layer from NVMe SSDs to run unquantized FP16 weights on 4GB VRAM GPUs and base Mac min
#VRAM
TAU-HOME.COM stories tagged VRAM.