[{"data":1,"prerenderedAt":78},["ShallowReactive",2],{"term-o\u002Fon-device-inference":3,"related-o\u002Fon-device-inference":60},{"id":4,"title":5,"acronym":6,"body":7,"category":40,"description":41,"difficulty":42,"extension":43,"letter":44,"meta":45,"navigation":46,"path":47,"related":48,"seo":54,"sitemap":55,"stem":58,"subcategory":6,"__hash__":59},"terms\u002Fterms\u002Fo\u002Fon-device-inference.md","On-Device Inference",null,{"type":8,"value":9,"toc":33},"minimark",[10,15,19,23,26,30],[11,12,14],"h2",{"id":13},"eli5-the-vibe-check","ELI5 - The Vibe Check",[16,17,18],"p",{},"On-device inference means the AI model runs right there on your phone or laptop instead of phoning home to a data center somewhere. No network round trip, no data leaving the device, and it still works on a plane with no wifi. The tradeoff is you're stuck with whatever compute and battery the device actually has, so the model has to be small and efficient.",[11,20,22],{"id":21},"real-talk","Real Talk",[16,24,25],{},"On-device inference runs a model locally on consumer or edge hardware (phones, laptops, IoT devices) rather than sending requests to a remote server. It requires aggressively quantized or distilled models sized to fit the device's memory and compute budget, using runtimes like Core ML, TensorFlow Lite, ONNX Runtime, or MLX. Benefits include zero network latency, offline availability, and data never leaving the device.",[11,27,29],{"id":28},"when-youll-hear-this","When You'll Hear This",[16,31,32],{},"\"We shrank the model to 4 bits so it fits for on-device inference on mid-range phones.\" \u002F \"On-device inference solved our GDPR data residency headache entirely.\"",{"title":34,"searchDepth":35,"depth":35,"links":36},"",2,[37,38,39],{"id":13,"depth":35,"text":14},{"id":21,"depth":35,"text":22},{"id":28,"depth":35,"text":29},"ai","On-device inference means the AI model runs right there on your phone or laptop instead of phoning home to a data center somewhere.","intermediate","md","o",{},true,"\u002Fterms\u002Fo\u002Fon-device-inference",[49,50,51,52,53],"Quantization","Ollama","GGUF","Local AI","Distillation",{"title":5,"description":41},{"changefreq":56,"priority":57},"weekly",0.7,"terms\u002Fo\u002Fon-device-inference","omR6k46rATdJDXv3h35Lu6PdlXaPMX7X3YAeXB51H1s",[61,65,68,72,75],{"title":53,"path":62,"acronym":6,"category":40,"difficulty":63,"description":64},"\u002Fterms\u002Fd\u002Fdistillation","advanced","Teaching a small AI by having it copy a big AI's homework.",{"title":51,"path":66,"acronym":6,"category":40,"difficulty":63,"description":67},"\u002Fterms\u002Fg\u002Fgguf","GGUF is a file format for running AI models on your laptop — it's like the MP3 of AI models.",{"title":52,"path":69,"acronym":6,"category":40,"difficulty":70,"description":71},"\u002Fterms\u002Fl\u002Flocal-ai","beginner","Local AI means running AI models on your own computer instead of sending data to the cloud.",{"title":50,"path":73,"acronym":6,"category":40,"difficulty":70,"description":74},"\u002Fterms\u002Fo\u002Follama","Ollama is Docker for AI models. One command downloads and runs any open-source AI model on your computer.",{"title":49,"path":76,"acronym":6,"category":40,"difficulty":63,"description":77},"\u002Fterms\u002Fq\u002Fquantization","Quantization is the art of making AI models smaller and faster by using less precise numbers.",1792528104747]