A camera on a factory floor spots a hairline crack in a metal bracket. It classifies the defect, tags it, and triggers an alert to the line supervisor. Total time from pixel capture to decision: 4 milliseconds. The internet was never involved. The cloud was never consulted. The entire inference happened on a chip smaller than a postage stamp, drawing less power than a phone charger.
That is edge AI. And honestly, it took us a while to pay attention.
We are a web development agency. At CODERCOPS, most of our days involve React components, API integrations, database queries, and deployment pipelines. Machine learning on microcontrollers was not exactly on our radar. But last year we took on a project for a client running an industrial monitoring setup -- they needed a dashboard to visualize sensor data streaming from about 200 devices across three warehouse locations. During that engagement, we had to understand what was happening before the data hit our dashboard. That is when edge AI clicked for us. The devices were not just collecting raw numbers. They were running tiny classification models locally, sending only anomalies and summaries upstream. The amount of bandwidth and compute that saved was staggering.
So we went down the rabbit hole. This post is what we found.
What Edge AI Actually Means
Strip away the marketing and it is straightforward: edge AI is machine learning inference that happens on or near the device that generates the data, instead of sending that data to a remote server for processing.
Your phone does it when it recognizes your face. Your car does it when the lane departure warning fires. A security camera does it when it distinguishes between a person and a shadow.
The "edge" just means the point closest to where data originates -- a sensor, a camera, a microphone, a wearable. Instead of shipping raw sensor readings to AWS, processing them in some GPU instance, and sending results back, you run a small model right there on the device. The result is available in single-digit milliseconds instead of hundreds.
This is not a new idea. But in 2026, the hardware and software have matured enough that it is genuinely practical for a wide range of applications. And the reasons to do it keep multiplying.
Why Edge AI Is Exploding Right Now
Three things converged.
Chips got good enough. Five years ago, running a meaningful neural network on a microcontroller was a research project. Now you can buy a $25 board that does 4 TOPS (trillion operations per second) of neural network inference. The Raspberry Pi 5 with the AI HAT+ accessory pushes 13 TOPS for under $100 total. NVIDIA's Jetson Orin Nano does 40 TOPS. These are real, shipping products you can order today.
Models got small enough. Techniques like quantization, pruning, and knowledge distillation have made it possible to shrink models dramatically. A MobileNetV3 image classifier quantized to INT8 can be under 2 MB. That fits comfortably in the flash memory of a microcontroller. Google's MediaPipe can run pose estimation at 30fps on a phone CPU. You do not need a data center for this.
Privacy and compliance forced the issue. GDPR, India's DPDP Act, HIPAA -- they all make it increasingly expensive and risky to ship raw data (especially video, audio, or biometric data) to cloud servers. If the inference happens on-device and only results leave the device, your compliance surface shrinks massively. A factory camera that detects defects locally and only sends "defect found at timestamp X" is a fundamentally different compliance story than one that streams video to a cloud endpoint.
And there is a fourth, quieter reason: cost. Cloud inference is not free. If you have 10,000 cameras each making 30 inference calls per second, you are looking at 26 billion API calls per day. Even at fractions of a cent per call, that number gets ugly fast. Running inference on-device costs you the hardware once and electricity thereafter.
The Hardware -- What You Can Actually Buy
This is where things get interesting. The hardware options in 2026 are remarkably diverse.
NVIDIA Jetson Orin Series
The workhorse of serious edge AI deployments. The Jetson Orin Nano delivers 40 TOPS of AI performance in a module roughly the size of a credit card. The full Jetson AGX Orin pushes 275 TOPS. These run a full Linux stack, support CUDA, and can handle multiple concurrent models -- object detection, classification, and tracking simultaneously.
- Jetson Orin Nano: 40 TOPS, 15W power, ~$249
- Jetson AGX Orin: 275 TOPS, 60W power, ~$999 (developer kit)
- Full CUDA and TensorRT support
- Can run small transformer models, not just CNNs
We have seen clients use these for real-time quality inspection on production lines. One setup runs YOLOv8 at 45fps on 1080p video for weld defect detection. Try doing that with a Raspberry Pi.
Raspberry Pi 5 + AI HAT+
The Pi 5 itself is a general-purpose board, but add the Hailo-8L based AI HAT+ and you get 13 TOPS of dedicated neural network acceleration for about $70 on top of the Pi's $80 price tag. That is enough to run MobileNet classification at well over 100fps or YOLOv5n at about 30fps on 720p.
The Pi ecosystem matters here. Thousands of developers already know it, there are libraries and tutorials for everything, and the community support is unmatched. For prototyping edge AI applications, it is hard to beat.
Google Coral
Google's Coral platform uses their Edge TPU -- a purpose-built ASIC for neural network inference. The USB Accelerator ($60) plugs into any Linux board and delivers 4 TOPS. The Dev Board ($130) is a standalone unit. The real appeal is the compiler toolchain: you compile your TensorFlow Lite model for the Edge TPU and get extremely consistent, low-latency inference.
Limitation: it only supports operations that the Edge TPU compiler can map. If your model has layers the compiler does not recognize, those fall back to CPU, and performance drops off a cliff. Stick to standard architectures and you are fine.
Qualcomm AI Engine (In Your Pocket)
Here is the thing people overlook -- the most widely deployed edge AI hardware is already in everyone's pocket. The Snapdragon 8 Elite chip in flagship Android phones packs 75 TOPS across its Hexagon NPU, Adreno GPU, and Kryo CPU. Apple's A18 Pro has a 16-core Neural Engine doing 35 TOPS.
These are not theoretical numbers. When your phone blurs the background in a portrait photo, recognizes text in the camera viewfinder, or transcribes a voice memo locally -- that is the NPU doing real-time inference. And app developers can access this hardware directly through frameworks like TensorFlow Lite, Core ML, and ONNX Runtime.
The Software Stack
Hardware is only half the story. The software that compiles, optimizes, and runs models on edge devices has matured enormously.
TensorFlow Lite / LiteRT
Google's framework for on-device ML. You train a model in standard TensorFlow or Keras, convert it using the TFLite converter (which handles quantization automatically), and deploy the resulting .tflite file to your device. The runtime is about 1 MB. It supports Android, iOS, Linux, and bare-metal microcontrollers (via TFLite Micro).
Quantization matters a lot here. A float32 model converted to INT8 is roughly 4x smaller and 2-4x faster on hardware with integer acceleration -- which is most edge hardware. The accuracy loss for well-designed models is typically under 1%.
# Converting a Keras model to TFLite with INT8 quantization
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model('my_model/')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_model = converter.convert()
# Result: ~4x smaller, runs on Edge TPU, Hexagon DSP, etc.
with open('model_int8.tflite', 'wb') as f:
f.write(tflite_model)ONNX Runtime
Microsoft's cross-platform inference engine. The appeal of ONNX is portability: train in PyTorch, export to ONNX format, run anywhere with ONNX Runtime. It has execution providers for CUDA, TensorRT, DirectML, CoreML, NNAPI (Android), and more. One model format, many deployment targets.
For edge deployments specifically, ONNX Runtime Mobile strips the runtime down to about 1-2 MB and supports dynamic quantization. We have seen benchmarks where ONNX Runtime on ARM CPUs outperforms TFLite for certain transformer-based models.
Edge Impulse
This is the one that surprised us. Edge Impulse is a development platform specifically built for creating edge ML models. You upload sensor data (accelerometer readings, audio clips, images), design a signal processing and ML pipeline in their web UI, and it spits out optimized firmware for your target device -- Arduino, STM32, Raspberry Pi, whatever.
The reason it matters is accessibility. You do not need to be an ML engineer. If you have domain expertise (you know what "good" vibration data looks like versus "bearing about to fail" vibration data), you can build a working model in an afternoon. We recommended it to the IoT client we worked with, and their in-house maintenance team -- mechanical engineers, not data scientists -- built a vibration anomaly detector in three days.
Apple Core ML and MediaPipe
Core ML is Apple's on-device framework, and it is deeply integrated with the Neural Engine on every Apple chip since A11. Models compiled with Core ML Tools run on the Neural Engine automatically, with fallback to GPU or CPU. For iOS and macOS apps, it is the path of least resistance.
Google's MediaPipe provides pre-built solutions for common tasks -- face detection, hand tracking, pose estimation, object detection, text classification. These are battle-tested models optimized for on-device performance. If your use case overlaps with something MediaPipe already does, you can go from zero to working prototype in an hour.
Cloud vs Edge vs Hybrid -- The Honest Comparison
This is the table we wish someone had shown us earlier. Not marketing claims -- actual numbers from deployments and benchmarks we have looked at.
| Factor | Cloud AI | Edge AI | Hybrid |
|---|---|---|---|
| Inference Latency | 100-500ms (network dependent) | 1-50ms | 10-100ms |
| Works Offline | No | Yes | Partially |
| Bandwidth Cost | High (raw data upload) | Near zero | Low (summaries only) |
| Compute Cost | Per-request pricing | Hardware cost (one-time) | Mixed |
| Model Size Limit | None (datacenter GPUs) | Constrained (MB to low GB) | Both available |
| Model Updates | Instant (server-side) | Requires OTA push | Server model updates easily, edge model harder |
| Data Privacy | Data leaves device | Data stays on device | Sensitive data stays, metadata goes |
| Power Consumption | Low on device (offloaded) | Higher on device | Moderate |
| Best For | Large models, complex reasoning | Real-time, privacy-sensitive, offline | Most production systems |
The honest answer for most real-world systems is hybrid. You run a small, fast model on the edge for time-critical decisions and send filtered, anonymized, or aggregated data to the cloud for heavier analysis, model retraining, and dashboarding.
Which is exactly what our IoT client was doing. The edge devices ran anomaly detection locally. When an anomaly was flagged, they sent a 200-byte summary (timestamp, sensor ID, anomaly type, confidence score) to the cloud instead of the raw 50KB/second vibration data stream. That is a 99.6% reduction in bandwidth. Our dashboard consumed the summaries. The cloud ran trend analysis across all sensors. It was elegant, and neither side could have done the other's job efficiently.
Real Use Cases That Are Actually Deployed
We are past the "imagine a world where..." phase. These are running in production right now.
Smart Cameras and Visual Inspection
Factories use edge AI cameras for quality control on production lines moving at speed. A camera with an NVIDIA Jetson module runs a custom-trained YOLOv8 model that identifies scratches, misalignments, or missing components on products flying by at one unit per second. The decision has to happen in under 100ms or the product is past the reject mechanism. Cloud round-trip would be 200-400ms. Not an option.
Predictive Maintenance
Vibration sensors with tiny ML models (we are talking models under 50KB running on ARM Cortex-M4 microcontrollers) that detect bearing wear patterns. These sensors run on coin cell batteries for months because the model only activates when vibration exceeds a threshold. When they detect an anomaly pattern matching early-stage bearing failure, they send a BLE alert. No WiFi needed. No cloud needed.
On-Device Voice Processing
"Hey Siri" and "OK Google" wake words are detected entirely on-device. Apple's voice processing since iOS 15 keeps dictation on the iPhone -- your speech never leaves the phone unless you are using a feature that explicitly requires it. Amazon's newer Echo devices run a local NLU model that handles common commands (lights, timers, music) without sending audio to AWS. Latency dropped from 1-3 seconds to under 500ms for those commands.
Health Monitoring
The Apple Watch's fall detection, irregular heart rhythm notification, and blood oxygen monitoring all run ML models on the S9 chip's Neural Engine. These need to work when your phone is not nearby and when you have no internet. A cardiac event does not wait for a good WiFi connection.
What We Learned Building IoT Dashboards
The client had roughly 200 industrial sensors across three locations monitoring equipment -- motors, compressors, conveyor systems. When they first described the project, we assumed we would be receiving raw sensor data in our API and doing all the processing server-side.
Wrong.
The sensors (running on ESP32-S3 microcontrollers with small TFLite Micro models) were doing edge preprocessing. Each sensor ran a simple anomaly detection model that classified vibration patterns into "normal," "watch," and "alert" states. Only state changes and periodic health pings actually made it to our cloud endpoint.
This changed our dashboard architecture completely. Instead of designing for high-throughput raw data ingestion (which would have needed time-series databases, downsampling pipelines, and significantly beefier infrastructure), we designed for event-driven updates. The dashboard showed real-time device status, historical anomaly trends, and alert management. The backend was a straightforward Supabase setup with a few API endpoints. Way simpler than what raw data ingestion would have required.
But it also taught us something: the edge model's quality directly impacts everything downstream. When the on-device model had a false positive rate of about 3%, the dashboard was noisy and the maintenance team started ignoring alerts. The client's ML engineer retrained the model, dropped false positives to under 0.5%, and suddenly the system was trusted. We did not change a single line of dashboard code. The fix was entirely on the edge.
The Hard Parts Nobody Talks About
Edge AI is not all upside. There are real challenges.
Model Updates Are Painful
Updating a model in the cloud is a deployment. Updating a model on 10,000 edge devices is a logistics operation. You need OTA update infrastructure, version management, rollback capability, and a way to validate that the new model works correctly on devices with varying hardware revisions and firmware versions. Some devices might be offline for weeks.
Debugging Is Hard
When a cloud model gives a wrong prediction, you can log the input, replay it, inspect intermediate activations, and iterate. When an edge model gives a wrong prediction on a sensor mounted 30 feet up on a factory ceiling, you have... less visibility. Remote logging helps, but bandwidth constraints mean you cannot log everything.
Hardware Fragmentation
If you are deploying to phones, you have to deal with the reality that a Snapdragon 8 Elite phone and a budget MediaTek phone have wildly different NPU capabilities. Your model might run at 60fps on one and 4fps on the other. You either target the lowest common denominator or maintain multiple model variants. This is eerily similar to the web development challenge of supporting both modern browsers and old ones. We know that pain.
Where This Is Heading
Two trends are worth watching.
On-Device LLMs
This sounded absurd two years ago, but small language models are now running locally on phones and laptops. Microsoft's Phi-3 Mini (3.8B parameters) runs on phones. Google's Gemma 2B is designed for on-device use. Apple Intelligence runs a ~3B parameter model on-device for text summarization, rewriting, and Smart Reply on iPhone 15 Pro and later.
These are not GPT-4 class models. But for summarization, classification, entity extraction, and simple Q&A against local documents, they are genuinely useful -- and they work without an internet connection. Qualcomm's Snapdragon 8 Elite can run a 7B parameter model at about 15 tokens per second. That is fast enough for real-time text generation.
Always-On AI in Wearables and Glasses
Meta's Ray-Ban smart glasses already run on-device models for scene understanding. The next generation of AR glasses will need constant, low-latency AI processing -- identifying objects, translating text, navigating spaces -- and there is no way to do that through the cloud at the responsiveness users expect. This has to be edge AI.
What This Means If You Build Web Applications
You might be reading this thinking "interesting, but I build web apps, not firmware." Fair. But the line is blurring.
More and more of the data that feeds into web dashboards, admin panels, and analytics platforms is being preprocessed at the edge. Understanding what that preprocessing does -- and what it can miss -- makes you a better architect. When a client says "we want real-time monitoring of our equipment," knowing the difference between raw data streaming and edge-processed events lets you design a system that actually works at scale.
And browser-based inference is a thing now. TensorFlow.js runs models directly in the browser using WebGL and WebGPU. ONNX Runtime has a WebAssembly build. You can do image classification, object detection, and text embedding entirely client-side. We have used this for a client's product image search -- the embedding model runs in the browser, similarity search happens against a pre-computed index, and the whole thing feels instant because it is.
Edge AI is not a separate world from web development. It is the same internet, the same data, the same users. The compute just moved to a different place. And for a lot of use cases, it is a better place.
Comments