AI at the edge: running models on local devices
In short: Edge AI means running trained models on devices or local servers close to where data is created. It gives fast answers, works when the link is down and keeps raw data on site. The price is limited compute, many devices to manage and models that can be stolen or tampered with if not protected.
What edge AI means
Most machine learning projects have two phases. Training builds the model from large datasets and needs a lot of compute, so it normally happens in a data centre or cloud. Inference uses the trained model to make a decision on new data, such as "is there a person in this frame?". Edge AI moves inference, and sometimes light retraining, onto devices close to the data: smart cameras, industrial gateways, phones, vehicles or small servers in a branch.
A warehouse camera that counts pallets is a simple example. Sending every video frame to the cloud uses a lot of bandwidth and adds delay. Running a small object-detection model on the camera means only the count and a few flagged images travel over the network.
Making models fit small hardware
Edge devices have limited memory, power and cooling. Several techniques help a model fit:
- Quantisation stores weights with fewer bits, for example 8-bit integers instead of 32-bit floating point. Models get smaller and faster with a small loss of accuracy.
- Pruning removes connections that contribute little to the result.
- Distillation trains a small "student" model to copy a larger "teacher" model.
- Accelerators such as neural processing units (NPUs), graphics processing units (GPUs) and dedicated vision chips run inference efficiently. Many recent laptops and phones include an NPU.
At the very small end, TinyML runs simple models on microcontrollers using milliwatts of power, for jobs such as keyword spotting or vibration analysis on a motor.
Federated learning and privacy
Federated learning trains a shared model without collecting the raw data centrally. Each device improves the model on its own data and sends only the update. A server combines the updates into a new global model. This helps privacy, for example with keyboard prediction or hospital data, but it is not a complete guarantee. Updates can still leak information, and a malicious participant can send poisoned updates, so techniques such as secure aggregation and update validation are used alongside it.
Securing edge AI
| Risk | Example | Control |
|---|---|---|
| Model theft | Someone copies the model file from a stolen device | Encrypted storage, secure enclaves where available, licence checks |
| Model tampering | A swapped model ignores certain objects | Signed models, verified at load time with secure boot |
| Adversarial input | A printed pattern confuses a camera model | Robust testing, multiple sensors, human review of key decisions |
| Data exposure | Stored frames contain faces | Process in memory, minimise retention, follow privacy law |
| Stale models | Accuracy drops as conditions change | Monitoring, staged updates and rollback |
Treat a model as software: version it, sign it, test it and patch it through the same pipeline as firmware. See adversarial machine learning for more on attacks against models.
Tip: a common mistake is assuming "on-device" means "private". Check what the device logs, what it uploads for retraining and who can access it remotely.
Practice
What is the difference between training and inference?
Name two ways to make a model small enough for an edge device.
Why does federated learning not fully guarantee privacy?
Key takeaways
- Edge AI runs inference close to the data for speed, offline operation and privacy.
- Quantisation, pruning and distillation shrink models for small hardware.
- Federated learning shares model updates instead of raw data, but still needs safeguards.
- Sign, version and monitor edge models like any other firmware.
- Protect stored data and models on devices that may be stolen or tampered with.
Original content written by Mohamed Fayas for FM TecHub. © 2026 FM TecHub. You're welcome to share a link to this page; please don't copy the text elsewhere without permission.