Emotion recognition on an ESP32 device using Scailable AI deployment & sending the results to the cloud via LoRa using the KPNThings network (image by author).

Emotion recognition on the Edge

How AI on the Edge (provided by Scailable) and LoRa (provided by KPN Things) jointly enable novel (I)IoT applications.

This tutorial describes a demo we recently presented during the 19th KPN Startup Afternoon; we demonstrated how we can use the Scailable Edge AI deployment platform to deploy a fairly complex AI model (a deep neural network recognizing human emotions) to a small ESP32 device. Subsequently, we used an Arduino MKR WAN 1310 and KPN Things to send the resulting emotions to the cloud via LoRa (find a short video here).

We aim to provide sufficient detail to replicate the full demonstration yourself. Although we think the demonstration is interesting in its own right, we have chosen this setup predominantly because we feel it is illustrative for a much larger class of applications that is enabled by the combination of Edge AI and LoRa. LoRa excels at long-range, energy-efficient transmission of data. However, it is not well suited for sending large amounts of data, such as live images. This is where Edge AI comes in. Trained AI models can often be thought of as computational functions that (severely) decrease data dimensionality. For example, a model might map an image (a tensor containing the image color values per pixel) to the location of a potential crack on the surface captured in the image (a set of x,y coordinates). Or, an AI model might map an picture of a human (again a potentially large tensor) to its emotion (a string: "happy"). In each case, the AI distills the sought-after information from the raw sensor data making it suitable for transmission using LoRa. As such, Edge AI and LoRa are a winning team.

In each case the AI model distills the sought after information from the raw sensor data making it suitable for transmission using LoRa. As such, Edge AI and LoRa are a winning team.

To summarize, if we can use trained AI models on the edge to pre-process sensor data and reduce its dimension, LoRa suddenly becomes a highly appealing platform to send the results to the cloud. In this tutorial we explain both steps. Admittedly, the tutorial is somewhat challenging: it combines the hard disciplines of AI / Data Science and Embedded Systems engineering. However, once combined, great things can happen!

We will cover the following steps:

1. Preliminaries: What do you need to get started?
2. The model: We provide some background regarding the AI model, and how we transpiled it to Scailable WebAssembly (which we use for deployment on the ESP32 device).
3. The Scailable runtime: This is the software running on the ESP32 device that can execute the model specified in WebAssembly (i.e., this is the embedded systems side of things).
4. The communication: Here, we specify how we set up the Arduino and connected it to the ESP32 device to send data from the AI model to the cloud using the KNP Things platform.

Let’s take it step by step!

Preliminaries

Before starting this tutorial it is good to have all the building blocks in place. You will need the following hardware:

• 1x Arduino MKR WAN 1310
• 1x LILYGO TTGO T-camera Plus
• Wires. Three of these.

Next, you will need a KPN Things account for sending your data. You can sign up for free here. For this specific AI model (i.e., the emotion recognition model), we have made the binary and required tools available here. However, if you would like to deploy different AI models to the edge please also register for a Scailable platform account.

1. The model: Recognizing human emotions.

So, let’s start with the AI model used to recognize emotions. Surprisingly to some, many well-performing, trained AI models exist and are publicly available. For this demo we used an emotion recognition model that is part of the gender and emotion classification system repository. The actual trained model, in TensorFlow format, can be downloaded here.

Getting the TensorFlow model to an ESP32 device however can be a daunting task. We used the Scailable deployment platform to create the demo. The steps involved are:

1. We downloaded the trained TF model. Next, we exported the TF model to ONNX, a universal format to store data processing models and pipelines, using the tensorflow-onnx tools.
2. We opened the model using the Scailable ONNX tools. This allowed us to check and clean the model, and validate that it can be converted to WebAssembly using the Scailable platform.
3. Finally, we used the Scailable Platform to convert the model to WebAssembly (check out this post for more on WebAssembly and why its great for edge deployment).

Note: The steps above might seem involved, but once you have the tooling setup they take less than ten minutes. For this tutorial however you do not need to carry out the steps yourself: you can simply fetch the resulting WebAssembly here (and, admittedly, that’s not even neccesary as we package the model with the runtime that we made available for this demo as we discuss in the next section).

2. The runtime: Running the model on the ESP32 device.

The ESP32 device used to recognize emotions (image by author).

Now that we have a model, we want to deploy it to the ESP32 device. Although several versions of the Scailalbe runtime are available for model deployment, for this demo we created a simple binary that can be flashed onto the device directly. The steps involved are spelled out in more detail here, but the short version is as follows:

1. Retrieve the ttgocameraplus.bin, bootloader.bin and partition-table.bin binaries here.
2. Use ESP32 tool found here to flash the binaries onto the device.

That’s it, really.

After installation, when you boot the device, the binary will use the WebAssembly emotion detection model that we saw earlier. For this demo, the WebAssembly is packaged within the binary; generally the Scailable runtime would fetch the model from the Scailable cloud thus making it possible to modularly and securely update the AI model running on the device.

Note: If you want to learn more about modular model updates, we suggest you visit the Scailable tutorials and / or play around with the javascript runtime.

When all of this is up and running, the binary ensures that the ESP32 retrieves a frame from the ESP32’s camera every second, runs it through the Scailable emotion recognition model, and sends the prediction over I2C to the Arduino MKR WAN which subsequently sends it to the KPN Things platform.

3. The communication: Sending the emotional state to the cloud using the KPNThings platform.

The connected Arduino used to submit the data using LoRa (image by author).

For this demo we coupled the EPS32 device to an Arduino which is responsible for sending the data, using LoRa, to the cloud. KPNThings provides a pretty awesome tutorial on how to setup the Arduino: you can find it here. A guide on how to download and install the ThingsML library can be found here.

After following the tutorial to setup the Arduino, the next step is to load, compile and upload arduino_mkrwan_slave.ino, which you can find in our accompanying repository (you will need to change the keys to match your KPN account as explained here).

Note: If you are unsure how to load, compile, and upload to the Arduino, do check out these docs.

Once the software is running on the Arduino, the two devices can be connected using the following wiring:

ESP32 TTGO Camera Plus          Arduino MKR WAN 1310

Ground                  <--->   Ground
SDA                     <--->   SDA
SCL                     <--->   SCL

Once all this is setup, the Arduino will start pushing data towards the KPN Things platform. You will have to connect to a REST endpoint from the KPN Things platform to further process the data. A tutorial on setting this up can be found here. How to configure your dataflow is described here. When your dataflow has been set up correctly, you now see your data arriving at its destination:

Configuration of the webhook we used for the demo (image by author).

After all this is set and done, you will be receiving the emotional state of every human in front of the ESP32’s camera. For us, this looked like this:

Data coming in! (image by author).

Yes! The data is coming in.

That’s it for this tutorial; we hope these materials are sufficient to get this demo up and running yourself! Do contact us if you have any questions.

Wrap up

This tutorial should provide all the materials necessary to recreate the demo we presented during the KPN Startup Afternoon. However, this only scratches the surface of the pretty amazing things one can do by combining the Scailable platform for AI deployment on the Edge, and the KPN Things platform for communication. For more uses case please visit our blog.

However, this only scratches the surface of the cool things one can do by combining the Scailable platform for AI deployment on the Edge, and the KPN Things platform for communication.

Disclaimer

It’s good to note my own involvement here: I am a professor of Data Science at the Jheronimus Academy of Data Science and one of the cofounders of Scailable. Thus, no doubt, I have a vested interest in Scailable; I have an interest in making it grow such that we can finally bring AI to production and deliver on its promises. The opinions expressed here are my own.