1. Introduction
What you'll build
In this codelab, we'll learn to use TensorFlow Lite For Microcontrollers to run a deep learning model on the SparkFun Edge Development Board. We'll be working with the board's built-in speech detection model, which uses a convolutional neural network to detect the words "yes" and "no" being spoken via the board's two microphones.
Machine Learning on Microcontrollers
Machine learning can be used to create intelligent tools that make users' lives easier, like Google Assistant. But often, these experiences require a lot of computation or resources that can include a powerful cloud server or a desktop. However, it's now possible to run machine learning inference on tiny, low-powered hardware, like microcontrollers.
Microcontrollers are extremely common, cheap, require very little energy, and are very reliable. They are part of all sorts of household devices: think appliances, cars, and toys. In fact, there are around 30 billion microcontroller-powered devices produced each year.
By bringing machine learning to tiny microcontrollers, we can boost the intelligence of billions of devices that we use in our lives, without relying on expensive hardware or reliable internet connections. Imagine smart appliances that can adapt to your daily routine, intelligent industrial sensors that understand the difference between problems and normal operation, and magical toys that can help kids learn in fun and delightful ways.
TensorFlow Lite For Microcontrollers (Software)
TensorFlow is Google's open source machine learning framework for training and running models. TensorFlow Lite is a software framework, an optimized version of TensorFlow, targeted to run tensorflow models on small, relatively low-powered devices such as mobile phones.
TensorFlow Lite For Microcontrollers is a software framework, an optimized version of TensorFlow, targeted to run tensorflow models on tiny, low-powered hardware such as microcontrollers. It adheres to constraints required in these embedded environments, i.e, it has a small binary size, it doesn't require operating system support, any standard C or C++ libraries, or dynamic memory allocation, etc.
SparkFun Edge (Hardware)
The SparkFun Edge is a microcontroller-based platform: a tiny computer on a single circuit board. It has a processor, memory, and I/O hardware that allows it to send and receive digital signals to other devices. It has four software-controllable LEDs, in your favorite Google colors.
Unlike a computer, a microcontroller doesn't run an operating system. Instead, the programs you write run directly on the hardware. You write your code on a computer and download it to the microcontroller via a device called a programmer.
Microcontrollers are not powerful computers. They have small processors, and not much memory. But because they are designed to be as simple as possible, a microcontroller can use very little energy. Depending on what your program does, the SparkFun Edge can run for weeks on a single coin cell battery!
What you'll learn
- Compile the sample program for the SparkFun Edge on your computer
- Deploy the program to your device
- Make changes to the program and deploy it again
What you'll need
You will need the following hardware:
- Linux or MacOS computer
- SparkFun Edge board
- SparkFun USB-C Serial Basic programmer
- USB-C to USB-A cable (If you're on a USB-C computer, instead get USB-C to USB-C cable)
- (optional) 3V 20mm coin cell lithium battery (CR2032) to run inference without a programmer and cable
You will need the following software:
- Git (check if it's installed by running
giton the command line) - Python 3 (check if it's installed by running
python3orpython --versionon the command line) - Pip for Python 3 ( helpful StackOverflow answer)
- Make 4.2.1 or higher (check if it's installed by running
make --versionon the command line) - SparkFun Serial Basic drivers
2. Set up your hardware
The SparkFun Edge microcontroller comes with a pre-installed binary that can run the speech model. Before we overwrite this with our own version, let's first run this model.
Power your board by:
- Inserting a coin cell battery into the battery connector on the back of the board (with the "+" side of the battery facing up. If your board came with a battery already inserted, pull out the plastic tab, and push the battery to ensure it's fully inserted)
- If you don't have a coin battery, you can use the SparkFun USB-C Serial Basic programmer device to power the board. To attach this device to your board, perform the following steps:
- Locate the six pin header on the side of the SparkFun Edge.
- Plug the SparkFun USB-C Serial Basic into these pins, ensuring the pins labelled "BLK" and "GRN" on each device are lined up correctly.
- Connect a USB-C cable between the SparkFun USB-C Serial Basic and your computer.
Once you've powered your board by inserting the battery or connecting the USB programmer, the board will wake up and begin listening with its microphones. The blue light should begin to flash.
The machine learning model on the board is trained to recognize the words "yes" and "no", and to detect the presence and absence of speech. It communicates its results by lighting colored LEDs. The following table shows the meaning of each LED color:
Detection result | LED color |
"Yes" | Yellow |
"No" | Red |
Unknown speech | Green |
No speech detected | No LEDs lit |
Give it a try
Hold the board up to your mouth and say "yes" a few times. You'll see the yellow LED flash. If nothing happens when you say "yes", here are some things to try:
- Hold the board around 10" from your mouth
- Avoid excessive background noise
- Repeat "yes" several times in quick succession (try saying "yes yes yes")
3. Set up your software
We're now going to download, install and run the speech model on the microcontroller ourselves. For this, we first download the source code for this program and the dependencies we need to build it. The program is written in C++, which must be compiled into a binary before being downloaded onto the board. A binary is a file that contains the program in a form that can be run directly by the SparkFun Edge hardware.
The following instructions are written for Linux or MacOS.
Download the TensorFlow repo
The code is available in the TensorFlow repository on GitHub, in the following location:
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/micro
Open a terminal on your computer, change to a directory where you usually store coding projects, download the TensorFlow repository and enter the directory created, as shown below:
cd ~ # change into your home (or any other) directory git clone --depth 1 https://github.com/tensorflow/tensorflow.git cd tensorflow
Download Python dependencies
We'll be using Python 3 to prepare our binary and flash it to the device. The Python scripts depend on certain libraries being available. Run the following command to install these dependencies:
pip3 install pycrypto pyserial --user
4. Build and prepare the binary
We're going to build the binary and run commands that prepare it for downloading to the device.
Build the binary
To download all required dependencies and create the binary, run the following command:
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=sparkfun_edge micro_speech_bin
If the build works successfully, the final line of the output should appear as follows:
arm-none-eabi-objcopy tensorflow/lite/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech tensorflow/lite/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin -O binary
To confirm that the binary was successfully created, run the following command:
test -f \ tensorflow/lite/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin && \ echo "Binary was successfully created" || echo "Binary is missing"
You should see Binary was successfully created printed to the console! If you see Binary is missing, there was a problem with the build process that will require debugging.
Prepare the binary
The binary must be signed with cryptographic keys to be deployed to the device. We'll now run some commands that will sign our binary so it can be downloaded to the SparkFun Edge.
Enter the following command to set up some dummy cryptographic keys we can use for development:
cp tensorflow/lite/micro/tools/make/downloads/AmbiqSuite-Rel2.2.0/tools/apollo3_scripts/keys_info0.py tensorflow/lite/micro/tools/make/downloads/AmbiqSuite-Rel2.2.0/tools/apollo3_scripts/keys_info.py
Now, run the following command to create a signed binary:
python3 tensorflow/lite/micro/tools/make/downloads/AmbiqSuite-Rel2.2.0/tools/apollo3_scripts/create_cust_image_blob.py \ --bin tensorflow/lite/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin \ --load-address 0xC000 \ --magic-num 0xCB \ -o main_nonsecure_ota \ --version 0x0
This will create the file main_nonsecure_ota.bin. We'll now run another command to create a final version of the file that can be used to flash our device with the bootloader script we will use in the next step:
python3 tensorflow/lite/micro/tools/make/downloads/AmbiqSuite-Rel2.2.0/tools/apollo3_scripts/create_cust_wireupdate_blob.py \ --load-address 0x20000 \ --bin main_nonsecure_ota.bin \ -i 6 \ -o main_nonsecure_wire \ --options 0x1
You should now have a file called main_nonsecure_wire.bin in the directory where you ran the commands. This is the file we'll be flashing to the device.
5. Get ready to flash the binary
What is flashing?
The SparkFun Edge stores the program it is currently running in its 512 kilobytes of flash memory. If we want the board to run a new program, we have to send it to the board, which will store it in flash memory, overwriting any program that was previously saved.
This process is called "flashing", and we'll use it to send our program to the board.
Attach the programmer to the board
To download new programs to the board, we'll be using the SparkFun USB-C Serial Basic serial programmer. This device allows your computer to communicate with the microcontroller via USB.
To attach this device to your board, perform the following steps:
- Locate the six pin header on the side of the SparkFun Edge.
- Plug the SparkFun USB-C Serial Basic into these pins, ensuring the pins labelled "BLK" and "GRN" on each device are lined up correctly.
Attach the programmer to your computer
We'll be connecting the board to your computer via USB. To program the board, we'll need to know the name that your computer gives the device. The best way of doing this is to list all the computer's devices before and after attaching it, and look to see which device is new.
Before attaching the device via USB, run the following command:
If you are using Linux: ls /dev/tty* If you are using MacOS: ls /dev/cu*
This should output a list of attached devices that looks something like the following:
/dev/cu.Bluetooth-Incoming-Port /dev/cu.MALS /dev/cu.SOC
Now, connect the programmer to your computer's USB port. Enter the following command again:
If you are using Linux: ls /dev/tty* If you are using MacOS: ls /dev/cu*
You should see an extra item in the output, as in the example below. Your new item may have a different name. This new item is the name of the device.
/dev/cu.Bluetooth-Incoming-Port /dev/cu.MALS /dev/cu.SOC /dev/cu.wchusbserial-1450
First, we'll create an environment variable to identified the device name:
export DEVICENAME=put your device name here
Next, we'll create an environment variable to specify the baud rate, which is the speed at which data will be sent to the device:
export BAUD_RATE=921600