Introduction: HackerBox 0095: A.I. Camera Lab
Welcome to HackerBox 0095. Prepare to experiment with embedded artificial intelligence and machine learning projects including computer vision, image processing, and human touch input. Assemble and configure the A.I. Camera Lab platform based on the ESP32-S3 Core Board featuring dual processing cores, hardware neural acceleration, Wi-Fi, Bluetooth, a full color TFT display, touchscreen input, and a two megapixel camera module. Deploy example projects to explore the fundamentals of Artificial Neural Network Intelligence and Model Training for Machine Learning.
HackerBox is the original monthly subscription box for electronics, computer technology, and hacker culture. Each HackerBox is a discovery box, which means all members await and enjoy a new surprise each month. Tech, toys, knowledge, and fun. It's like having a hacker convention, your birthday, and the first day of school - every month - right in your mailbox.
There is a wealth of information for current and prospective members in the HackerBoxes FAQ. Almost all of the non-technical support emails that we receive are already answered there, so we'd really appreciate it if you can take a few minutes to read the FAQ.
Supplies
This Instructable contains information for getting started with HackerBox 0095. The full box contents are listed on the product page for HackerBox 0095 where the box is also available for purchase while supplies last. If you would like to automatically receive a HackerBox like this right in your mailbox each month, you can subscribe at HackerBoxes.com and join the party. Subscribers save at least $15 every month and get each new HackerBox shipped immediately off of the production line.
A soldering iron, solder, and basic assembly tools are generally needed to work on the monthly HackerBox. A computer for running software tools is also required. Have a look at the HackerBox Workshops for tools and supplies along with a wide array of introductory activities and experiments.
The most import thing you will need is a sense of adventure, hacker spirit, patience, and curiosity. Building and experimenting with electronics, while very rewarding, can be tricky, challenging, and even frustrating at times. The goal is progress, not perfection. When you persist and enjoy the adventure, a great deal of satisfaction can be derived from this hobby. Take each step slowly, mind the details, and don't be afraid to ask for help.
WEAR SAFETY GLASSES WHEN SOLDERING, WHEN TRIMMING WIRE LEADS, OR WHEN CUTTING, DRILLING, ETC.
Step 1: ESP32-S3 Core Board N16R8
The ESP32-S3 Core Board is a development board featuring the ESP32-S3-WROOM-1 SoC module that integrates complete Wi-Fi and Bluetooth Low Energy functionality. The ESP32-S3 has support for vector instructions in the MCU, which provides acceleration for neural network computing and signal processing workloads. The N16R8 variant of the WROOM-1 module includes 16 MB of Octal SPI Flash and 8 MB of Octal SPI PSRAM.
A schematic diagram for the ESP32-S3 Core Board is attached here.
Fully test programming the Core Board BEFORE soldering the module.
The Core Board can be powered through either USB port, but for now, let's just use the COM USB port.
When the Core Board is first powered, a small red power LED will illuminate. The other two small LEDs (RX-blue and TX-green) will flicker when the serial interface signals are used.
If you do not already have the Arduino IDE set up and ESP32 support configured, now is the time. The Random Nerd Tutorial covers the necessary steps, but stop at the heading Testing the Installation and instead continue here...
Select Settings:
Tools > Board > esp32 > ESP32S3 Dev Module
Tools > CPU Frequency > 240 MHz
Tools > Flash Mode > QIO 120MHz (using octal flash mode requires burning an eFuse)
Tools > PSRAM > OPI
Download the ESP32_serial_out.ino sketch attached here.
Program the sketch to the Core Board and open the Serial Monitor to watch the MCU count over the serial port. The small green LED will also flicker while the MCU is transmitting serial text.
Onboard RGB LED
The onboard RGB LED is a 5x5 mm WS2812B, which is a common NeoPixel variant. Its Data In pin is connected to I/O pin 48 of the ESP32 through an unpopulated resistor pad marked RGB. To use the RGB LED, that resistor pad must be shorted out, however we will later use I/O 48 with the navigation switch, so we don't suggest permanently shorting the resistor pad.
Step 2: A.I. Camera Lab P.C.B.
The A.I. Camera Lab Printed Circuit Board is a custom experimental platform for exploring the A.I. features of the ESP32-S3 SOC. The platforms neatly interconnects the ESP32-S3 Core Board, a touch screen, a camera sensor, and a five-way navigation switch.
Mount the ESP32-S3 Core Board
Using the headers provided with the core board, carefully soldering each side the header.
With only the core board soldered into place, verify that the ESP32_serial_out.ino sketch still functions correctly. If not, search out and fix any soldering issues.
Mount the Display Module
The 3.2 inch TFT display module features a full-color resolution of 320 x 240. The integrated display chip is an ILI9341. The TFT module also features an SD card reader and integrated touch sensor. A stylus pen for use with the touch sensor is included.
Since the metal frame of the SC card reader will rest against the platform PCB, it is wise to insulate it with a piece of a post-it note or some tape prior to installing the display module.
The display module comes prepopulated with its longer header (for display and touch signals). However, the four-pin header for the SD card reader is provided separately. Solder that four-pin header in place on the red TFT module now.
When placing the display module onto the platform PCB, note that the SD card reader prevents the headers from fully seating. This is perfectly correct. Both the longer (prepopulated) header and the four-pin header will "float" 1-3mm above the platform PCB. Solder the headers to the platform PCB with the same 1-3mm spacing on each pin to keep the display module parallel to the platform PCB - without torquing or bending.
Configure the Display Library
We'll be using the TFT_eSPI Arduino Library by Bodmer. This Arduino graphics library includes performance optimizations for RP2040, STM32, ESP8266, and ESP32 microcontroller SOCs. The library supports TFT displays using driver chips (such as the ILI9341) that operate with hardware SPI or 8/16 bit parallel interfaces.
Install the library through the Arduino IDE, by navigating to Tools > Manage Libraries
From there, search for and install TFT_eSPI by Bodmer.
Next, we need to edit a header file associated with the library.
It can be found in the library folder such as Arduino/libraries/TFT_eSPI.
From that folder, open the file User_Setup.h
First, there are some uncommented lines under the section titled:
EDIT THE PIN NUMBERS IN THE LINES FOLLOWING TO SUIT YOUR ESP8266 SETUP
Comment these lines out
Then, scroll down to the section titled:
EDIT THE PIN NUMBERS IN THE LINES FOLLOWING TO SUIT YOUR ESP32 SETUP
Uncomment the lines as shown below and change the I/O pin numbers as indicated
#define TFT_MOSI 11
#define TFT_SCLK 12
#define TFT_MISO 13 // Not wired to the Display Chip (only Touch and SD)
#define TFT_CS 15
#define TFT_DC 2
#define TFT_RST 16
// #define TFT_RST -1
...
#define TOUCH_CS 1
Save the header file.
Be slow and careful with this process. If the pins are not correctly assigned, the library will not work.
The display, touch sensor, and SD card reader all use the same MOSI/CLK/MISO pins for the SPI bus (11,12,13) but each uses a different chip select (CS) pin to keep the signals from conflicting.
Demonstrate the Display Module
Open the example sketch:
File > Examples > TFT_eSPI > 320 x 240 > TFT_Mandlebrot
Program the example sketch onto the Core Board.
Feel free to try out other example sketches from the TFT_eSPI > 320 x 240 collection.
Additional Details on the Display Module
Can be found on the LCD Wiki Page.
Step 3: Touchscreen Input
A resistive touchscreen panel comprises several thin layers, the most important of which are two transparent electrically resistive layers facing each other with a thin gap between. The top layer (that which is touched) has a coating on the underside surface; just beneath it is a similar resistive layer on top of its substrate. One layer has conductive connections along its sides, the other along top and bottom. A voltage is applied to one layer and sensed by the other. When an object, such as a fingertip or stylus tip, presses down onto the outer surface, the two layers touch to become connected at that point. The panel then behaves as a pair of voltage dividers, one axis at a time. By rapidly switching between each layer, the position of pressure on the screen can be detected. (Wikipedia)
The TFT display module features an integrated touch sensor based on the ADS7843 (datasheet).
Open the example sketch:
File > Examples > TFT_eSPI > Generic > On_Off_Button
Program the example sketch onto the Core Board.
The first time the sketch is run, calibration information is collected and stored into non-volatile memory.
Step 4: Reading SD Card Files
Open the example sketch:
File > Examples > TFT_eSPI > Generic > ESP32_SDcard_jpeg
Following the comments in the top of the sketch file...
- Use the IDE Library Manager to search for and install JPEGDecoder by Bodmer
- Copy the example images to an SD Card
Modify the pin numbers in the sketch:
digitalWrite(1, HIGH); // Touch controller chip select
digitalWrite(15, HIGH); // TFT screen chip select
digitalWrite(7, HIGH); // SD card chip select
and change
if (!SD.begin()) {
to
if (!SD.begin(7)) {
A flexile update to this project scans for JPEG files with any name on the SD card. It can be found here. If the compiler trips on min(~), try changing to min<int>(~).
Step 5: Five-Way Navigation Switch
Download the HB0095_ButtonSerial.ino sketch attached here.
Program the sketch to the target and open the Serial Monitor to demonstrate operation of the Five-Way Navigation Switch.
Attachments
Step 6: Camera Module
The Camera Module includes an OV2640 two megapixel image sensor. The sensor is fitted with a wide angle lens and mounted on a flexible printed circuit (FPC) cable. The FPC terminates in a DVP 24 pin "gold finger" connection.
The Carrier PCB for the Camera Module breaks the camera interface out to 20 (2x10) header pins and also provides flash LEDs and support circuitry for power and clock oscillator. The Camera Module mates to the Carrier PCB via an FPC right angle ZIF connector.
On the side of the Camera Carrier PCB with the two white/yellow flash LEDs, locate the FPC connector and gently lift open the black flap of the connector. Place the gold fingers of the FPC cable onto the FPC connector and gently press the flap back into place to grip the end of the FPC cable. Carefully note the repeated use of the word "gently".
Later, once operation of the camera module verifies that the FPC cable and FPC connector are properly mated, the silver backside of the camera sensor can be affixed to the Carrier PCB with a tiny piece of double sided tape, or other adhesive. This can prevent the camera sensor from flopping about.
Forward Facing Camera Position (Standard)
In the forward position, the Camera Carrier PCB is inserted into the 2x10 header at the edge of the A.I. Camera Lab Platform PCB above the TFT display module. The Camera Carrier PCB stands up perpendicular to the A.I. Camera Lab Platform PCB with the camera sensor facing forward away from the TFT display.
Prior to soldering the 2x10 right angle header, carefully verify all details of the following fitment. Place the header onto the opposite side of the Camera Carrier PCB from the camera (this is the side with the crystal and a lot of little components, not the side with the two LEDs). The pins of the right angle header should be extending from the bottom edge of the Camera Carrier PCB. Most importantly, pin 1 of the camera module should mate to pin 1 of the 2x10 header above the TFT display with the camera facing forward away from the TFT display.
Once the proper positioning and pin 1 mating is verified, the header can be soldered to both PCBs.
Selfie Cam Position (Optional)
In selfie mode, the Camera Carrier PCB is inserted into the 2x10 header to the right of the TFT display and just above the five-way navigation switch. The Camera Carrier PCB lies flat (parallel) to the A.I. Camera Lab Platform PCB with the camera sensor facing up.
For selfie mode, the pins of the right angle header can be trimmed near the bend to modify the header from right angle to a straight header. Most importantly, pin 1 of the camera module should mate to pin 1 of the 2x10 header on the main platform PCB.
Once the proper positioning and pin 1 mating is verified, the header can be soldered to both PCBs.
Flexible Camera Position (Optional)
Using any necessary 2x10 male/female/straight/right-angle headers or even a short 2x10 ribbon cable, you can flexibly locate the camera module however you'd like. Just always be sure to match the pin 1 indicators between the Camera Carrier PCB and the A.I. Camera Lab Platform PCB.
Web Cam with Facial Recognition
Open the example sketch:
Files > Examples > ESP32 > Camera > CameraWebServer
In the main sketch tab, comment out the line:
#define CAMERA_MODEL_ESP_EYE
and the uncomment the line:
#define CAMERA_MODEL_ESP32S3_CAM_LCD
Fill in the ssid and password strings to match a 2.4GHz Wi-Fi network in your operating environment.
In the camera_pins.h tab, locate the following section and paste the definitons below over the original values:
#elif defined(CAMERA_MODEL_ESP32S3_CAM_LCD)
#define PWDN_GPIO_NUM 4
#define RESET_GPIO_NUM 5
#define XCLK_GPIO_NUM 40
#define SIOD_GPIO_NUM 17
#define SIOC_GPIO_NUM 18
#define Y9_GPIO_NUM 39
#define Y8_GPIO_NUM 41
#define Y7_GPIO_NUM 42
#define Y6_GPIO_NUM 9
#define Y5_GPIO_NUM 3
#define Y4_GPIO_NUM 14
#define Y3_GPIO_NUM 47
#define Y2_GPIO_NUM 10
#define VSYNC_GPIO_NUM 21
#define HREF_GPIO_NUM 38
#define PCLK_GPIO_NUM 46
Set Tools > Partition Scheme to HUGE and compile the sketch to the Core Board.
Open the Serial Monitor to see:
WiFi connected
Camera Ready!
Use 'http://xxx.xxx.xxx.xxx' to connect
Copy the URL into a web browser and hit "Start Stream" and remove the lens cap.
Turn on "Face Detection" to demonstrate use of the Viola-Jones Object Detection Algorithm.
Turn on "Face Recognition" to identify faces as intruders and hit "Enroll Face" to register an intruder with a known ID number. (not hotdog)
Integrate the Camera Module and TFT Display
Install the TFT_eFEX library.
Download the Cam_to_TFT.ino sketch attached here to display images captured from the camera sensor directly onto the TFT display.
Exploring Further
These additional camera-based projects (along with many others that can be found online) haven't been tested with the new hardware platform, but they should work fine with some configuration adjustments, particularly adjusting the I/O pin definitions.
Learn more about Edge Impulse in general and the Edge Impulse Arduino Library.
Object Detection with Edge Impulse
Attachments
Step 7: Understanding Neural Networks
Some important mathematics is covered in this video... might be cool to have that on a keychain.
The attached touchscreen_mnist.ino sketch elegantly attacks the 28x28 pixel numerical digit identification problem presented in the first portion of the video. The sketch leverages a TensorFlow Lite model trained on the MNIST database of handwritten digits. The trained model is used to identify numerical digits handwritten onto the touchscreen of the A.I. Camera Lab. The accuracy is pretty impressive.
After downloading the sketch, grab the mnist_model.h file from this project and put it in the same directory with the sketch.
Use the Arduino IDE Library Manager to search and download the library:
TensorFlowLite_EPS32
Compile and upload the sketch. Be patient, it takes a while to compile.
Learn how Trevor Lee trained the TensorFlow model using Google Colab.
But wait, there's more...
If you enjoyed the video above, continue with this one: Watching Neural Networks Learn
Free generative AI courses from Google.
Attachments
Step 8: Trust Your Technolust
We hope you are enjoying this month's HackerBox adventure into electronics, computer technology, and hacker culture. Reach out and share your success in the comments below or on other social media. Email support@hackerboxes.com anytime with questions or whenever you need some help.
Want more? Surf over to HackerBoxes.com and join us as a monthly HackerBox subscription member. You'll get a cool box of hackable gear delivered right to your mailbox every month and you'll enjoy a generous member discount.
Please consider sharing this free Instructable with others who may be interested in learning about these subjects. We really appreciate your support and "word of mouth advertising" is the greatest compliment that we can receive.