Introduction: Personal Assistant

By
About: I do things with computers and other stuff too.

Update: Since Yahoo is turning off their weather API on or around June 1, 2021, you need to find a replacement to build this project.

All of the RAW photos in this article are converted to JPG using Vertopal Online Converter.

---

In this instructable, I'm going to show you how you can use the power of ESP8266, creativity in software design and programming, to make something cool and educational.

I named it Personal Assistant, cause it's pocket sized, talks to you, and can gives you some useful, and (of course) a few not useful (but still cool) information about weather, time and date, gmail messages, live birth and death rates and etc.

I tried to keep the design simple. The device has two user interfaces. A physical push button, and a web based application, that user can access it using a web browser and change the device settings and configuration.

How it works?

The main components of this project are a Microcontroller and a Music Player module. Our microcontroller (NodeMCU) uses WiFi technology to connect to an access point with internet connection; so it can get its required data, process it, and tells the Music Player (DFPlayer Mini) when, which MP3 file should be played.

For now, that's all you need to know. I will give you more detailed information in the next steps, so don't worry.

Step 1: Parts Needed

Other parts:

  • PCB (you can order a prototype online or visit a local shop)
  • Laser cut acrylic sheet
    • 2 mm thickness clear sheet
    • 2.8 mm thickness two different colors sheets (orange and green, red and green, it's up to you and the colors don't matter)
  • Any 5 volts (at least) 1 ampere micro USB charger (to power up the device)

Step 2: How It Works

Okay, I want to give you more detailed information on how the software works.

The software consists of some Services. Each Service, has its own Modules. (You can consider a service as a class and its modules as its methods). Each module, can be considered as an executable object. So, our software consists of some executable objects.

Here we have some services and sub-services or their modules:

  • Gmail
    • Unread Messages
  • Weather
    • Current Temperature
    • Today Condition
    • Today Low / High
    • Tomorrow Condition
    • Tomorrow Low / High
    • Precipitation Forecast
    • Sunrise / Sunset
  • Time
    • Current Time
    • Current Date
  • Birth & Death
    • World Birth
    • World Death

There is a circular queue that holds modules in it. We call it Operation Queue. I said, each module is an executable object. So, when you press the push button on the device, it looks into the operation queue, and execute the next module (or object).

You can edit the members of operation queue in the web based user interface that I will explain later. For now, I will give you an example. Consider current operation queue as this:

QUEUE (Unread Messages | Precipitation Forecast | Current Time)

You press the push button, Unread Messages should be executed.

QUEUE (Unread Messages | Precipitation Forecast | Current Time)

So, the device will use the data it collected (here, number of your unread messages that is grabbed from google mail API feed) to speak to you. But how? Here, NodeMCU will tell the MP3 Module, when it should play which MP3 piece to make a meaningful sentence. To achieve this, I have designed different queues, timers and algorithms. (If you are a c++ guy and like microcontrollers, you can study the code for yourself.)

So, you will hear, the device starts talking: You have 4 unread messages in your gmail inbox.

You press the push button again, next module will be Precipitation Forecast that should be handled.

QUEUE (Unread Messages | Precipitation Forecast | Current Time)

So, you will hear, something like: Don't forget your umbrella, it's rainy tomorrow. And so on...
One more cool thing: for some modules (like precipitation forecast), you can expect random sentences for the same states. For example, if tomorrow there is precipitation and it's rain, and not snow, you can expect "there is possibility for rain tomorrow", "bring your own sunshine, it's rainy tomorrow", "tut, tut, looks like rain for tomorrow", or ...

How we get different data for each service?

  • Gmail
    • Unread Messages
      Google has a powerful API that you can access its different services including Gmail. But, for security purposes, you need different authentication and authorization methods like OAuth. ESP8266 is not that powerful to run different complicated hash algorithms. So, I used an older and simple sign-in technology to access gmail inbox. It is Google Atom Feed that can also be used by RSS Readers. We send an HTTP request to access gmail feed and its response is in XML format. So, we count number of unread messages and use it in our program.
  • Weather
    We use Yahoo Weather API to get different weather information. Recently, just like Google, Yahoo has changed its Weather API, so you will need to use OAuth standards to access its data. Unfortunately, ESP8266 cannot handle its complexity, so we will use a trick to solve the problem. Instead of accessing Yahoo Weather API directly, we will send our request to a custom file on a server. Our file gets data from Yahoo Weather and simply sends it to us.
    • Tomorrow Condition
      Tomorrow Condition will tells you if tomorrow is warmer or colder than today, or there if there will be no sensible change in temperature. We compare "today low / high" with "tomorrow low / high" to achieve this. You can check how I wrote this algorithm and how it works in the program library file.
    • Precipitation Forecast
      If you check Yahoo Weather documentation, you can see Condition Codes table. As it says, Condition codes are used in the response to describe the current conditions. We will use tomorrow condition codes and their meanings to find out if there will be any precipitation and is it rain or snow.
  • Time
    NTP stands for Network Time Protocol. It's a networking protocol for clock synchronization between computer systems. Since we have Internet access, we will use a NTP Client to get time from a NTP Server, and sync it across ESP8266 internal timer (like the one you use it with millis() if you are an Arduino guy).
  • Birth & Death
    We will calculate number of births and deaths from the beginning of the day (Thanks to NTP Client, it's simple to get number of seconds since the beginning of the day). I used world birth and death rates from ecology.

Step 3: Setup the Software

We will use Arduino IDE to upload our program to NodeMCU. You can download and install the latest Arduino IDE from their official site: https://www.arduino.cc/en/main/software

Before starting, you need to setup Arduino IDE for Nodemcu. I'm not gonna tell you the steps here, because it could be off the topic. But you can follow the steps and explanations of this excellent instructable.

Our program has some library dependencies. What is a software dependency?

Dependency is a broad software engineering term used to refer when a piece of software relies on another one.

Here is a list of Arduino libraries you need to have on your computer to be able to compile the Personal Assistant program:

You can download them one by one from their Github page, then extract the zip files to the Arduino library directory. Its path on your system is:
C:\Users\[your-username]\Documents\Arduino

I wrote a library to keep the code clean and avoid complexity. Download the file PersonalAssistant-Library.zip and extract it to Arduino library directory. Just like what you did for those three libraries before.

YahooWeather.php file

Since ESP8266 is not powerful enough to perform hash algorithms, we cannot use it directly to send HTTP requests to Yahoo Weather API, based on OAuth standards. So, we will use a file between our device and Yahoo Weather API. You can download YahooWeather.zip file, extract it and put YahooWeather.php file in a web server. For example if your domain is example.com and you put the file in api directory, your api endpoint becomes example.com/api/YahooWeather.php
You will send weather data requests to this endpoint.

The Program Sketch and the FFS (Flash File System)

Your NodeMCU board has a 4 MB flash file system for storing data. So, when we have it, why not use it?

Remember when I said our device has two user interfaces? Beside that lonely push button, our second user interface is a simple web based application. With this application, you can manipulate the operation queue by enabling / disabling each module, changing service settings or device configuration, like setting WiFi SSID and password. We will store all these files in NodeMCU Flash File System, and run a light web server to handle user requests from their web browser.

Editing Config File

Download the file PersonalAssistant-Sketch.zip and extract it somewhere in your computer. Open the file config.json which is located:

PersonalAssistant/data/config.json

You can use any text or code editor like notepad, notepad++, Atom, etc. The file is a json data structure, so it's human readable key / value pair and you can easily edit it. You can change these fields:

  • Gmail
    • username: your Gmail username with its @gmail.com
    • password: your Gmail password
  • Weather
    • woeid: the location you want to receive weather information for. the WOEID (Where On Earth IDentifier) is a reference identifier uses by Yahoo for location. You can perform a search on WOEID locations in this link.
    • api: it's the API endpoint. The link to your yahooweather.php file.
    • appId, consumerKey and consumerSecret: to access Yahoo Weather API, you need to create a project in Yahoo developers page. This will give you a Consumer Key and Secret which are needed for using the API. To get started, visit Yahoo Weather Developer page and Create an APP.
  • Timezone
    • timezone: enter timezone based on your location. It can be a positive or negative float number and its unit is hours.
  • WiFi
    • ssid: SSID of your network.
    • password: your network password. NodeMCU will use ssid and password to connect to your wifi network.

Uploading the Program Sketch and FFS data

Connect the NodeMCU to your computer, using a micro-USB to USB cable.

Now open the file PersonalAssistant.ino which is located:

PersonalAssistant/PersonalAssistant.ino

In Arduino IDE, from Tools > Board, select NodeMCU 1.0 (ESP-12E Module).
From Tools > Port, select the correct port. It represents your NodeMCU.

Now, select Tools > ESP8266 Sketch Data Upload, this will upload the contents of the data folder to the ESP8266. Wait a few moments until it's finished. Then, select Sketch> Upload or simply press Ctrl + U buttons on your keyboard to start uploading the program. Wait until you see the message "upload is done".

Attachments

Step 4: Setup the Micro SD Card

We use a micro SD card to store the MP3 file pieces. It's NodeMCU who decides which file should be played at what time and DFPlayer Mini helps him in making a meaningful sentence by decoding the MP3 files.

I used Amazon Polly to generate the voice pieces I needed.

Amazon Polly is a service that turns text into lifelike speech, allowing you to create applications that talk, and build entirely new categories of speech-enabled products.

Don't forget, our device doesn't use Amazon Polly API to speak dynamically. We have some static offline voice pieces, and by putting them together, we make different sentences.

I used this site to generate MP3 files. The voice output I selected was US English / Salli.

The only thing you need to do, is to download microSD.zip file, then extract it into your micro SD card. It contains all the 78 required MP3 files.

Probably your Micro SD Card comes with and adapter. You can insert your Micro SD Card into its adapter and attach it to your laptop. If your computer doesn't support card reading, then you should use an external Card Reader.

Attachments

Step 5: Designing the Schematics and Setting Up the PCB

I've designed the Schematic and Board using the Autodesk EAGLE. I have included both SCH and BRD files in PersonalAssistant-PCB.zip. You can easily edit and / or send it to a local or online PCB manufacturer to order and get your board.

One more thing to mention is, ESP8266 operates in 3.3v while DFPlayer Mini works in 5v. Since these two modules need to talk to each other through serial interface, we cannot directly connect a 5v output to a 3.3v input as it damages your ESP8266. So we will need a level conversion from 5v to 3.3v. We use a signal diode and a 10K resistor to make it happen.

Attachments

Step 6: Solder It Up

Assembling the board is pretty simple as you have a few components. Follow the schematic and board designs in step 5 to easily put each item on its correct place.

I started by soldering the resistors and the diode, as they are small. You can easily cut their unneeded tails using a wire cutter. From top to bottom, you should put 1K, 10K and 10K resistor.

You don't have to solder all the NodeMCU and DFPlayer Mini pins on the PCB. Soldering the pins with a path is enough.

Don't forget, speakers and diodes have polarity. You have one speaker and one diode in your components. For the diode, the side with a black line is its negative side or the cathode.

Step 7: The Enclosure

I decided to design a fancy enclosure in a creative way. I was worried about its weird shape during the design, but at the end, It wasn't that bad. At least it looks like a grand piano and feels great holding in your hand!

Instead of the classic hexahedron cubic shape with 6 faces, I designed a multi-layer enclosure. From bottom to top, each layer lays down on its lower layer. (I named them L0 to L6, from bottom to top)

Colors and Thickness

You can use two complementary colors to make the strongest contrast, like:

  • Red and Green
  • Blue and Orange
  • Yellow and Purple
  • Blue and Yellow

I used clear acrylic for the top layer, so you can see inside the device.

The thickness of the top layer (layer-6) should be 2 mm.
The thickness of the other layers (layer-0 to layer-5) should be 4 mm. If you want to use a 2.8 mm acrylic shield, like I did, there is no problem. But you need to cut two series from layer-1 and layer-3 for the offset.

To assemble the enclosure, start from the bottom layer (L0). Put the board on it, use the shorter bolts, and tighten it using the nuts. You can now, attach the four longer bolts from bottom of the layer-0. Something like a tower. Then you can easily continue to mount other layers on them.

Note: You can use an optional washer between bottom layer and the board.

I have also added info texts for device ports (power and micro SD card). You can use laser engraving on top layer.

I have included both CDR and DXF file formats. You can download them, edit them and use them for laser cut.

Attachments

Step 8: Accessing Web Based User Interface

Power Up the Device

You can power up the device, using any 5v micro USB charger. Attach the micro USB to the device Power port, that is the micro USB input on your NodeMCU.

Access the User Interace

Do you remember we uploaded some files into ESP8266 Flash File System? It's time to use it. All you need, is the IP Address assigned to ESP8266 on the network. There are many different ways to find the IP address. I will list some of them here:

  • In your router configuration page, somewhere in DHCP Lease List, you can see list of devices with their IP Addresses in your network.
  • In Microsoft Windows and macOS you can run commands like arp -a in terminal.
  • In Android and iOS, you can use applications like Fing. (Android / iOS)
  • In Linux, you can use tools like Nmap.

After finding IP Address, open it using your web browser. You can manipulate operation queue by enabling / disabling modules.

Step 9: Final Thoughts

This project was very time and energy consuming. You can add many more options to Personal Assistant. I have left some parts open for future development. Some parts like:

  1. Adding more services and module. For example counting numbers, rolling a dice or flipping a coin.
  2. After connecting to network, device can speak the IP Address. You can add this option to simplify the finding IP Address process.
  3. Adding the ability to change WiFi settings in web based control panel.
  4. Adding the ability to change service settings in web based control panel. (Their html form are ready. you need to handle requests)
  5. Adding more voice responses in different states of the device.
  6. Adding a login page for web based control panel. You can do this by adding / comparing Cookies in HTTP header lines.

And, I would love to know your ideas about this instructable. :)