Introduction: Use RPi, Azure, and Cortana to Automate Your Home!

By

Introduction

There is new technology all around us and only more coming every day. Our micro controllers are getting faster, are phones getting smarter, and the cloud is becoming stronger. With all this new technology everyone is asking themselves: How can I utilize it all?

Well, today we will do just that. This guide will walk you through using a Raspberry Pi, Microsoft Azure, and Cortana on Windows Phone to automate your house hold. Did you forget to turn the lights off when you left for work? No problem. Want to turn on the Slow-Cooker a few hours before you get home? Consider it done. We can leverage the power of Internet of Things and Cortana to control our home in an entire new way!

Goals

For this guide, our primary goal is to show off a proof-of-concept device cycle that is functional and really lets you see the power of combing all of these technological resources. We won't be making production ready code, and won't be utilizing every device/technology to its fullest ability. This guide is already fairly long given all the ground we have to cover, and doing a full dive into each would give us quite the novel. Instead, we're focusing on something that most "tinkers" could make as long as they have the hardware and a limited knowledge of programming. We want to give you a working project and inspire you to push it even further. This project is your 'getting started' guide and the real fun begins when you pick up after that!

Step 1: Requirements

Knowledge Requirements

We have tried to make this guide as 'friendly' as possible to the general audience, but in order for us to move quickly and condense the guide down from a three hour epic to a 1 hour project, we have to assume you know a little something before hand.

*Don't know something? Don't worry! Throughout the guide we include lots of links to other sources where you can study up beforehand and catch up quickly!

1. Limited Python and C# Knowledge

Basic programming knowledge will be needed. We won't be diving into anything too complicated, but it's good to have a basic understanding of these two languages

2. Raspberry Pi Basics

Because it's a bit more popular compared to Windows Phone and Azure, I won't be going too in depth on setting up your Raspberry Pi and wiring it all together. There are also a lot of great tutorials out there already to help you get started. Make sure you can ssh into one, know your basic Linux commands, and feel comfortable putting a circuit together (a very basic one).

3. General App Development Knowledge (Windows Phone a Plus!)

Obviously since we are going to use Cortana we will be building a Windows Phone app. While setting up the project isn't too difficult, and we really won't be doing anything too complicated, it's a good idea to have a basic understanding of the application files and what they do. The code logic versus the design logic, the application manifest, etc. If all these buzzwords are second nature to you, then don't worry. Otherwise, make sure to read up on the resources provided in that section of the guide.

Hardware Requirements

What would a project be without some sweet hardware behind it? This project will actually be pretty bare in the hardware department and instead focus a lot on the software. We'll walk through how to install the specific software packages right before they are needed in an effort to make that process easier.

  • A Windows 8 Computer
  • Windows Phone with 8.1 software update (Must have Cortana!)
  • Raspberry Pi
  • SD Card with Raspbian for Rasberrry Pi (NOOBS works just fine)
  • 3 Wires for your simple circuit
  • 1 LED
  • 1 270 ohm resistor
  • Breadboard for your LED circuit

Step 2: Setting Up Azure 1: a Crash Course in Azure

It's very likely that you have never even used a service like Microsoft Azure so we will start off with a bit of a crash course. Obviously we won't be covering everything (that's what the hours and hours of online material is for) but instead focusing on what exactly we are doing. This way you're not simply building a black box and have no idea that magic happening behind the scenes, but rather a high-level conceptual understanding. This will allow you to expand on it in the future, and also understand the purpose behind why we are using Azure versus another service.

Why Azure?

Why not Node.js? That's probably a question a lot of you will be asking, and it's a very legitimate question. Why not just create a basic REST server that we can hit to command our devices? Well the issue is that we live in the future, and the future is all about the Internet of Things (IoT). IoT deals with tens if not hundreds of little devices all around your home, all connected giving you unparalleled control. Will a simple Node server running on a Pi be able to handle all of that? Isn't dealing with all of that funky server code another guide (if not an entire book) in it's own right? Yes, yes it is, and that's why Azure is here to the rescue. While we really won't see the benefits of Azure in this initial guide because we are only hooking up one device, once we begin to hook up more and more devices we will be able to see the true benefits.

What Will We Be Making?

For our project, we're going to make a Service Bus that will process Topics and Subscriptions. Don't worry, I know we're throwing a lot of fancy words around early, but I assure you that it doesn't take long to get a basic understanding. A Service Bus, in a nutshell, provides a highly robust messaging framework that serves as a relay between two (or more) endpoints. It is essentially the magical 'cloud' that we hear so much about. Something sends it a message, it decides where that message should go, sends it, and another device gets that message. The service bus is our mail sorting facility, make sense?

So what about these Topics and Subscriptions? Why can't we just call them messages? Well, because it's not quite that simple. A Topic contains a message, but you can't say that a Topic is a message. It's just incorrect. So what is a Topic then? A Topic forms a relationship (both logical and physical) between publishers and subscribers so that a publisher (Cortana) can publish messages to multiple subscribers (all of our IoT devices). Think of it this way: Say we had 10 different IoT devices all around our home, all hooked up to different light switches. When we give the command "Turn the Lights Off" we want to send a message to each and every IoT device telling it to turn off, but we don't want to send 10 different messages. Furthermore, how much of a pain is it that every time we add an IoT device we need to re-code our entire Cortana logic? Instead, we publish a message to the Topic "LightControls" and that Topic now publishes to all subscribers (which would be every IoT device that controls a lightswitch) to go to the "OFF" position.

Still Confused? Don't worry, this isn't something that's easy to pick up (let alone explain) in a paragraph or two. If you still want to learn more, here are some great resources:

Introducing Queues and Topics in Azure Service Bus - Code Magazine

How to User Service Bus Topics/Subscriptions - Microsoft

Windows Azure Service Bus Topics and Subscriptions - Neudesic

In a Nut Shell...

Cortana is going to send a message to a Topic on the Service Bus (the cloud). Our Cloud will then send that message to every device that has "subscribed" to that topic. So when we send "DeskLightsOff" to the "LightControl" Topic, our DeskLights will have Subscribed to it, will receive it, and then will process that command.

Step 3: Setting Up Azure 2: Creating Your Azure Service Bus

Time to Code! Well...not yet. First, we have to set up our Azure Service Bus. Microsoft has provided a pretty slick online interface that actually lets us create the whole thing without typing a single line of code. Pretty Cool. Let's get started!

1. Create Your Azure Account

Chances are you don't have an Azure Account, so you will need to sign up for their free trial. It's fairly straight forward, although you will need to enter a credit card

Worried about paying monthly for this? Don't. The service bus we will set up will likely receive less than 1 million calls a month. It currently costs about $1.00 for 30 million. You'll be paying a few dimes at most. If you want to continue past the free trial, however, be prepared to shell out about $11 a month.

2. Create A Service Bus

  • Log into the Azure Management Portal after you have created your account.
  • Figure 1: Click on 'Service Bus' to go to the 'Service Bus' Dashboard.
  • Figure 2: Click on 'Create' In the lower pane.
  • Figure 3: Enter in a name for your namespace (Must be Unique!). Note that my Region is Central US because that's where I am. Azure will likely put in whatever region it thinks is best, and you should just leave it. Click on the Checkmark in the lower right corner to continue.
  • Your Namespace will be "activating" for a short bit, and then Boom! Your Service Bus has been Created!
  • ***NOTE***For this guide, I have named my Namespace "CustomNamespace". Anywhere you see that String, you should replace it with your own Namespace name.

Recap

So basically all we did here was get started with Azure. We created an account and then a Service Bus. Remember that a Service Bus provides us with a Cloud "Mail Sorting Facility", but right now that Facility doesn't really have any direction. In our next step we will work on adding actual logic with Topics and Subscriptions.

Step 4: Setting Up Azure 3: Creating Topics and Subscriptions

Now that we've created the Service Bus, it's time to add our Topics and Subscriptions. Remember from our previous write-up that we send messages to a Topic, which is then relayed to one or more subscriptions.

Conceptual Setup

Before we just start hacking away, let's take a step back and remember what exactly we are doing. Look at Figure 1 and make sure you have an idea of what's going on. We won't be creating that exact model today, but it is definitely something you could do further down the road. We have a LightTopic which is going to be where we send commands dealing with turning off the lights. We then have our LightSubscription which we will label DeskLightSubscription because it's what we want the Desk lights (the led we have hooked up to the Pi) to listen to.

1. Create the 'LightTopic' Topic

  • Figure 2. Click on your Service Bus to go to the Service Bus Dashboard
  • Figure 3. Click on 'TOPICS' on the upper panel to go to the Topics Page
  • Figure 4. Click on 'CREATE A NEW TOPIC'
  • Figure 5. Type in 'LightTopic' (Or whatever you want to call it) Then Click 'CREATE A NEW TOPIC'

Great! We've created our first Topic. This is where we will send all of our commands from Cortana.

2. Create the 'LightSubscription' Subscription

  • Figure 6. Click on 'New Subscription' At the bottom of your Topics page (Where you should have left off)
  • Figure 7. Enter 'LightSubscription' into the TextBox and click on the Arrow in the Lower Right Corner.
  • Figure 8. You'll be taken to a 'Details' Page. Leaving everything at the Defaults will be fine, so simply click on the arrow again.
  • Figure 9. Once you go back to the Topic Dashboard, you'll notice that there is now 1 Subscription attached to the topic.

Recap

Yup. We're done. It really is that simple. Now, navigating through Azure can be a little daunting at first, especially with all the different terms which we may not all be familiar with. As we saw though, the process is actually pretty simple. I know it doesn't really feel like we've done anything yet because there isn't really anything tangible or code, and unfortunately there won't be for a while. But we have created an essential and important part of our project!

Step 5: Setting Up Your Raspberry Pi 1: Environment Setup

Now that we've got our Azure Service Bus setup, it's time to setup our IoT device, AKA our Raspberry Pi.

What Our Raspberry Pi Will Do:

To Recap, our Raspberry Pi is simply going to be a IoT slave. It will do what it is told to do which is either turn on or off the lights. In the future, this could be expanded to sending messages regarding different variables you want to measure, but for now we're going to keep it simple for the sake time and length of this guide. The Pi will be attached to our 'LightSubscription' and listen for anytime it gets a message from that subscription, which is of course triggered by our topic.

Hardware Setup

Setting up the circuit, dealing with GPIO, and general Raspberry Pi shenanigans is a bit beyond the scope of this guide, simply because it's quite literally an instructable in itself. Basically to get a general setup you will need to create a very basic "Blinky" circuit with your Raspberry Pi. If you need help with that, I would highly suggest checking out a few of these guides that cover the subject.

Software Setup

Now that you have a basic led circuit working on your Pi, we will need to install the software packages needed to interact with our Azure Service Bus. Lucky for us, there is a Python SDK for Azure which works quite well. We can simply use git to clone the repo down and install it on our machine:

>>>>git clone https://github.com/Azure/azure-sdk-for-python.git
>>>>cd azure-sdk-for-python 
>>>>python setup.py install

If you're having any trouble, make sure that you have done an 'update' and 'upgrade' recently.

>>>> sudo apt-get update
>>>> sudo apt-get upgrade

To test and make sure that the azure SDK installed correctly, do the following commands and make sure the output is the same:

>>>> python  
Python 2.7.3 
>>>> import azure
>>>> 
>>>> exit()

If it installed correctly, typing the line `import azure'` should result in a blank line following it. Simply type `exit()` to leave the python terminal.

Step 6: Setting Up Your Raspberry Pi 2: Coding the Initial Setup

Alright! Time to actually get down and dirty with some code! The code for this is actually pretty straightforward, and it's only about 50 lines of code. That being said, we'll try and walk through it slowly so you can get a better conceptual understanding of how it works. If you just want to "Grab and Go" so to speak, the entire file is attached for those who wish to do so.

Import Libraries

We have quite a few libraries we need to import for all these moving parts to work. Remember that we are also controlling the pins, we will need to import the GPIO functionality as well. In total, the top of your import statements should look like the following:

import RPi.GPIO as GPIO #For Controlling the Pins<br>import threading #To Run Async
import sys
import select
from azure.servicebus import *
import os

The 3 oddball ones are obviously the GPIO, Threading, and Azure libraries. The middle one, threading, might seem a little strange at first. Essentially we need it to make sure we are actively "listening" to the server subscription. To do this effectively, we run that "listening" on a separate thread.

Create Constant Variables

Yes, this can seem a bit trivial, but it will also help a lot with understanding the connection between Azure and Python. The code is pretty straight forward:

# Make sure you set the following:

AZURE_SERVICEBUS_NAMESPACE='CustomNamespace'
AZURE_SERVICEBUS_SHARED_KEY_NAME='RootManageSharedAccessKey'
AZURE_SERVICEBUS_ACCESS_KEY_VALUE='<INSERT_YOUR_ACCESS_KEY_HERE>'
GPIO_BCM_PIN = 11 #The Pin your LED is controlled by

The 'namespace' and 'GPIO' variables should be pretty obvious, but the middle two could cause some confusion. Essentially, this is your special "login Key' that will grant access to your Azure Service Bus. For now, don't worry about it (We'll find out where to get this Key in the next step!).

Set up Lights on Start-up

Whenever we start our program, we want to give our Lights (or in our case, our little LED) a specific state. In this case, we will set it to 'OFF"

# setup the GPIO for the LED
GPIO.setmode(GPIO.BCM)
GPIO.setup(GPIO_BCM_PIN,GPIO.OUT)
# Initially turn off the LED
GPIO.output(GPIO_BCM_PIN, 0)

Start Incoming Messages Thread

Here is where we start to work the magic. We will create a thread and have it target a new function (which we have not created yet) called process_messages. We will create this function in the next step. For now, let's create and start the thread.

# start a thread listening for incoming messages
t = threading.Thread(target=process_messages) #will create 'process_messages' next step
t.daemon=True;
t.start()

Wait Clean-up

We will then 'wait' for any raw_input from our user. Essentially, don't end this program unless somebody hits a key. Finally, we'll release any GPIO resources to ensure a safe exit from our program.

# wait until the user enters something
char = raw_input("Press enter to exit program")
# release any GPIO resources
GPIO.cleanup()

Recap

So what does this code do? Well, not much. In fact in won't even compile right now (we're missing that process_messages function!). But we've set up the structure for how our IoT device will work. We will set the light to 'OFF', then listen for any command from the Service Bus on a separate thread. The next step will show us how to do that.

Step 7: Setting Up Your Raspberry Pi 3: Coding the Subscription

This step completely revolves around the process_messages function that we talked about in the previous step. We will use the Python Azure SDK to actively listen for messages from the Azure subscription and update our LED accordingly.

Initialize the Service Bus Object

First things first, we will create this 'process_messages()' function and then the service bus object. 

def process_messages():    # Initialize the service bus
    service_bus = ServiceBusService(service_namespace=AZURE_SERVICEBUS_NAMESPACE,			 shared_access_key_name=AZURE_SERVICEBUS_SHARED_KEY_NAME,			
	 shared_access_key_value=AZURE_SERVICEBUS_ACCESS_KEY_VALUE
)

Pretty simple, right? Notice that we used our namespace, Key Name, and Key Value to hook up to the Service Bus. These are the key things you need to actually log in and interact with your service bus. We will find out how to add the key name and value at the end of this step.

Get the Topic and Subscription

Pretty Straightforward. We're going to get our 'LightTopic' topic and 'LightSubscription' Subscription'. This way our Service Bus knows who to interact with.

service_bus.get_topic("lighttopic")
service_bus.get_subscription("lighttopic","lightsubscription")

Looping and Logic

Now comes the interesting part. We will do a basic 'While True' loop (an infinite loop) to actively listen for our Subscription.

while True:      
      msg = service_bus.receive_subscription_message('lighttopic', 'lightsubscription', peek_lock=False)
      if msg.body is not None:
        print(msg.body)
        if msg.custom_properties["led"]==1:
            print("turning on the LED")
            GPIO.output(GPIO_BCM_PIN, 1)
        else:
            print("turning off the LED")
            GPIO.output(GPIO_BCM_PIN, 0)

You can see that we have a few debugging statements in here (by printing the message body to the console, along with LED commands). You are free to delete them if you wish.

This code is pretty straight forward. Our 'msg' object gets whatever message came from the subscription, and then we look at the custom_properties of it. If there is an Object 'led' that equals 1, we turn the LED On. Otherwise, we turn it off. When we start to program Cortana and deal more with the publishing side of things, we will see how exactly we interact with this 'led' custom property.

Find and Insert your Azure Key

Yes, I know, I'm a tease. It was the first thing I started with and now I'm ending with it. The process is pretty straight forward:

  • Go to your 'CustomNameSpace' Dashboard on the Azure Management Portal
  • Click on 'Configure'
  • You will see see a section labeled 'shared access key generator'
  • The 'POLICY NAME' is your Key Name
  • The 'PRIMARY KEY' is your Key Value
  • Insert those two things into your code in their respective spots
  • See Figure 1. for help

Run Your Code!

And now you're ready to run! Everything should compile and work. If it doesn't, be sure to check out the attached code sample to see if you missed any parts.

That being said, it really doesn't do much. That's because right now it's only listening, but we're not sending anything! It's just sitting there and happily listening to an empty cloud. Next, we'll dive into Cortana and Windows Phone to publish a message to the cloud so that our Pi actually has a message to hear!

Attachments

Step 8: Creating Your Windows Phone App Part 1: Intro to Windows Phone 8.1

Introduction to Window's Phone 8.1

Despite its low market share, Window's Phone really is a great phone that gives you a lot of power as a developer. Because I know that it may not be the most frequently developed platform, and not everyone is familiar with C# here, I'll be trying to go a bit slower and with a bit more detail in this section of the instructable. If you already are comfortable with the Window's Phone platform, Natural Language Processing (SSML), and C#, you will probably be able to zip through this pretty quickly. Otherwise, grab a cup of coffee, find a comfy chair, and buckle up. We've got an app to write!

Setting up your Environment

First things first, as with all coding projects, is setting up our environment. If you've never developed with C# before, chances are you might not even have visual studio installed. Let's get it all set up.

Not too hard, right? Visual Studio is a big download and quite a large software package. It may take some time to install, but it usually works out of the box pretty well. Next we'll get started on actually creating our Windows Phone App!

Step 9: Creating Your Windows Phone App Part 2: Configure and Navigating Your Project

Creating Your Application

Let's go ahead and create our Windows Phone 8.1 project, then we can walk through some of the nitty gritty and discover what's inside.

  • Figure 1 - Click on 'New Project...'
  • Figure 2 - Look under: Templates > Visual C# > Store Apps, and you should see an project template named 'Blank App (Universal Apps)'. Select it, and type in a name for your Home Automation app.
*NOTE* - I named my app 'BACH', which stands for 'Badass Automated Cloud Home'. You're free to name your project whatever you please, but any reference to 'BACH' in my code or pictures should be replaced with your own project name

Configuring your Project

If you have never used Visual Studio before, this may look a little daunting (and I also apologize you've missed on what I believe to be the best IDE hands down). I'll try to walk you through the thousands of different knobs and buttons pretty specifically, so don't worry.

An important first step is to take a look at Figure 3 and understand a bit what is going on. Remember how we click "Universal App" back on that project template? Well that's because this app could actually be deployed on Windows Phone and Windows Desktop (8.1 versions and higher, of course). That's why we see two different projects within our 'Solution Explorer' on the right-hand side. Everything under BACH.Windows(Windows 8.1)? Just ignore it. We won't be developing a desktop app in this tutorial.

Another thing to take in mind is that within our code files, we will see a lot of "Platform Specific Code". That's code that looks like this:

#if WINDOWS_PHONE_APP
        private TransitionCollection transitions;
#endif

That's how the compiled app knows if it should a piece of code or not, depending on if it is deployed as a Windows 8.1 app or as a Windows Phone 8.1 app. Why does this matter? Well, because Visual Studio is pretty slick, and we can actually tell it what we are currently working on. In Figure 3 you'll notice I have also circled something in the upper left corner: a tab that currently says BACH.Windows. That's basically us telling Visual Studio that we are currently working on the Windows app, except that we don't want it to say that! Instead, let's switch it over to say BACH.Windowsphone (Figure 4).

Now, we also need to tell Visual Studio that when we click "Run Program" we want it to run the Windowsphone version, not the Windows version. To do that, we will simply right-click on the Bach.WindowsPhone (Windows Phone 8.1) project and select 'Set as StartUp Project' (Figure 5).

Navigating the Project Structure

Now that we've configured everything to run in 'WindowsPhone' mode, lets do a very quick overview of your project structure (Figure 6):

  • Properties Beyond Scope of Guide
  • References Other libraries you might use, we'll add some libraries letter to connect to our Azure Service
  • Assets Where you store all those pretty pictures. You'll see some already in there by default. We won't be dealing with this folder within this guide.
  • MainPage.xaml > MainPage.xaml.cs Your "Design" logic and "Code" logic, respectively. We could make an entire guide in itself about how to code these two documents. Basically what you need to know is this is the "Front Page" of your app.
  • Package.appxmanifest App Name, requirements, packages, etc. We will be turning on the "Internet" permissions later on.
  • BACH.SHARED
    • App.xaml > App.xaml.cs What's all this 'Shared' nonsense? Well, remember this app is 'Universal', so this is code that is shared between your desktop and phone app. This is where we will put all of the Azure Service calls.

Hopefully that makes you a bit more comfortable with the Windows Phone structure, and don't feel overwhelmed. If you do, or would like to learn more, I would highly suggest you check out Channel9's video tutorial:

Windows Phone 8 Development for absolute beginners - Channel9

Step 10: Creating Your Windows Phone App Part 3: Azure on Windows Phone 8.1

Where Things Get Complicated...

That super nice, easy to explain format we've had for so long? Yup. Say goodbye. This specific step, while not hard can be really confusing for anyone not familiar with basic REST services and network protocols. In the interest of not blowing this instructable into a full on novel, we will treat the Azure Service call here within Windows Phone as essentially a "Black Box" call. That means we will copy and paste some code down and just compile it. We won't walk through it, and some of you may have no clue what it even does, but that doesn't really matter, because it works.There is a great list of resources at the end of this guide that can help you understand it better, but for now, let's just bite our lip and press forward.

Installing Libraries and Permissions

We will be using some fancy internet protocol libraries to access Azure since unfortunately there is not currently a specific Azure SDK for Windows Phone (yet). In order to do that, we will have install a Package(a library) from the NuGet Package Manager.

  • Figure 1 - Right Click on the WindowsPhone project and select 'Manage NuGet Packages...'
  • Figure 2 - Search for 'Json' in the search bar on the upper right, and click on 'Json.NET', then 'Install'
  • Close out of the Package Manager

Great, we've got the libraries we need. Now we need to enable the 'Microphone' and 'Internet' capabilities for our app. This will let our app allow us to use Cortana and post messages to our Azure Service Bus.

  • Double Click on 'Package.appxmanifest'
  • Figure 3 - Switch to the Capabilities Tab
  • Figure 3 - Make sure that 'Internet(Client and Server) and 'Microphone' are checked.

Now, at the top of your file, insert the following code to allow us to reference all these new fancy packages within our code:

using Newtonsoft.Json;<br>using Windows.Security.Cryptography;
using Windows.Security.Cryptography.Core;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;

Coding the Azure Call

The attached code file AzureCall.cs is not the entire App.xaml.cs file, it is simply 3 functions that you should copy and paste into your App.xaml.cs file. I would recommend copy and pasting these 3 functions underneath the OnSuspending function. Let's do a very brief overview of these 3 functions:

  • SendSBMessage - The primary function you will call. We will dive deeper into this function below.
  • SASTokenHelper - This function helps us encode out SAS token to log in to our Azure Service Bus
  • HmacSha256 - This function is purely there for Cryptography. If you don't understand it, don't worry.

Briefly Looking At SendSbMessage()

Let's just very briefly peek into this message. Specifically, I want us to look at this piece of logic within the function:

 HttpContent content = new StringContent(json, Encoding.UTF8);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
content.Headers.Add("led", message);
string path = "/lighttopic/messages";
var response = client.PostAsync(path, content).Result;

Specifically, let's look at that call 

content.Headers.Add("led", message);
string path = "/lighttopic/messages";

Remember that 'lighttopic' is our Topic name, and that in our Python script on our Raspberrry Pi we checked for the header variable attached to the `led` header. Hopefully this is starting to come together a little bit. Essentially, this function takes in a 'Message' which is going to be a 0 or 1 for either "OFF" or "ON". We will attached that to our LED Header and then send it off to our Azure Service Bus.

Attachments

Step 11: Creating Your Windows Phone App Part 4: Cortana

Introducing Cortana

Cortana is the personal assistant within Window's Phone, and soon the entire Windows ecosystem. She is Microsoft's answer to Apple's 'Siri', and a very strong answer at that. One of the most important features of Cortana is the third party support for app developers. This means that we can actually use Cortana within our app and interact with her through our own personal app that we developed rather than be restricted to just commands she is programmed with.

How we will use Cortana: An Overview

For our App, we won't get too deep into all the different ways we can leverage Cortana simply because there isn't enough time. We'll hard-code Cortana to respond to 2 distinct phrases:

"Turn my lights Off" - Will send a message to the Azure Service Bus to turn off our lights

"Turn my lights On" - Will send a message to the Azure Service Bus to turn on our lights

I've provided some more resources for learning more about Cortana and different ways we can develop with her at the end of this guide but without dissecting every little detail of Cortana, let's learn just enough to get by:

SSML - Speech Synthesis Markup Language is how we tell Cortana what phrases to listen to and basic responses.

Installing - When we first install our app, we won't be able to interact with our app through Cortana until we manually start the app for the first time. This is because opening our app installs the voice commands Cortana needs to recognize our app.

Pick your App name Wisely - We can't directly interact with our app through Cortana, we have to tell Cortana that we want to use the commands from a specific app, rather than her general list. For example, if I made an sports app and told Cortana "What is the Score of the Packer Game?" she wouldn't use the information from my app, because she doesn't know that my app can provide such information (she instead would look it up herself and give you the correct answer anyway). So instead, we have to say "SportsApp, what is the score of the Packer game?". This tells Cortana that the command 'what is the score of the Packer game?' belongs to the 'SportsApp' and she should consult that app to give the proper feedback.

Want to learn more about Cortana and her features? I highly recommend Channel9's excellent video lecture on her.

Step 12: Creating Your Windows Phone App Part 4: Coding Cortana - SSML

SSML

For the first part of coding Cortana we will focus on the SSML or Speech Synthesis Markup Language document. This xml document, also called the will allow us to program Cortana and tell her what to listen for and what kind of phrases should open our app. First things first, let's go ahead and create our SSML document.

  • Figure 1 - Right click on the WP8.1 project, click Add and then New Item
  • Figure 2 - Scroll to find the Voice Command Definition template, and create it with a name. I named mine ControlCommands.xml
  • Click OK to create the document.

Navigating SSML

This might look overwhelming at first glance, but that's just because the template is nice enough to fill it with tons of examples. In reality, we don't need a lot of this code, and once you get the hang of it is actually pretty simple. Let's take a look at the first block at the top:

<CommandSet xml:lang="en-US">
	<CommandPrefix> Contoso Rodeo </CommandPrefix>
	<Example> play a new game </Example>

Figure 3

The 'CommandPrefix' tag is what we talked about in the last step: Your app's name as known by Cortana. This doesn't really even have to be your actual app's name, it could be anything. I suggest something easy that you are going to remember. Remember that I named this app 'B.A.C.H" so I am going to insert 'bach' as my CommandPrefix.

The 'Example' tag is a suggestion to the user. When they scroll through Cortana and see all the available apps, it can show them suggested things to say.

Now erase the other large tag blocks in the file!

What? Why did I have you do that? Well, because we're using KISS in this project (Keep It Simple Stupid). At the risk of boring/teasing you to death with all the different SSML write-up we can use for Cortana, we will keep it simple and just use what we need (Also, inserting xml code on Instructables is very difficult due to their editor!).

So what do we need to put in? Let's insert the following command block:

<Command Name="DeskLightsOn">
	<Example> turn on my desk lights </Example>
	<ListenFor> turn on my desk lights </ListenFor>
	<Feedback> Turning On Your Desk Lights... </Feedback>
	<Navigate />
</Command>

So what the heck is going on here?

  • Command - the tag block is what we reference in the actual code. When we refer to this later on, we'll call it 'DeskLightsOn' after it's name attribute.
  • Example this is, again, the suggested input to the user for this command
  • ListenFor - What phrase 'triggers' this command to 'fire' . Pretty self explanatory hopefully.
  • Feedback - What Cortana responds with (she actually speaks it out loud).
  • Navigate This would be used it we were going to a specific page in the App, but we're not, so let's just leave it blank for now.

Take a look at Figure 4 to see the XML code for both the "ON" and "OFF" commands for the desk lights. If you want to check out the entire code file, you can download it from the attached file.

A Final Note

The above usage of Cortana is very primitive, and I will fully acknowledge there are quite a few things we could do to improve this code. However, this isn't an instructable about Cortana (Coming Soon though!), it is simply an introduction to her. If you wish to learn how to leverage and improve this SSML even more, I highly suggest you check out my resources guide at the back of this instructable!

Attachments

Step 13: Creating Your Windows Phone App Part 4: Coding Cortana - Logic

Cortana's Logic

Now that Cortana can hear you, we need her to understand you. This part of the guide goes into the 'backend' of Cortana and pairing her with our app's logic. We want to program her so that when she hears one of these commands, she sends a message to our Azure Service Bus and turns on or off the lights.

Installing the Voice Commands

Don't Forget! This is a problem a lot of people have when first using Cortana, simply because it is easily overlooked. Let's do it right away to make sure it is done and over with.

Put the following code in your MainPage.xaml.cs document.

First, import the correct reference calls at the top of your file:

using System.Threading.Tasks;
using Windows.Media.SpeechRecognition;
using Windows.Storage;

Then the code to install our voice commands can be written as a function like so:

private async Task InstallVoiceCommandsAsync()
{
	var storageFile = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///ControlCommands.xml"));
	await VoiceCommandManager.InstallCommandSetsFromStorageFileAsync(storageFile);
}

Not familiar with Async and Await? Read up on the MSDN documents here.

Now that we have our function, let's make sure to call it! A good spot to put it is within the OnNavigatedTo function.

Because our install is an asynchronous task, we will have to turn the OnNavigateTo function into an async function by simple adding the following code:

protected override async void OnNavigatedTo(NavigationEventArgs e)

Then we just check that this is the first time the page has been navigated to and install our voice commands:

if (e.NavigationMode == NavigationMode.New)
{
	await InstallVoiceCommandsAsync();
}

Great! Now our voice commands will install as soon as open the app for the first time!

Voice Command Logic

Now, we will use what command the user spoke to figure out which command to send to our Azure Service Bus. Open up your App.xaml.cs file and scroll to just below the RootFrame_FirstNavigated function. We will put the following function under there:

protected override void OnActivated(IActivatedEventArgs args)
		{
			base.OnActivated(args);
			if (args.Kind == ActivationKind.VoiceCommand)
			{
				var commandArgs = args as VoiceCommandActivatedEventArgs;
				if (commandArgs != null)
				{
					SpeechRecognitionResult speechRecognitionResult = commandArgs.Result;
					var voiceCommandName = speechRecognitionResult.RulePath[0];
					
					switch (voiceCommandName)
					{
						case "DeskLightsOff":
							SendSBMessage("0");
							break;
						case "DeskLightsOn":
							SendSBMessage("1");
							break;
					}
				}
			}
			Window.Current.Activate();
		}

This function is called every time somebody uses our Command Prefix (BACH) when giving Cortana an instruction. While it may seem a little confusing at first, the real important part are the following lines of code:

switch (voiceCommandName)
	{
		case "DeskLightsOff":
			SendSBMessage("0");
			break;
		case "DeskLightsOn":
			SendSBMessage("1");
			break;
	}

We use a switch statement to jump through the different commands in our SSML document. What we are checking is for the String to match up with our Command Name from our Command blocks in the ssml doc. Once we know what command we have received, we can then act accordingly by firing our SendSBMessage which sends a message to our Azure Service Bus. For "Off" we send a '0' and "On" we send a '1'.

Code not working 100%? Don't worry, I've attached both a MainPage.xaml.cs and App.xaml.cs that should give you more clarity when following along.

Attachments

Step 14: The Final Product!

You Did It!

With a few simple clicks, you can deploy your app to your Windows Phone device and you are all set! Initially, you'll simply see a black screen (Figure 1), but that is nothing to be alarmed about. Remember we didn't really code any GUI for this project and simply based it entirely around the voice commands with Cortana.

By holding down the search button you can activate Cortana. Simply say

"BACH turn on my desk lights"

Cortana should return with the message that she is turning them on (Figure 2).

Hopefully, this will trigger your LED to go on! Congrats!

Moving Forward...

In this guide, we've only just barely scratched the surface of what this technology is capable of. Here are just a few ideas that you can take going forward to really flesh out your automated home with Cortana and Azure:

  • Hook up your music player to Cortana, set wake-up alarms!
  • Hook up a relay with your RPi to control large lights
  • Use Cortana's Geofencing feature to have the lights turn off when you leave your house (Guide on this coming soon!)
  • Create a GUI interface on your Windows Phone app to control your lights
  • Create an Azure Storage Table and keep the 'status' of different IoT devices around your home and send them to your App!

Got more ideas? Post them in the comments and I will add them to the list!


Step 15: Troubleshooting and References

Troubleshooting

I keep getting the error:

Error	2	The type or namespace name 'SpeechRecognition' does not exist in the namespace 'Windows.Media' (are you missing an assembly reference?)

Yes, this is a strange one. I actually made two of these projects (one to test it, the other as a code-along). On one project I did have this error, and the other I didn't, despite identical code. The problem is that 'SpeechRecognition' is a library only available to Windows Phone, and not Windows in general. Because this code is in our 'App.xaml.cs' file, it's technically shared code. To get around this, simply wrap it in a Windows Phone 8 IF:

#if WINDOWS_PHONE_APP
using Windows.Media.SpeechRecognition;	
#endif

That will make it exclusive to your Windows Phone project, and should compile.

I keep getting a "Authorization Denied" when trying to connect to my Azure Service Bus through the Windows Phone app

Let me guess, you were the rebellious one who decided to name all their variables on their own? Double check that you have substituted all my instructable names with your own custom names. Doing a 'CTRL-F' on certain variables works well.

It takes almost 20 seconds from giving the command to my phone to the light actually turning on/off, what gives?

Yes, I've encountered this too. I've seen the 'delay' range anywhere from 10-30 seconds, with an average of just around 17 seconds. My initial thought is that this slow-down is coming from the code on the RPi, and it isn't 'listening' to the subscription as fast as we would hope for. If I find a fix for it in the future, I will add to this instructable.

I will post more troubleshooting tips as people comment on various problems!

References

Raspberry Pi

Azure

Windows Phone 8.1 and Cortana