Voice-controlled Drone

This Instructable will guide you through how to make a voice-controlled drone. If you are experienced in coding and would like to try coding the app interface from scratch, there will be several steps to help lead you on the right track. Don't worry if you do not know how to code! You will still be able to work on this project.

What you need:

- Crazyflie 2.0 Drone

- Mac computer with Xcode (free from the App Store)

- iPhone or iPad with iOS 9.0 or higher

- Apple ID (preferably a developer ID, but not required)

- Optional: 3D printer and OnShape (or other CAD software) account

Step 1: Create a Nuance Developer Profile

Whether you are looking to build the app from scratch, or if you simply would like to use our app, you will still need your own profile from Nuance. Do not select "Want Nuance Mix Beta?". Once you have confirmed your account, log in and view your sandbox credentials. Copy and paste the URL, SSL Host, SSL Port, Add ID, and App Key under "SpeechKit 2.x" into an easily accessible document. You will need this information later.

Step 2: Download the App

We customized the Crazyflie Example Project by adding in an additional button for speech that works with the SKSASRViewController Class in the SpeechKit Example App, and a slider to adjust the power as the battery drains. If you have a foundation in Objective-C or related coding language, you can try building the app from the ground up. Steps 3-6 will outline the basics of creating an app like we did. If you don't like the idea of building the app yourself, you can just download our finished app from Github here. Follow steps 7-8 to get it running on your phone.

Step 3: Getting Started With the SDK's and Cocoapods

Skip to Step 7 if you would not like to build your own app.

Once you have downloaded the Crazyflie project and the SpeechKit project, you first have to install cocoapods to Crazyflie. The process is outlined as follows:

1. Open Terminal and type "sudo gem install cocoapods"

2. Enter your password

3. Type "pod setup --verbose". If you get no response from terminal, don't worry - the set up will likely take a few minutes to complete.

4. If you have Xcode open, quit it.

5. Type "cd ~" followed by the path to the folder containing your Xcode file with the Crazyflie project. You should omit the /Users/Name/ part of the directory, so your path should be something like "/Downloads/crazyflie2-ios-client-master" unless you have moved it elsewhere.

6. Type "pod install"

That's it! Cocoapods is set up for Crazyflie, and you are ready to start looking at the projects in Xcode. Make sure that you are viewing "Crazyflie client.xcworkspace" instead of "Crazyflie client.xcodeproj" or the pods will not show up correctly.

Step 4: Adding Nuance Developer Credentials

1. Open the "SpeechKitSample" project and input your credentials that you copied earlier. Ensure that everything is correct or the app will not be able to communicate with the server.

2. For language, put: @"eng-USA";

3. Run the app and play around with the various documents. Try looking at "SKSASRViewController.m" under the "Controllers" group. This code contains the speech recognition methods necessary for this project.

4. Copy the SpeechKit.framework file onto your desktop.

***Alternatively, if you would not like to use Nuance, you can try OpenEars, an open source speech recognition process that only recognizes commands you tell it to. Although OpenEars is far less accurate than Nuance, its limited set of recognizable commands is actually helpful for this app, as you only need a couple of commands to fly the drone. Also, OpenEars does not require internet or any sandboxing credentials like Nuance does.

Step 5: Exploring the Crazyflie Client

Now that you have cocoapods installed, you can open the Crazyflie client. Focus mainly on the ViewController.m file and the method sendCommander: in particular. If you are not well acquainted with NSTimers, I recommend you read the Apple documentation or find some other source of information about them. You should not have to edit the other documents, and be careful what you change in the ViewController class, as deleting even a single line can cause many errors to pop up and be fatal to the app. Just remember to hit ⌘Z if anything goes wrong.

Also, make sure that you select the general Crazyflie Client project information and change the bundle identifier to your own. Switch the team to your Apple ID, and click "Fix Issue" if applicable.

This client does not run well in simulator, and cannot communicate with the drone in simulator. To run the app, plug in a device first, wait for it to recognize it, then select it from the upper left menu and hit run.

Once you have flown the drone a little bit, begin altering the code to see what happens. Just remember to undo any fatal errors and re-download the project if you completely destroy it (that happened to me about 6 times).

Step 6: Combining Crazyflie and SpeechKit

Now that you are well acquainted with these two apps, combine them by moving things from the SpeeckKitSample App to the Crazyflie Client App.

1. Copy the SpeechKit.framework file INTO THE PROJECT FOLDER. Then navigate to Crazyflie Client > Build Phases > Link Binary With Libraries > + > Navigate to SpeechKit.framework

2. Copy the SKSConfiguration.h and SKSConfiguration.mm files that you added your credentials to into the project.

3. Navigate to the ViewController.m file and add the following:

#import "SKSConfiguration.h"
#import

4. Copy the void methods found in SKSASRViewController.m into Crazyflie and implement them accordingly. A statement in the -(void) recognizeSpeech:(NSString *)message{} method will allow you to run code specific to each command, by comparing the received string (variable named "message") against predefined strings:

if ([message isEqualToString:@"Your command here"]) {
//Run code for your command

}

Also, remember that string comparison is case-sensitive, so make sure that any strings you fill in for "Your command here" have their first letter and ONLY their first letter capitalized, or the if statement will be false.

Alright! It's up to you to finish up the project. There is lots of trial and error that goes into this, so just be patient, and reference our app if you need. You can skip steps 7 and 8.

Step 7: Getting the Project Working in Xcode

Steps 7 and 8 are only necessary if you are NOT coding your own drone control app

1. Download this project as a zip. Once downloaded, unzip the file before proceeding.

2. Open Terminal and type "sudo gem install cocoapods"

3. Enter your password

4. Type "pod setup --verbose". Note that this will likely take a few minutes to complete.

5. If you have Xcode open, quit it.

6. Type "cd ~" followed by the path to the folder containing your Xcode file with the project. You should omit the /Users/Name/ part of the directory, so your path should be something like "/Downloads/Drone-Voice-Control" unless you have moved it elsewhere.

7. Type "pod install"

8. Open the Xcode .xcworkspace file and navigate to the top item on the left-hand list, "Crazyflie Client"

9. Click "General," type your own bundle identifier in the top field under "Identity," select your own Apple ID under "Team," and click "Fix Issue" if it asks you to.

10. Navigate to the SKSConfiguration.mm file, and paste the Nuance developer credentials you saved earlier. These must be in string format (@""), so if, for example, your App Key was 12345, the line of code would look like this:

NSString* SKSAppKey = @"12345";

11. The line involving SKSLanguage should look like this:

NSString* SKSLanguage = @"eng-USA";

12. Ignore the SKSNLUContextTag

13. Plug in your iPhone or iPad, select it from the list, and click Run.

Step 8: Tweaking the Code

Although it may seem scary, you will likely need to tweak the code to suit your particular drone.

Navigate to the ViewController.m file and find the "-(void) recognizeSpeech:" method. Under each speech request in an if(){ } statement, you will see something like "self.commanderTimer = [NSTimer scheduledTimerWithTimeInterval:0.05 target:self selector:@selector(flyDrone:) userInfo:@[@0.75,@0,@0,@0] repeats:YES];"

- The userInfo: parameter defines (in order) thrust, pitch, roll, and yaw (see picture).

- Thrust (the amount of power passed to the motors) can be any value from 0 to 1

- Pitch, roll, and yaw can be any value from -1 to 1, with 0 being off and 1/-1 being maximum in either direction.

- If you are having trouble understanding how to modify the code, I recommend using StackOverflow.

Step 9: CADing a Plastic Drone Shell

The plastic motor mounts that the Crazyflie 2.0 comes with are very susceptible to breaking, so it can be helpful to 3D print a more sturdy shell. Along with this benefit, 3D printed shells can be designed to better protect the drone than the motor mounts do, and they add more weight, allowing for significant improvement in the stability of hovering. Although this step is not required (and impractical if you don't have access to a 3D printer), it is helpful to design a drone and stick to that particular weight, as you will have to tweak your values in Xcode for every change in weight that you make.

- Check out our example shell here (.stl). Simply import this file into the app that accompanies your 3D printer (e.g. Cura, Autodesk Print Studio) and the software will take care of it from there.

- Want to try by yourself? Try Onshape or Fusion 360. Onshape is completely free and Fusion has a 30 day free trial. Fusion also allows students 3 years of free access.

- If you would like to edit our project, you will need Fusion 360 and this file (.f3d).

- New to CAD? Learn here.

Step 10: Assembling the Drone

If you printed our drone shell, follow the directions below to get the drone assembled. If you want to use the standard company assembly, follow these directions. Otherwise, skip to the next step.

1. Bind the battery to the underside of the drone chip. Having the back side up makes attaching the motors much easier, so instead of using the included battery case, I simple taped it to the chip.

2. Insert the motors into their positions

3. Attach the motor wires to the board. The dimensions of the drone shell are just enough for the cables to reach. If you are having trouble, try unwinding the motor wires.

4. This set up allows for the drone to sit flat, which allows for easier calibration and take off.

Step 11: Understanding the App

Refer to the image to find out about the UI elements in out example app.

That's it!