Turn your Raspberry Pi into a Personal Assistant using Cortana

Microsoft is constantly updating its latest version of Windows, version 10. For me, as a developer, it’s a wonderful operating system to program for. The UWP apps I build, run on both PC’s, laptops, Windows Surface Hub (up to 84 inches), The Xbox One and even on a Raspberry Pi. Yes, Windows 10 is running on a 35 dollar device.

But before you run to the store to replace your PC, I have to tell you it’s running the core of Windows 10, actually. There is no shell (no menu, no start bar etc.).

So this means you can run one visual (headed) UWP application and multiple background applications. And yes, you will love it!

This is a great interface for kiosk-like devices. And with the latest update (build 15063), it’s easy to add Cortana support.

Cortana is the speech service, available in Windows 10. If you know Siri or Alexa, then you know Cortana. Just ask her a question and she will try to answer it. The answer will be provided by speech or supported by browsers or other visual help.

Let’s take a look on how to enable Cortana on a Raspberry Pi.

Hardware

First of all, you need a Raspberry Pi 2 or 3, because these are supported by Windows 10 IoT Core. I can recommend version 3 because the first one tends to be sluggish.

Cortana must be able to hear you, you need a microphone. I have this Microsoft LifeCam Cinema so I tried and it is supported! And Cortana must be able to speak so give her some speakers. Again, I had some cheap ass USB speakers laying around and they were accepted.

The hardware list of ‘officially’ supported gear is pretty short but it seems that mainstream USB speakers and microphones (like in a webcam) are supported if these work on your own laptop running Windows 10.

Cortana needs a face. Give her a screen using the HDMI port. I was experimenting with an 800×480 screen. It works but this is pretty poor when Cortana has more information to show (in a browser eg.).

Next to speech, you can interact with Cortana using touch or mouse/touchpad/keyboard. A software keyboard is available in the newest version of Windows 10 IoT Core but a physical keyboard is much easier.

Cortana is not actually living on your device, she is part of the cloud. So make sure your internet connection is up and running.

And there is one more thing, invest into a USB power supply with enough amperes. Because the extra devices are attached, the will drain from the same power source. In my case, my Raspberry Pi kept rebooting.

Installation and Windows Device portal

If you are not already running Windows 10 IoT Core on a Raspberry Pi. You will have to flash a MicroSD card. For this, you use the IoT Dashboard. It is essential for all users wanting to get started.

With this tool, you can flash a card and even add wifi support to it.

Once the card is flashed, plug it into your Pi, plug all other devices in and connect the power to your Pi.

The Raspberry Pi will boot for the first time and install Windows 10 Core. You will receive some questions so answer them. One of the questions could be about enabling Cortana. A Windows account could be requested for this. For Cortana usage, there will be more occasions to enable it, later on.

Connect to Windows 10 Core using the browser. Each Windows 10 has a ‘web browser’ running for extra configuration. On Windows 10 IoT Core this option is enabled by default:

Double-click the device line and open the Windows Device portal:

A browser will be opened and credentials are requested. If you have not changed them (which is recommended) use ‘administrator’ and ‘p@ssw0rd’.

Check out the Device portal

At several places, you will have to enable Cortana. But first, let’s look if your hardware is accepted.

At the first page of the portal, the hardware will be shown:

Here you can see that my USB speakers are accepted and my Microphone has a certain level of sensitivity set. You can change both sliders.

In comparison, this is the same dialog without USB devices:

Note: in the right upper corner you should be able to see you are actually running “v.10.0.15063.0”.

Note: this is also the page where you can check if the right Time zone and time is set.

And this is a good moment to enable the Remote Server:

This way you can access the Pi even if no screen is attached! I can recommend using the Windows IoT Remote Client UWP app on your PC while you are experimenting with your Raspberry PI. This tool shows the screen of the Pi within the app and you can interact with it.

Note: several screenshots in this blog are made using this Remote Client.

Enable Cortana on boot

And at the bottom of the first screen of the Device Portal, where the microphone and speakers settings are shown, enable this checkbox ‘Start Cortana on boot‘:

Now the Cortana background service will be running on your Pi when it boots.

Let Cortana respond

Let Cortana actually respond when you call her.

The Raspberry Pi should show something like this right now:

:

Notice that both the connected devices and IP address are shown here too.

Navigate to the settings here (the gearing wheel in the upper right corner) and within this menu, select Cortana:

If not already enabled, do it now.

If asked for, you could ‘personalize’ Cortana by providing your personal Windows account credentials (just follow the wizard):

Activate Cortana as default app

The last step is to enable Cortana as the default application in the Device Portal.

At the Apps Manager page, all currently installed and running headed or background applications ar shown.

A headed app named ‘Cortana’ is available too. So enable it (check the startup radio button):

For a short while, a magnifying glass is shown. And soon after, Cortana will appear:

Note: Cortana knows my name. This screenshot is taken after I trained Cortana.

And there you are, Cortana wants your attention. So ask her anything, just start saying ‘Hey, Cortana’. And when sound waves are shown, ask your question.

For example, “Hey Cortana, what’s the weather forecast in degrees celsius” gives:

Because Cortana is currently only supporting a limited amount of countries (read: The USA and a few others) and I live in Europe, degrees Fahrenheit is not that practical to me. But I did not find a way to set the right (metric) system. So the height of the Mount Everest is shown in 29,029 feet.

Cortana having issues

But maybe you get something like this:

There are several little issues with Cortana. First of all, a stable network connection is appreciated! Without a network connection, Cortana is away. And especially on the Raspberry Pi 2, Cortana is a bit sleepy for the first couple of calls. She need to warm up 🙂

Personalizing Cortana

Cortana is positioned as a Personal Assistant. I am not quite sure what the status is for Cortana on the Raspberry Pi. Simply because this device is NOT a personal device, she is more of an entity on herself. The Raspberry Pi is more like a kiosk, everybody can ask her questions. And for now, she is not able to switch between persons.

That’s why Cortana will already work even if no Microsoft Account is supplied.

But there are several ways to make her a bit more personal.

First, during the first boot, you will be asked to enter the credentials of a Windows  Account on behalf of Cortana.

Secondly, in the default application, you can personalize Cortana:

For example. you can learn her about your favorite places:

This is just the same wizard as shown on your PC or your Windows Phone when you activated Cortana.

And the last and easiest way is to ask Cortana: “Hey Cortana, make an appointment”. This will activate the wizard too:

But right after you signed in, you get this message:

Cortana has no notion of your calendar on the Raspberry Pi.

And the same goes for taking a note of making a picture. We have to wait for that:

Cortana knows my name

You can ask Cortana “Hey Cortana, how do you call me”. In my case, she calls me ‘Sander’. Why? Because I told her so.

Just ask “Hey Cortana, change my name” and she opens the name settings:

You will need a keyboard to enter your own name. Cortana will let you hear how your name sounds.

If you have an on-screen keyboard activated, that will look like this:

Then ask her again “Hey Cortana, how do you call me”…

Advanced settings

It is possible to get and set multiple types of languages used by Windows 10 IoT Core:

IoTSettings -get speechlanguage
IoTSettings -get region
IoTSettings -get uilanguage
IoTSettings -list uilanguage
IoTSettings -list speechlanguage
IoTSettings -set uilanguage language_tag - (e.g.: fr-CA  => set default UI language French Canadian)
IoTSettings -set speechlanguage language_tag - (e.g.: fr-CA => set speech language French Canadian)
IoTSettings -set region region_code - (e.g.: CA => set default region to Canada)

We need a command prompt to do this. In the Device Portal, go to Processes, Run Command. Here we can enter a command at the top and we see the result of the execution at the bottom.

This gives results like this:

Conclusion

Ok, this is a 35 dollar device, running something amazing like a Personal Assistant. I was very impressed by the simplicity to get things work. And yes, it’s not perfect because not all features are available yet.

But imagine, Cortana is not a raspberry Pi. Cortana is put in a shiny cylinder with sublime speakers and a stellar microphone array… There is evidence that Microsoft will come up with it, this year. Just imagine…

Advertenties