Using Cognitive Services with Xamarin Forms

I did a talk at DotNext conference about this and thought I share the demo here as well. In this demo we’ll create simple Xamarin Forms app for Android and UWP, camera and add the Cognitive Services Emotion and Computer Vision APIs to analyze the photo taken by the camera.

Setup

It all starts with File, New Project… and selecting from under Cross Platform (in Visual Studio 2015) project type Blank App (Xamarin.Forms Portable). Once you have created the project, you can remove the other projects, but leaveyourapp, yourapp.droid and yourapp.UWP for this demo.

Next you’ll need to add three Nuget -packages by selecting the Solution from the Solution Explorer and right clicking it and Manage Nuget packages for solution…. Go to Browse tab, and type Microsoft.ProjectOxford and hit enter.

Now you are presented with a list of all Cognitive Services Nuget packages, but you need to select only two from the list. First select Microsoft.ProjectOxford.Emotion and from the right side panel, click only the main project, not the yourapp.Droid or yourapp.UWP projects. Now select from the list Microsoft.ProjectOxford.Vision and repeat the previous selection.

We need to have the camera access, so we’ll add one more Nuget package, Xam.Plugin.Media (type that to Browse search box like previous) but this time you need to install this to all projects, yourapp, yourapp.droid and yourapp.UWP.

We need to do one more step inside the Nuget package manager. Go to updates tab, empty the field where you typed the package names and find the Xamarin.Forms from the list and update it to the latest version (if you don’t empty the search box, you will not see anything on updates list).

Now we need to ensure that security requirements are in place, you need to open Package.appxmanifest in the yourapp.UWP project and from the Capabilities -tab select Pictures Library and Webcam and save it. There’s one last step we need to do for configuration before we’re ready start coding, and that is to include the UWP version to build (it’s not by default, which is weird!). On the toolbar select the Debug dropdown and Configuration Manager from the list. Make sure that Build and Deploy options are selected for yourapp.UWP project.

Next step is to get the keys to use to authenticate your app with Cognitive Services. Open the link Cognitive services subscription and login with your Microsoft account to request keys for Computer Vision and Emotion.

Code

We’ll implement a simple UI with a button to snap a photo and send it to Cognitive services and image to show the image from the camera. We’ll implement all this code within the app.cs so it will run on all platforms without need for any platform specific code. For the first step we’ll add the required using statements to the beginning of the file:

using Microsoft.ProjectOxford.Emotion;
using Microsoft.ProjectOxford.Vision;
using Plugin.Media;
using System.Diagnostics;

Next we’ll add the following member variables inside your app -class:

string key = "YOUR_EMOTION_API_KEY", key2 = "YOUR_COMPUTER_VISION_KEY";

Now copy your new key values inside to those variables from the webpage and you’re all set to start calling Cognitive services.

Then we need to add the following member variables under the key -variable that you just added in previous step:

Image image = new Image();
StackLayout layout;

The image is for showing the camera photo and layout is the forms base layout control, which will hold all the controls for the app.

Inside the app.cs go to the public App() -constructor and delete everything else but leave only the

MainPage = new NavigationPage(content);

Now we add the UI for the app just above that Mainpage -line like this:

// The root page of your application
layout = new StackLayout
{
    BackgroundColor = Xamarin.Forms.Color.Gray
};
var button = new Button
{
    HorizontalOptions = LayoutOptions.Center,
    Text = "photo"
};
button.Clicked += Button_Clicked;
layout.Children.Add(button);
layout.Children.Add(image);
var content = new ContentPage
{
    Content = layout
};

We set the background color ourselves as Android is using by default dark theme and UWP light theme, so the text will be visible. Next we create a button in the middle of the screen and add event handler for it, and lastly we add image and button controls to the screen layout.

Now we are ready to take the photo, and call the cognitive services with our photo. We’ll add the button click handler where we do all this by adding this method:

private async void Button_Clicked(object sender, EventArgs e)
{
    // Camera magic
    await CrossMedia.Current.Initialize();

    if (!CrossMedia.Current.IsCameraAvailable || !CrossMedia.Current.IsTakePhotoSupported)
    {
        return;
    }

var mediaOptions = new Plugin.Media.Abstractions.StoreCameraMediaOptions
{
    CompressionQuality = 100,
    PhotoSize = PhotoSize.Full,
    Directory = "Faces",
    Name = $"{DateTime.UtcNow}.jpg"
};
var file = await CrossMedia.Current.TakePhotoAsync(mediaOptions);
}

The first line is initializing the camera function, then we’ll check there is a camera on the device which can take a photo before continuing. The camera options are set so it knows where to save the images and using what filename (we use current date+time). Last line opens the platform specific camera launcher and returns the file from there which includes the photo which the user took.

Now we’re ready to send this image for analysis on the cloud. Add the following code in the end of Button_Clicked -method:

if (file != null)
{
    image.Source = ImageSource.FromFile(file.Path);
    var emotionClient = new EmotionServiceClient(key);
    var imageStream = file.GetStream();

    // Emotion API
    var emotion = await emotionClient.RecognizeAsync(imageStream);
    Debug.WriteLine($"Anger: {emotion[0].Scores.Anger : #0.##%}");
    Debug.WriteLine($"Contempt: {emotion[0].Scores.Contempt: #0.##%}");
    Debug.WriteLine($"Disgust: {emotion[0].Scores.Disgust: #0.##%}");
    Debug.WriteLine($"Fear: {emotion[0].Scores.Fear: #0.##%}");
    Debug.WriteLine($"Happiness: {emotion[0].Scores.Happiness: #0.##%}");
    Debug.WriteLine($"Neutral: {emotion[0].Scores.Neutral: #0.##%}");
    Debug.WriteLine($"Sadness: {emotion[0].Scores.Sadness: #0.##%}");
    Debug.WriteLine($"Surprise: {emotion[0].Scores.Surprise: #0.##%}");

    // Computer Vision
    var imageStream2 = file.GetStream();
    var visionClient = new VisionServiceClient(key2);
    VisualFeature[] visualFeats = new VisualFeature[]
    {
        VisualFeature.Description,
        VisualFeature.Faces
    };

    var analysis = await visionClient.AnalyzeImageAsync(imageStream2, visualFeats);
    Debug.WriteLine($"{analysis.Faces[0].Gender}, age {analysis.Faces[0].Age}");
    Debug.WriteLine(analysis.Description.Captions[0].Text);
    foreach (var tag in analysis.Description.Tags)
        Debug.WriteLine(tag);
}

On the code above we’ll set first the image control to show the image we captured. For the next step we use the key we requested earlier to open connection to the Emotion api service in the cloud, and then get stream of the image which we send to the API for analysis. Next you should check in real code that you actually got results, but here we assume it worked and just dump output to the console, formatting the results in percentage numbers.

In the Computer Vision -section of the code we need to get another stream as the previous call closed the earlier stream, and then set the visual features which we want to analyze from the picture. There are more options available, but more you ask, the slower it is, so we just want to get Description of the image and faces of the image this time. We call the AnalyzeImageSync with the file stream and visual feature request and again assume we get results, and dump them to the output console inside Visual Studio.

That was pretty easy to implement, but was a bit tricky for a beginner so I thought this would be useful guide for someone. Hope you enjoyed and let me know if you did or didn’t!

About Jani Nevalainen

Windows developer who's been working on Microsoft Platforms since 1996. Windows Phone dev MVP 2013, Windows Development Platform MVP 2014. Currently working as Technical Evangelist for Microsoft Finland Developer Experience team.
This entry was posted in Cognitive Services, Universal Apps, UWP, Xamarin. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *