Our evening at Cognitive Services with Martin Beeby

By Rachel Breeze | 01/09/2016

Have you ever uploaded a photo to a website and wondered, how on earth has that recognised my face?!  Or used the Skyscanner bot in Skype and puzzled over how it returns the exact results you were after? Have you ever wished you could evolve your website from a purely brochure-ware feel to something a little more interactive? Or wanted to make your apps more useful?

Well, all of these questions and more were answered at Cognitive Services with Martin Beeby (@thebeebs) for the Manchester .Net Meetup that both Dave Brislane (@davebrislane) and myself went along to last week. It was a pleasure to see Martin speak and learn from his experiences around Microsoft's Cognitive Services offering. We delved further into the workings of Cognitive Services and saw a myriad of examples of how they give developers access to readymade machine learning interfaces. These machine learning interfaces are a fantastic tool to work with as they don't require us to understand their functionality or make up; we can just dive straight in and use them. 

"Machine Learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data." - Techtarget 

Basically, it can apply itself to any scenario where a pattern can be recognised in the data. Pretty cool, right?

Microsoft spend around £7 billion each year on research worldwide, with Cambridge acting as the main base for machine learning research and Cognitive Services app development.

So how exactly do they work? 

When calling the Cognitive Service, you are asking it a question and it will respond with an answer that includes the probability of being correct. This probability is known as "confidence". For example, the Face API performs auto-detection of a face which can return a lot of information including gender and age, as well as recognising where the face is located on a photo: 

Facial recognition of Dave's key features

In addition to this, the emotion API can provide analysis from a photo about how a person is feeling and it's worth noting, the answer is in percentages, showing the likelihood of each emotion. Here's some analysis of our team, enjoying a night out (the score is for the person at the far right).

Emotional analysis of the team's faces

The second analysis below is for the person third from the left: 

Facial recognition software of emotions

Microsoft's APIs

Vision: Computer Vision, Emotion, Face & Video 
Speech: Custom Recognition Intelligence Service, Speech Recognition & Speech
Language: Bing Spellcheck, Language Understanding Intelligent, Linguistics Analysis, Test Analytics & Video Language Model 
Knowledge: Academic Knowledge, Linking Intelligence Service, Knowledge Exploration Service, Recommendations
Search: Bing Autosuggest, Bing Image Search, Bing Name Search, Bing Video Search, Bing Web Search. 

Combining APIs 

The APIs don't have to be used by themselves; you can combine them to perform interesting outcomes. Martin demonstrated combining the Face API with a Recommendations API. He took a photo of himself and then used this to drive a recommendation from the result. The Face API indicated he was probably 39 and male; the recommendations API suggested a power tool! 

APIs can also be combined with Azure functions to "do stuff" when something happens in Azure. 

Training Cognitive Services 

Most of the services that Microsoft offer are available via Azure and out of the box. However, due to the nature of some services, they require training. For example, Skyscanner had to train it's bot to recognise travel phrases such as "I want to go to Amsterdam next Sunday" and before it can implement recommendations, the recommendations API needs knowledge of the products in your system.

It is also possible to train the services to respond to events that happen on the Azure IoT Suite. For example, this can be used to predict maintenance schedules for pieces of equipment. 

Implementing Cognitive Services in Your App/Website

Microsoft have provided some handy SDKs (software development kits) for each Cognitive Service and access to each service is configured via Azure. The code then makes a call to service and the call responds with a JSON response, that the code then consumes. 

Costs vary per service and preview services are only available to be configured in Azure in the West US location (and may be something that needs to be considered if you want quick responses). 

Sounds Great! I want to Play


Microsoft have some example applications available here: https://www.microsoft.com/cognitive-services/en-us/applications. Better yet, My Moustache has implemented facial recognition and scored moustaches!

An application rating Dave's moustache - he's in the top 90%.

As does https://www.twinsornot.net/

Can't recognise your dog breed? 

https://www.what-dog.net/ and the great thing is when using these apps you can tell the app they got it wrong (sturdy, feisty, and sweet yes – Norfolk Terrier no). By providing feedback you’re helping with the machine learning process, and improving the service.

A Golden Retriver incorrectly identified as a Norfolk Terrier




If you want to use the SkyScanner bot previously mentioned, in Skype click on “Add new bot to your contact list”

Skype Bots Icon Details of the Skyscanner Bot in Skype The Skyscanner Bot, responding to "I want to fly to London tomorrow" Skype Skyscanner Bot conversation asking about flights to San Francisco Skype Skyscanner Flight Suggestions

Other Takeaways

In addition to what I've discussed, we learnt that in Azure you can configure custom dashboards to group the services together and also, we can use Microsoft Azure Storage Explorer to examine the files (previously I was using CloudXplorer). 

In Summary 

This was a brilliant event and we are feeling inspired and left with a lot of ideas. We’ve shown what we’ve seen to our Head of UX, Chris Bush (@suthen) who has already thought of some brilliant ideas from a UX perspective. Watch this space as we investigate what Cognitive Services can do for us. Thank you Martin!