Gerald & Matt

Implementation of the Cognitive Services + Xamarin Combo Challenge!

How to use Azure Speech-to-Text & Bing Photo Search with Xamarin

You definitely hear about the awesome Xamarin Combo Challenge from Gerald & Matt 😋. If no — please jut check the link below, otherwise, you may miss something in my story because I do not plan to describe Challenge details here and my story starts right after video ends

My Implementation Priorities

It so many opportunities with this Challange 🤔, so many ways to enhance this app, so many things to be done. But what things, what we wanna do there? Hmm 🤨, first of all, it shouldn’t take a lot of time from me, because I heard only on the third day of the Challange and have some other stuff to do in that week(that’s why this blog published only now) 🙁. The second one — this shouldn’t be some dummy changes like small code refactoring, re-styling of Main Screen, and other similar one hour changes just to make Pull Request(PR later). The third one does not make PR with staff that already was done. Quite simple requirements — right? 😎

First Problems

And here I got my first problem with 11 PR’s on the repo 😱. Most of the simple(but not only) and interesting things were already done. Like:

• Adding Shell
• Text-to-Speech with playing translated text to the user.
• Translating Picture Description
• Speech-to-Text with Translating recognized text.
• Polish localization(what??? 🤯)
• UWP Project for solution

So, what is left to implement? Here I stock for a moment with stuff to implement in my PR because obvious things already are done. During that time I’ve checked Travel Monkey Code, check how it was implemented that PR’s. Meanwhile, I recognized that no one(for that moment) touch Bing Search 😃, which was used in the original app to get photos for some common places(”Seattle”, “Maui”, “Amsterdam”, “Antarctica”) and shows them on the Main Screen.

Ideas for Implementation

So I imagine to add a new screen, let it be a “Plan Your Trip” Screen. With navigation from Main Screen via a button on it. I don’t have additional time that week. So the final implementation plan:

• One button with Speech-to-Text by Azure functionality
• Bing Search photos by request “Sightsee in {Place you Tell to the App}”
• Showing Collection of Top 10 Bing Photos with Indicator View.

All of that takes for me ~4 hours of coding and shame, shame on me, I try to set the correct width of the button at Main Screen for 1 hour!!! One Hour Carl!!!

I don’t know that was wrong with me and why I can’t set width dynamically to fit the text “Plan Your Trip” in my button but it was a most mystical task on that challenge. All other goes really well, Speech-to-Text and Bing Search are quite good documented(according to requested functionality in my app) and really, really easy to use. Now let’s speak more in-depth about the implementation process.

Button at Main Screen

So I want to have a button at Main Screen by which we could navigate to our new Screen. So let’s start with it. First of all, we should select a place for our button:

Also would like to pay your attention that the structure of the project is well organized, everything is in a place where you expect it. XAML code is just ok, nothing bad or special there. But one thing looks strange and wrong for me — how icons from Font-Awesome are used in XAML. Please never, never use icon itself(placed like text) at code neither like icon code. Please just go and create some static class with constants for all used icons in the project:

After that in all XAML files, you can easily identify what icons used there and easily change them or add more:

I will not say anything bad that all colors are set inline(no predefined color scheme) or no global styles defined — because it’s ok for such small pet-projects that should be written fast and never be enhanced or supported for real customers. But with such a well-ogranized project structure and ok XAML code, we must refactor it.

Now when my conscience is clean we could add our button. I want to have:

• Black Background Color
• With Rounded Corners
• And White Border Color
• With Text “Plan Your Trip”

To achieve that we have a couple options:

• Use Frame with Button inside
• Use PancakeView with Button inside
• Or use Grid with two BoxView (one for border and one for background color) with Button on the same level

Here I decide to use PancakeView:

Design of Plan Trip Page

As I mentioned I don’t have a lot of time, so I plan to create a simple screen.

It should be common with other screens so I add a Close button on the top of the page.

Under the Close Button, I plan to locate a button with Speech-to-Text functionality and some tip for the user to clarify what Screen expect to hear.

Here I show the other way to create a rounded button, without PancakeView with two BoxView.

And the last is a small Label where the user can see what was recognized and collection(CarouselView & IndicatorView) with photos from Bing.

So, UI is really simple and not looks so stupid. But we still can enhance it just by adding simple animations when Images are loading or when we scroll them. Also, it would be nice to add some layout for empty collection when no pictures are added 🤔

Speech-to-Text Implementation

Now, time to show how ViewModel for Plan Your Trip Page was implemented. And we start from adding CongnitiveService at Azure Portal. It’s not a big deal to do that, but if you need a details — please go and check Official Documentation for Speech Recognition:

Speech recognition using the Speech Service API - Xamarin

It’s the fourth Cognitive Service in this app now 😅. Next Step, like with other Azure Services is copying a Secret Key into ApiKeys.cs file. I’ve added it with a name CongnitiveServiceApiKey 😐, are you expecting any other name 🤨?

Now I create one more Service to manage Azure SpeechRecognizer . It’s a good practice and base idea of MVVM not to hold logic now connected with UI in separate Services with Interfaces for better code understanding, and easy way to writing Unit Tests, or switching to mock implementation. And in TravelMonkey Services for Bing, Computer Vision, and Translation Services already exist, it’s one more plus for this project. Yeah, it without Interfaces, but it’s ok for such a small project without Unit Tests or mock implementation. So, why then I’ve added it? Mean add an Interface right now?

Because we need to provide a separate implementation on each platform. Here I will show code for Android only, for iOS and UPW it will be 90% the same, more details you may found under the link with official Microsoft Documentation.

I will have only one button with three states:

And one global function ListenAndTranscribe with a callback when text is recognized TextRecognized :

Implementation for that will look like:

As I told you at the beginning it’s really easy to use Speech Recognition with Azure(in my case 😝)

Bing Images Search

As I mentioned in my plan was to search 10 pictures in Bing by $”Sightsee in {place}” search request, where place is what was recognized by Azure Speech-to-Text. Implementation is ~70% the same as for GetDestinations but it has some difference, same only request and in the project that should be supported or enhanced, we definitely should extract code which making request to Bing Search.

Plan Your Trip View Model

I think a good start point is a constructor, where we initialize our commands and services:

And here I would like to continue with CognitiveService_TextRecognized event handler where we will get a recognized text and make a request to Bing:

Still, quite stupid and simple code, let’s go further. Next is a SpeackAction :

And here we have something interesting 😮. To hear what users speak we need to get access to the microphone. We can do that quite simply with Xamarin.Essentials:

In a real application, we should also check if users do not click on “Never Show Again” or it was three times denied and we should show system permission screen, but we don’t need it in our simple example and can be happy just with that code 😇🥳

And that's it. It all code required for that screen and all our Challenge Implementation 🤠

Conclusion?

It was a really interesting Xamarin Challenge with a quite cool project and awesome Splash Screen Monkey animation. Challenge requirements allow making small changes and just have fun, which is most important for me.

So my first Xamarin Challange is completed and I with new skills is ready for the next one 🤩

Thanks, Gerald and Matt 😉🙃

Gerald

Matthew Soucoup