#cognitiveservices

Experts say automation can boost retail banking solutions in this modern era. It's mainly because innovation and automation help businesses gain resilience and agility during a crisis. Due to such reasons, all industries are going through a digital makeover.

Even though the transformation level is still at the teething stage, the retail banking sector is not that far from administering technology. According to a study, around 90% of banks have said that cognitive and RPA technologies are crucial for the future of business.

This declarative approach is called the application modernization strategy. It involves dealing with critical application elements in legacy software and is considered the lever for business transformation and API management platforms.

 To Read more “Application Modernization and its Impact on Business Transformation”

 Mphasis (BSE: 526299; NSE: MPHASIS), an information technology (IT) solutions provider specializing in cloud and cognitive services.

... or "How to remove a thumbnail from an mp3 using ffmpeg?"

Update 2020-12-10: Since writing this post, the team at Microsoft have rolled out an update of the transcription service that allows embedded media, so the workaround described in this blog post won’t be necessary any more. They have also introduced a new, more powerful API (v3) for Batch transcription; an introduction is available here.

While building the site RadioText.net (which I've blogged about here) I found that a few of the files I tried to transcribe returned an undocumented error: "The recordings URI contains invalid data." and I'll describe the solution to the (at least my) problem below.

The problem

So, the problem appeared when using Azure Speech Services - Batch transcription. Speech Services is one of the AI services in Azure provided through Azure Cogntitive Services. Batch Transcription allows for large scale, asyncronous transcription of audio files.

Initiating the transcription went fine is done using a HTTP POST like this (which of course can vary depending on your preferences):

HTTP POST: https://westeurope.cris.ai/api/speechtotext/v2.0/Transcriptions/

{ "Name":"RadioText - Episode 1482554", "Description":"RadioText", "RecordingsUrl":"https://STORAGEACCOUNT.blob.core.windows.net/media/Audio.mp3", "Locale":"sv-SE", // ... }

The error is returned to you when you are querying for the transcription status:

HTTP GET: https://westeurope.cris.ai/api/speechtotext/v2.0/Transcriptions/TRANSCRIPTION-GUID

{ "recordingsUrl": "https://STORAGEACCOUNT.blob.core.windows.net/media/Audio.mp3", // ... "statusMessage": "The recordings URI contains invalid data.", "status": "Failed", "locale": "sv-SE", // ... }

Video stream / JPG

After some investigation, the issue is seems undocumented but is related to that the batch service requires that the audio file being transcribed need to contain audio streams exclusively. The mp3 files I were trying to transcribe had an embedded video stream containing a jpg file - basically, the thumbnails used in media players. At the moment, batch transcription can't handle audio files containing such image/video stream (confirmed with the team at Microsoft).

Solution

Until support for these media files is built into the service, it's easy for us to extract only the audio channel ourselves using ffmpeg. The following command will extract the first audio channel from Input.mp3 and output it as Output.mp3 - ready to use for batch service. It's documented under Stream selection in the ffmpeg documentation.

ffmpeg -i Input.mp3 -map 0:a -codec:a copy Output.mp3

If we break it down:

ffmpeg: The utility

-i Input.mp3: Use Input.mp3 as input, could also be a URL (https://STORAGEACCOUNT.blob.core.windows.net/media/Audio.mp3).

input docs

-map 0:a: Use the audio (a) from the first input file (0)

map docs

-codec:a copy: Set codec option for audio (a) to only copy, this will make it very efficient as it does not need to encode anything.

Stream copy

Output.mp3: Output the new file that only contains the audio stream(s) into Output.mp3

There might be other options or tweaks that work for you, but I found this to be fast and work well to solve my problem.

Calling ffmpeg from CSharp

In my case, I wanted to run the above ffmpeg command as part of my transcription pipeline and therefore call it from C#. The below code is a minimal approach to do so:

const string ffMpegLocation = "PATH_TO_FFMPEG.EXE"; var inputFile = "Input.mp3"; var outputFile = "Output.mp3"; var processStartInfo = new ProcessStartInfo { FileName = ffmpegLocation, Arguments = $"ffmpeg -i \"{inputFile}\" -map 0:a -codec:a copy \"{outputFile}\"", UseShellExecute = false, RedirectStandardOutput = true, CreateNoWindow = true }; var process = Process.Start(processStartInfo); process.WaitForExit();

Contribute

RadioText.net is a site that transcribes news episodes from Swedish Radio and makes them accessible. It uses multiple AI-based services in Azure from Azure Cognitive Services like Speech-to-Text, Text Analytics, Translation, and Text-to-Speech.

By using all of the services, you can listen to "Ekot" from Swedish Radio in English :) Disclaimer: The site is primarily a technical demo, and should be treated as such.

Background

Just to give you (especially non-Swedish) people a background. Sveriges Radio (Swedish Radio) is the public service radio in Sweden like BBC is in the UK. Swedish Radio does produce some shows in languages like English, Finnish and Arabic - but the majority is (for natural reasons) produced in Swedish.

The main news show is called Ekot ("The Echo") and they broadcast at least once every hour and the broadcasts range from 1 minute to 50 minutes. The spoken language for Ekot is Swedish.

For some time, I've been wanting to build a public demo with the AI Services in Azure Cognitive Services, but as always with AI - you need some datasets to work with. It just so happens that Sveriges Radio has an open API with access to all of their publically available data, including audio archive - enabling me to work with the speech API:s.

Architecture

The site runs in Azure and is heavily dependant on Cognitive Services. It's split into two parts, Collect & Analyze and Present & Read.

Collect & Analyze

The collect & analyze part is a series of actions that will collect, transcribe, analyze and store the information about the episodes.

It's built using .NET Core 3.1 and can be hosted as an Azure function, Container or something else that can run continuously or on a set interval.

The application periodically looks for a new episode of Ekot using the Sveriges Radio open API. There is a NuGet-package available that wraps the API for .NET (disclaimer, I'm the author of that package...). Once a new episode arrives, it caches the relevant data in Cosmos DB and the media in Blob Storage.

JSON Response: https://api.sr.se/api/v2/episodes/get?id=1464731&format=json

The reason to cache the media is that the batch version of Speech-to-text requires the media to be in Blob Storage.

Once all data is available locally, it starts the asynchronous transcription using Cognitive Services Speech-to-text API. It specifically uses the batch transcription which supports transcribing longer audio files. Note that the default speech recognition only supports 15 seconds because it is (as I've understood it) more targeted towards understanding "commands".

The raw result of the transcription is stored in Blob-storage, and the most relevant information is stored in Cosmos DB.

The transcription contains the combined result (a long string of all the text) the individual words with timestamps. A sample of such a file can be found below:

Original page at Sveriges Radio: Nyheter från Ekot 2020-03-20 06:25

Original audio: Audio.mp3

Transcription (Swedish): Transcription.json

This site only uses the combined result but could improve the user experience by utilizing the data of individual words.

All of the texts (title, description, transcription) are translated into English and Swedish (if those were not the original language of the audio) using Cognitive Services Translator Text API.

A sample can be found here: https://radiotext.net/episode/1464731

All texts mentioned above are analyzed using Cognitive Services Text Analytics API, which provides sentiment analysis, keyphrases and (most important) named entities. Named entities are a great way to filter and search the episodes by. It's better than keywords, as it's not only a word but also what kind of category it is. The result is stored in Cosmos DB.

The translated transcriptions are then converted back into audio using Cognitive Services Text-to-Speech. It produces one for English and one for Swedish. For English, there is support for the Neural Voice and I'm impressed by the quality, it's almost indistinguishable from a human. The voice for Swedish is fine, but you will hear that it's computer-generated. The generated audio is stored in Blob Storage.

Original audio: Audio.mp3

English audio (JessaNeural, en-US): Speaker_en-US-JessaNeural.mp3

Swedish audio (HedvigRUS, sv-SE): Speaker_sv-SE-HedvigRUS.mp3

Last but not least, a summary of the most relevant data from previous steps are denormalized and stored in Cosmos DB (using Table API).

Present & Read

The site that presents the data is available at https://radiotext.net/. It's built using ASP.NET Core 3.1 and is deployed as a Linux Docker container to Dockerhub and then released to an Azure App Service.

Currently, it lists all episodes and allows for in-memory filtering and search. From the listing, you can see the first part of the transcription in English and listen to the English audio.

By entering the details page, you can explore the data in multiple languages as well as the original information from the API.

Immersive reader

Immersive Reader is a tool/service that's been available for some time as part of Office, for example in OneNote. It's a great way to make reading and understanding texts easier. My wife works as a speech- and language pathologist and she says that this tool is a great way to enable people to understand texts. I've incorporated the service into Radiotext to allow the user to read the news using this tool.

Primarily, it can read the text for you, and highlight the words that are currently being read:

It can also explain certain words, using pictures:

And if you are learning about grammar, it can show you grammar details like what verbs are nouns, verbs, and adjectives:

I hadn't used this service before, but it shows great potential for making texts more accessible. Combined with Speech-to-text, it can also make audio more accessible.

Cost

I've tried to get a grip on what the cost would be to do run this service and I estimate that to run all services for one episode of Ekot (5 minutes) the cost is roughly €0,2. That includes transcribing, translating, analyzing and generating audio for multiple languages.

Speech pricing

Translation pricing

Text analytics pricing

Also, there will be a cost for running the web, analyzer, and storage.

Ideas for improvement

The current application was done to showcase and explore a few services, but it's not in any way feature complete. Here are a few ideas on the top of my mind.

Live audio transcription: Speech to text supports live audio transcription, so we could transcribe the live radio feed. Could be comined with subtitles idea below.

Improve accuracy with Custom Speech: Using Custom Speech we could improve the accuracy of the transcriptions by training it on some common domain-specific words. For example, the jingle is often treated as a words, while it should not.

Enable subtitles: Using the timestamp data from the transcription subtitles could be generated. That would enable a scenario where we can combine the original audio with subtitles.

Multiple voices: A natural part of a news episode are interviews. And naturally, in interviews, there are multiple people involved. The audio I'm generating now is reading all texts with the same voice, so in scenarios when there are conversations it sounds kind of strange. Using conversation transcription it could find out who says what and generate the audio with multiple voices.

Improve long audio: The current solution will fail when generating audio for long texts. The Long Audio API allows for that.

Handle long texts: Both translation and text analytics has limitations on the length of the texts. At the moment, the texts are cut if they are too long, but they could be split into multiple chunks and then analyzed and concatenated again.

Search using Azure Search: At the moment the "search" and "filtering" functionality is done in memory, just for demo purposes. Azure Search allows for a much better search experience and could be added for that. Unfortunately, it does not allow for automatic indexing of Cosmos DB Table API at the moment.

Custom Neural Voice: I've always wanted to be a newsreader, and using Custom Neural Voice I might be able to do so ;) Custom Neural Voice can be trained on your voice and used to generate the audio. But, even if we could to this, it doesn't mean we should. Custom Neural Voice is one (maybe the only?) service you need to apply for to be able to use. In the world of fake news, I would vote for not implementing this.

Disclaimer

This is an unofficial site, not built or supported by Sveriges Radio. It's based on the open data in their public API. It's built as a demo showcasing some technical services.

Most of the information is automatically extracted and/or translated by the AI in Azure Cognitive Services. It's based on the information provided by Swedish Radio API. It is not verified by any human and there will most likely be inaccuracies compared to the source.

All data belongs is retrieved from the Swedish Radio Open API (Sveriges Radios Öppna API) and is Copyright © Sveriges Radio.

Try it out and contribute

The Source code is available at GitHub and Docker image available at Dockerhub.

Working more and more with Cognitive Services in Azure, I saw the need for a tool to explore and visualize the API.

Cognitive Workbench is a site to learn about and demo the capabilities of Azure Cognitive Services. The site aims to be a complement to already existing demos, samples and sites to make it easier to try out the capabilities, visulize results and explain how the services works.

The sites is targeted towards people familiar with Microsoft Azure and Cognitive Services and will require you to create resources in Azure and bring your own keys.

You can use the site at: https://cognitive-workbench.azurewebsites.net/

The Sourcecode is available at GitHub and Docker image available at Dockerhub.

As of today, the site covers these services and API:s:

Vision

Computer Vision

AnalyzeImage (ImageType, Faces, Adult, Categories, Color, Tags, Description, Objects, Brands)

AreaOfInterest

RecognizeText

RecognizePrintedText

BatchReadFile

Face

Detect (Age, Gender, HeadPose, Smile, FacialHair, Glasses, Emotion, Hair, Makeup, Occlusion, Accessories, Blur, Exposure, Noise)

Samples

Computer Vision

Input

Analyze

OCR

Face

Contributions

Feel free to contribute. A few ideas of improvements are available as GitHub issues.

As a technical person and developer, I'm a big fan of XKCD which makes entertaining comics about our industry. A couple of years ago I stumbled upon this one:

Turns out that it was published September 2014, almost exactly 5 years ago, the time the team would need to solve the problem according to the comic. So, I thought, this should be solved now, right? And it turns out it did!

Today I'm proud to announce: BirdOrNot.net

BirdOrNot.net uses the Artificial Intelligence (AI) powered object detection capability in Azure Cognitive Services to find out if the image contains a bird and, in some cases, it even finds out what species it is. I didn't have to hire a team of researchers for five years, instead Microsoft did and made the result available as a service in Azure.

Using that specific service, I was able to go from idea to working concept in under an hour, with a few more hours spent on CSS :)

Below you can see a few samples, feel free to try it out yourself on the site: https://birdornot.net/

It builds on the default object detection model provided by Azure Cognitive Services, which can detect anything from buildings, vehicles to animals. It does a very good job of detecting birds, but as it's not specialized on birds, there will be cases where it makes mistakes. Still, shows how the power of AI and ML can be used by everyone who knows how to call a webservice.

If you boil it down, these are the few lines that powers the site:

var imageAnalysis = await computerVisionClient.AnalyzeImageAsync(url, new List { VisualFeatureTypes.Objects }); var isBird = imageAnalysis.Objects.Any(x => x.ObjectProperty.Equals("bird")); Console.WriteLine(isBird ? "It is a bird." : "It is not a bird.");

You can see a simple sample with the code above at GitHub.

Behind the scenes

The source code is fully available at GitHub, feel free to dig around to learn how it works. If you want to run it yourself, the image that runs on the site is available at dockerhub (peterorneholm/orneholmbirdornotweb).

docker run --env AzureComputerVisionSubscriptionKey=XYZ --env AzureComputerVisionEndpoint=ABC peterorneholm/orneholmbirdornotweb

For those of you that are interested, here comes a brief explanation on the setup.

Architecture

Object recognition

The service is using Azure Cognitive Services to analyse the image. It fetches the following info:

VisualFeatureTypes.Objects (Used for detecting the birds and getting their bounding rectangles. Checks the object hierarchy to detect species.)

VisualFeatureTypes.Description (Used to describe the image with text and tags.)

VisualFeatureTypes.Adult (Used to decline images classed as Adult, Gory or Racy.)

Code

The website is written in ASP.NET Core 3 and packaged as a container running Linux.

Hosting

The container is published to dockerhub and then automatically deployed to Azure App Service for easy management and scale out if needed.

Cache

The costly part in the application (both money and time) is the analysis of an image, therefore every unique URL is cached. The application uses the IDistributedCache interface and is configured to use Azure cache for Redis (hosted) as the implementation.

In case the site would go viral, I'm distributing it through Azure Frontdoor. By doing so I have HTTP cache at the edge so that the samples and landing page will have no impact on the backend and be served immediately.

Analytics

The site is monitored through Azure Application Insights / Azure Monitor. I store anonymous telemetry on how the site is used and by collecting feedback on wheter or not the audience agrees with the AI motor, it could be used to improve over time.

I've activated a custom alert that will send me an email in case the traffic to the site would rapidly increase.

Cost

There are a few basic costs involved. But the part that could become expensive is the analytics of images. I've estimated it to cost roughly $5 USD / 1000 images analyzed. By caching the results I'm preventing the samples to cost me any money and I will be able to handle the cost for quite a lot of images.

Next steps

Here are a few things I've thought about implementing:

Custom Vision: I would like to add som custom models to detect "Bird like" things, like Big Yellow Bird, Jack Sparrow etc. I could use Azure Custom Vision for this, just give me some time and I'll fix it :)

The XKCD above also mentions the task of detecting if the photo is taken in a national park. I've thought of using Azure Maps to find the POI based on GPS coordinates stored in the picture.

Make it a PWA and publish it to Store(s)

Interested in even more details?