New Post has been published on DREW5.net
New Post has been published on http://drew5.net/apps/cognitive-services-computer-vision-api-powers-teamstream-video-search/
Cognitive Services Computer Vision API powers Teamstream Video Search
Teamstream and its clients have a problem understanding the content inside their video libraries—a problem not just for them, but for everyone who uses traditional methods of archiving and retrieving videos. A video is typically created by a team, indifferently tagged, and saved out to a file in some repository—a file server or a cloud-based shared service like YouTube. In both cases, the videos are stored in a folder structure which is organized by client and by project. This type of methodology is extremely limiting when searching for particular videos. What is the job number? Which client was that? Did we do similar videos? Is this the right version? The difficulty multiplies when you want to search for specific content within videos. If a video is not tagged or named correctly—which is often the case—it can be impossible to find what you’re looking for. Teamstream Productions was looking for a “fix” that offered deep searching capabilities within a set of videos, a fix that would be accessible through multiple channels so that if the team is on the go or giving a presentation, they can access this search through mobile or web.
Cognitive Services Computer Vision API
Teamstream Productions is a content production and marketing firm that services some of the biggest consumer brands in the world. They work to understand each client’s business and communication goals, then use their expertise to create, publish, and promote content that services those goals across all channels, including videos, web and mobile content, social promotion, traditional print, events, and more.
Solution, Development, and Delivery
We believe Cognitive Services API is a great choice for us on this initiative for two reasons: First, it offers an already robust set of search parameters that can be utilized to sort videos. Second, its our understanding that the capabilities set is constantly expanding, which is a powerful differentiating factor as we offer this solution as an ongoing SaaS offering to clients—they’re signing on for a service that will only grow richer and better with time. – Keith Blanchard CEO Teamstream Productions
Microsoft Azure Search would sit on top of this data and return the results to a traditional web search but more importantly to a Bot Framework search that can be used via the Web Chat, text messaging and with Slack and Microsoft Teams.
There are many parts to this project but in essence it is broken down into 2 parts. The administrative tasks such as uploading the videos and then client side for searching the videos and the client side to search for the videos.
For the administration side is where most of the heavy lifting will take place.
ASP.NET MVC Web Application and Azure Web App Service to provide the tasks of uploading the videos to Azure Storage.
Azure Function with a Blob Trigger to start the video “slicing” and moving those images to another Azure Blob Storage.
A second Azure Function with a Blob Trigger will take each of the new images and call the* Cognitive Service Computer Vision API* to get the tags and descriptions for each of the frame images.
Azure MS SQL Database for storing all of the data.
ASP.NET MVC Web Application and Web App Service
We first created out Web App Service out on Azure for our ASP.NET Web Application which will be the main integration point for the Admin tasks as well as the Bot. Then we created our ASP.NET Web Application project in Visual Studio 2017.
Our first step was to build out the views and handle the form data. We wanted to use a lightweight and fast object mapper for .NET so we chose Dapper. This will extend our IDConnection interface and allow for quick and easy access to our DB. There is a NuGet libary that makes it easy to add by issuing the following command
We then used a Repository Pattern for our DB interaction. In order to help with the Object Relational Mapping we used
internal interface IVideoRepository List<Video> GetVideos(); Video GetVideo(Guid videoId); bool InsertVideo(Video video); void DeleteAllVideos(); public class VideoRepository : IVideoRepository private readonly IDbConnection _db; public VideoRepository() _db = new SqlConnection(ConfigurationManager.ConnectionStrings["SQLConnectionString"].ConnectionString); ... public bool InsertVideo(Video video) int rowsAffected = this._db.Execute(@"INSERT Video([Id],[Name],[Keywords],[RawUrl],[Date]) values (@Id, @Name, @Keywords, @RawUrl, @Date)", new Id = video.Id, Name = video.Name, Keywords = video.Keywords, RawUrl = video.RawUrl, Date = video.Date ); if (rowsAffected > 0) return true; return false; ...
Then in our Admin controller constructor we can instantiate our repository and send the data from the form to our database.
public AdminController() _videoRespository = new VideoRepository(); ... [HttpPost] public async Task<ActionResult> UploadAsync(Video video) ... if (ModelState.IsValid) _videoRespository.InsertVideo(video);
Now that data is being stored we can update our UploadAsync method to include sending the video to Azure Blob Storage.
First we need to make sure we add the WindowsAzure.Storage package. We can then reference the libraries needed.
using Microsoft.WindowsAzure.Storage; using Microsoft.WindowsAzure.Storage.Blob;
Here is the full UploadAsync method:
[HttpPost] public async Task<ActionResult> UploadAsync(Video video) CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString")); // Create a blob client for interacting with the blob service. blobClient = storageAccount.CreateCloudBlobClient(); blobVideoContainer = blobClient.GetContainerReference(blobVideoContainerName); await blobVideoContainer.CreateIfNotExistsAsync(); await blobVideoContainer.SetPermissionsAsync(new BlobContainerPermissions PublicAccess = BlobContainerPublicAccessType.Blob ); try string blobFileName = string.Empty; var client = new HttpClient(); HttpFileCollectionBase files = Request.Files; int fileCount = files.Count; if (fileCount > 0) for (int i = 0; i < fileCount; i++) blobFileName = GetRandomBlobName(files[i].FileName); //upload to Azure Blob Storage VIDEOS CloudBlockBlob blob = blobVideoContainer.GetBlockBlobReference(blobFileName); await blob.UploadFromStreamAsync(files[i].InputStream); if (ModelState.IsValid) video.Id = Guid.Parse(GetGuidFromBlobName(blobFileName)); video.RawUrl = baseUrl + "videos/" + blobFileName; _videoRespository.InsertVideo(video); return RedirectToAction("Index"); catch (Exception ex) ViewData["message"] = ex.Message; ViewData["trace"] = ex.StackTrace; return View("Error");
We now have the meta information from the user being saved into the DB along with the video moving over to Azure Blob Storage.
Admin ASP.NET MVC Web Application API
Eventually we will create Azure Functions that process these videos and images. We will need a way to send that data back to our app for storage in our Azure MS SQL Database. We will create an API endpoint that will allow those functions to send back data.
In order to add an API Controller to an existing project the following steps will be required:
WebApiConfig.cs file will need to be created in the App_Start directory to register your API Controller
public static void Register(HttpConfiguration config) config.MapHttpAttributeRoutes(); config.Routes.MapHttpRoute( name: "DefaultApi", routeTemplate: "api/controller/id", defaults: new id = RouteParameter.Optional );
In Global.asax.cs, the WebApiConfig needs to be registered with the GlobalConfiguration
GlobalConfiguration.Configure(WebApiConfig.Register);
Then a API controller can be created.
public class VisionAPIController : ApiController ... [HttpPost] public void Post([FromBody]Models.VisionAnalysis model) _visionRespository.InsertVision(model);
We will add a few more admin views but we are done with the Admin portion of the project.
Once a video is stored in the video blob, we want to trigger a function to process the video. We also needed a way to determine the length of the video and a way to create images from frames or certain locations within the video. We are using FFMpeg to help with this. NReco has a C# .NET wrapper for FFMpeg as well as a C# .NET Wrapper for FFProbe. These wrappers will help with getting duration of the video along with grabbing a frame within the video.
We decided to use an Empty Web Project for publishing a .NET Class Library for our Azure Functions. This would allow us to pull in packages as needed for building out our full functions.
When starting off with an Empty Web Project for use with Azure Functions there are a few key files and command line tools that are needed.
First, install the Azure Functions Core Tools. Once installed and once you create your Empty Web Project perform the following steps:
Open the project properties. In the WEB tab choose START EXTERNAL PROGRAM
For the program path, enter the path to func.exe from the Azure Functions Core Tools. Location will be C:\Users\USERNAME\AppData\Local\Azure.Functions.Cli\1.0.0-beta.93\func.exe if you installed the tooling or installed via NPM C:\Users\USERNAME\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\func.exe
For COMMAND LINE ARGUMENTS: set host start
For WORKING DIRECTORY, specify the root of the web project on your machine
Now when you run your project Azure Functions command line tooling will take over and you can still debug your Azure Function but it will be connected to whatever triggers you have set.
Speaking of triggers we will set those up now.
Our structure of our Azure Functions project is the following:
root – this will host appsettings.json
Utils – class library for common routines
Models – any db and json models
Functions – the actual code for our function
VideoTrigger – functions.json definition file our the trigger and function
ImageTrigger – functions.json definition file our the trigger and function
App_Data – our FFMpeg exe files
Lets look at appsettings.json. There are a couple of ways you can build this file. By hand. Pull it down manually via Kudo on Azure or via Azure Function Command Line Settings. You only need to pull it down by hand if you already configured your Azure Functions in the portal. That is what we did.
We first went into the Portal and searched for Azure Function App and configured it as an App Service Plan. Once we had the Azure Function created we opened up a command prompt and issued the following commands:
func azure functionapp list
func azure functionapp fetch-app-settings [name]
This then downloaded the appsettings.json (encrypted) and we moved this into our project root. This file contains variables for the storage accounts that functions will use as well as which ones will be triggered when new data is contributed to the account. Be cautious since this file contains secrets so you don’t want to check that into source control.
Azure Functions – Video Blob Trigger
Now that we have our project set up we need to create a definition file for our function. This will allow the Azure Function App to use a precompiled dll and the function entry point to start the job when it is triggered. We will create a file called function.json and add the following:
"disabled": false, "scriptFile": "..\\bin\\TeamStreamFunctions.dll", //Compiled DLL "entryPoint": "TeamStreamFunctions.VideoProcessor.Run", //What method and class to run "bindings": [ "name": "myBlob", //Variable name of the blob that will be passed into function "type": "blobTrigger", //Type of trigger "direction": "in", //Direction "path": "videos/name", //Path of blob "connection": "teamstreamstorage_STORAGE" //Connection variable in appsettings.json ]
We added this to our VideoTrigger directory (and we will do this for our other function as well and change the entry point and pathing for the blob). Now lets build our function!
We need to add a package from NuGet Microsoft.Azure.WebJobs and any other packages that we will be needing (Storage, Threading and NReco)
Here is the video trigger.
public static async Task Run(Stream myBlob, string name, TraceWriter log)
The first step is to build out some paths, one for the actual video and the location of the tooling (FFMpeg.exe and FFProbe.exe). We then get the duration of the video. This is used to determine how many frames/images we will get per video. We arbitrarily chose these numbers but will fine tune later on. Then we loop through the video and get the thumbnail from the video at a given frame location. Once we have the image we then move that over to the other Blob storage for hosting the image.
Azure Functions – Image Blob Trigger
We will create another file called function.json and put that in the ImageTrigger directory and add the following:
"disabled": false, "scriptFile": "..\\bin\\TeamStreamFunctions.dll", //Compiled DLL "entryPoint": "TeamStreamFunctions.ImageProcessor.Run", //What method and class to run "bindings": [ "name": "myBlob", //Variable name of the blob that will be passed into function "type": "blobTrigger", //Type of trigger "direction": "in", //Direction "path": "thumbnails/name", //Path of blob "connection": "teamstreamstorage_STORAGE" //Connection variable in appsettings.json ]
The Image Trigger will now kick off when a new image is added to the blob. This function will call the Computer Vision API and get the results back in JSON and then update the MS SQL DB accordingly.
public static async Task Run(Stream myBlob, string name, TraceWriter log) if (myBlob.Length == 0) return; string videoId = Utils.GetGuidFromBlobName(name); string result = await CallVisionAPIAnalyze(myBlob); VisionAnalysis data = JsonConvert.DeserializeObject<VisionAnalysis>(result); data.thumbnailUrl = Keys.baseUrl + "thumbnails/" + name; data.thumbnailIndex = Utils.GetThumbnailIndex(name); data.thumbnailCount = Utils.GetThumbnailCount(name); data.id = Guid.NewGuid(); data.videoId = Guid.Parse(videoId); await CallVisionAPISave(data); static async Task<string> CallVisionAPIAnalyze(Stream image) using (var client = new HttpClient()) var content = new StreamContent(image); client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", Keys.cogservAPIKey); content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream"); var httpResponse = await client.PostAsync(Keys.visionApiURL, content); if (httpResponse.StatusCode == HttpStatusCode.OK) return await httpResponse.Content.ReadAsStringAsync(); return null; static async Task<string> CallVisionAPISave(VisionAnalysis content) using (var client = new HttpClient()) string json = JsonConvert.SerializeObject(content); var requestData = new StringContent(json, Encoding.UTF8, "application/json"); var response = await client.PostAsync(Keys.videoApiURL, requestData); var result = await response.Content.ReadAsStringAsync(); return result;
For each of the images in the Blob we make a call to the Computer Vision API and then the results are sent to API endpoint within our Admin tool application.
The access to these results will be provided by a traditional search box as well as integration with the Bot Framework.
We chose to use the Bot Framework as the interface for the search for multiple reasons:
The Embedded Web Chat can serve just as good as a search box but could also direct the user into a faceted search
Using the Bot Channels, we can easily extend the search to Slack and Microsoft Teams where clients tend to work and communicate about projects.
Along with the channels mentioned above, access the search on the go was a key requirements and having the ability to integrate with SMS easily was also an added benefit.
Videos are being stored, images are being created and data is being saved. We now need to retrieve it. The first step is to create a View to retrieve the data we want to be indexed by the Azure Search. Then we set up our Azure Search to index our Azure SQL Database.
Now we can work on the Bot. We did a similar course of action by adding Bot Framework components to an existing application.
First we install the Bot Builder Library
Install-Package Microsoft.Bot.Builder
Then we add the keys value pairs to the Web.config (Remember, for local testing these values can be blank.)
<add key="BotId" value="" /> <add key="MicrosoftAppId" value="" /> <add key="MicrosoftAppPassword" value="" />
Now we added a MessageController class for the Bot to communicate with
public class MessagesController : ApiController public async Task<HttpResponseMessage> Post([FromBody]Activity activity) if (activity.Type == ActivityTypes.Message) using (var scope = DialogModule.BeginLifetimeScope(Conversation.Container, activity)) await Conversation.SendAsync(activity, () => scope.Resolve<IDialog<object>>()); else this.HandleSystemMessage(activity); return Request.CreateResponse(HttpStatusCode.OK); private Activity HandleSystemMessage(Activity message) if (message.Type == ActivityTypes.DeleteUserData) else if (message.Type == ActivityTypes.ConversationUpdate) else if (message.Type == ActivityTypes.ContactRelationUpdate) else if (message.Type == ActivityTypes.Typing) else if (message.Type == ActivityTypes.Ping) return null;
Now our project is setup for building the Bot.
We want the user to be able to search for videos using the Bot and its dialogs and prompts. We decided that we would use the Bot Search Dialogs.
The documentation was easy to follow to get up and running. A few tweaks where needed to the SearchHitStyle to accommodate the way we wanted our results to look.
var cards = hits.Select(h => new VideoCard Title = h.Name, Subtitle = h.Text, Image = new ThumbnailUrl Url = h.ThumbnailUrl , Media = new List<MediaUrl> new MediaUrl() Url = h.RawUrl , Buttons = new[] new CardAction(ActionTypes.ImBack, "Pick this one", value: h.Id) , );
The code above was to style the card collection.
We also had to modify the models, specifically the SearchHit to map to the results of our Azure Search.
public string Id get; set; public string Name get; set; public string Keywords get; set; public string RawUrl get; set; public string Tags get; set; public string Text get; set; public string ThumbnailUrl get; set;
Then the last modification was with the AzureSearchClient and using our search keys and index to return the proper results.
this.mapper = mapper; SearchServiceClient client = new SearchServiceClient( ConfigurationManager.AppSettings["SearchServiceName"], new SearchCredentials(ConfigurationManager.AppSettings["SearchServiceApiKey"])); this.searchClient = client.Indexes.GetClient("tags");
We created an IntroDialog that is invoked by the MessageController and will call the VideoSearchDialog to kick off the full dialog with the user to perform the search.
The dialog, search and mapper were all placed in a ContainerBuilder which was placed in the Global.asax.cs. A ContainerBuilder is an Autofac container based inversion pattern to help with dependency injection.
ContainerBuilder builder = new ContainerBuilder(); builder.RegisterType<IntroDialog>() .As<IDialog<object>>() .InstancePerDependency(); builder.RegisterType<TeamStreamMapper>() .Keyed<IMapper<DocumentSearchResult, GenericSearchResults>>(FiberModule.Key_DoNotSerialize) .AsImplementedInterfaces() .SingleInstance(); builder.RegisterType<AzureSearchClient>() .Keyed<ISearchClient>(FiberModule.Key_DoNotSerialize) .AsImplementedInterfaces() .SingleInstance(); builder.Update(Conversation.Container);
We were able to test locally using the Microsoft Bot Framework Emulator. The Bot Framework Emulator provides the ability to test your bot application locally as it would if it was being called by the Bot Framework Cloud Service.
The Bot Framework has a portal where we configured our Bot Application for publishing.
Our bot is now ready to go!
This project will save time because video is inherently difficult to manually search and to manually tag, and this reduces the amount of time for any client to spend on both ends. The reduced timeframe to find any particular dataset from a video library (e.g., “how many of our videos from the last five years are funny and feature cars”) makes new targeted sales initiatives possible, including being able to authoritatively answer questions about the contents of a video library within the context of an ongoing presentation.
We’ve completed the first phase of the project, with Juhi Saha’s team at Microsoft. At the start of their next fiscal year we’ll be applying for a continuation grant to take this to the next level and fully develop the mobile interface and finalize the back-end with whatever functionality we can include in the scope, and create a working mobile demo we can start showing to clients (using a small subset of their own video libraries). – Keith Blanchard CEO Teamstream Productions
There was a big time blocker with the Azure Functions and running an exe. We first tried adding the exe as a site extension however, Azure Functions are a site extension in itself. Reaching out to support for NReco provided a property that we can set for the path of the FFProbe and FFMpeg files. Once we did that it worked like a charm.
Another area that we needed to spend a bit of time on was with the DB. We originally planned for using Azure Cosmos DB as our data storage but there are limitations with the size of each document which we would quickly exceed with indexing each frame of a larger video.
We also started this project a few weeks back and since we begun the Microsoft Garage and Azure Media services team released product very similar to what we are accomplishing. The service is called Video Indexer. There is a REST API and a portal that you can upload your videos and gain insight from the frames within. Our build can be adapted to use this new service by adding the REST API calls within the first Azure Function Video Trigger. We would eliminate the need for the “slicing” of the video and sending it to a second storage container and issuing calls for each of the frames. The team will be looking into this for use in an upcoming release.
Cognitive Services Computer Vision API
Language and Understanding Intelligence Service
