#ai-900

Analyzing images with the computer vision service

Analyze an image evaluate objects that are detect

Generate human readable phrase or sentence that can describe what image is detected

If multiple phrases are created for an image, each will have an associated confidence score

Image descriptions are based on sets of thousands of recognizable objects used to suggest tags for an image

Tags are associated with the image as metadata and summarizes attributes of the image.

Similar to tagging, but it can identify common objects in the picture.

It draws a bounding box around the object with coordinates on the image.

It can identify commercial brands.

The service has an existing database of thousands of recognized logos

If a brand name is in the image, it returns a score of 0 to 1

Detects where faces are in an image

Draws a bounding box

Facial analysis capabilities exist because of the Face Service

It can detect age, mood, attributes, etc.

Currently limited set of categories.

Objects detected are compared to existing categories and it uses the best fit category

86 categories exist in the list

Celebrities

Landmarks

It can read printed and hand written content.

Detect image types - line drawing vs photo

Detect image color schemes - identify the dominant foreground color vs overall colors in an image

Genrate thumbnails

Moderate content - detect images with adult content, violent or gory scenes

Classify images with the Custom Vision Service

Image classification is a technique where the object in an image is being classified

You need data that consists of features and labels

Digital images are made up of an array of pixel values. These are used as features to train the model based on known image classes

Most modern image classification solutions are based on deep learning techniques.

They use Convolutional neural Networks (CNNS) to uncover patterns in the pixels to a particular class.

Model Training

To train a model you must upload images to a training resource and label them with class labels

Custom Vision Portal is the application where the training occurs in

Additionally it can use Custom Vision service programming languages-specific SDKs

Model Evaluation

Precision - percentage of the class predictions made by the model that are correct

Recall - percentage of the class predictions the model identified correctly

Average Precision - Overall metric using precision and recall

Detect objects in images with the Custom Vision service

The class of each object identified

The probability score of the object classification

The coordinates of a bounding box of each object.

Requires training the object detection model, you must tag the classes and bounding box coordinates in a training set of images

This can be time consuming, but the Custom Vision portal makes this straightforward

The portal will suggest areas of the image where discrete objects are detected and you can add a class label

It also has Smart Tagging, where it suggests classes and bounding boxes to use for training

Precision - percentage of the class predictions made by the model that are correct

Recall - percentage of the class predictions the model identified correctly

Mean Average Precision (mAP) - Overall metric using precision and recall across all classes

Detect and analyze faces with the Face Service

Involves identifying regions of an image that contain a human face

It returns a bounding box that form a rectangle around the face

Moving beyond face detection, some algorithms return other information like facial landmarks (nose, eyes, eyebrows, lips, etc)

Facial landmarks can be used as features to train a model.

Another application of facial analysis. Used to train ML models to identify known individuals from their facial features.

More generally known as facial recognition

Requires multiple images of the person you want to recognize

Security - to build security applications and is used more and more no mobile devices

Social Media - use to automatically tag people and friends in photos.

Intelligent Monitoring - to monitor a persons face, for example when they are driving to determine where they are looking

Advertising - analyze faces in an image to direct advertisements to an appropriate demographic audience

Missing persons - use public camera systems with facial recognition to identify if a person is a missing person

Identity validation - use at port of entry kiosks to allow access/special entry permit

Blur - how blurry the face is

Exposure - aspects such as underexposed or over exposed and applies to the face in the image not overall image exposure

Glasses - if the person has glasses on

Head pose - face orientation in 3d space

Noise - visual noise in the image.

Occlusion - determines if any objects cover the face

Read text with the Computer Vision service

Submit an image to the API and get an operation ID

Use the operation ID to check status

When it’s completed get the result.

Pages - one for each page of text and orientation and page size

Lines - the lines of text on a page

Words - the words in a line of text including a bounding box and the text itself

Analyze receipts with the Form recognizer service

Matching field names to values

Processing tables of data

Identifying specific types of field, such as date, telephone number, addresses, totals, and other

Images must be JPEG, PNG, BMP, PDF, TIFF

File size < 50 MB

Image size between 50x50 pixels and 10000x10000 pixels

PDF documents no larger than 17 inches x 17 inches

You can train it with your own data

It just requires 5 samples to train it

Microsoft Azure AI Fundamentals: Explore decision support

Monitoring blood pressure

Evaluating mean tie between failures for hardware products

Part of the decision services category

Can be used with REST API

Sensitivity parameter is from 1 to 99

Anomalies are values outside expected values or ranges of values

The sensitivity boundary can be configured when making the API call

It uses a boundary, set as a sensitivity value, to create the upper and lower boundaries for anomaly detection

Calculated using concepts known as expectedValue, upperMargin, lowerMargin

If a value exceeds either boundary, then it is an anomaly

upperBoundary = expectedValue + (100-marginScale) * upperMargin

The service accepts data in JSON format.

It supports a maximum of 8640 data points. Break this down into smaller requests to improve the performance.

When to use Anomaly Detector

Process the algorithm against an entire set of data at one time

It creates a model based on your complete data set and the finds anomalies

Uses streaming data by comparing previously seen dat points to the last datapoint to determine if your latest one is an anomaly.

Model is created using the data points you send and determines if the current point is an anomaly.

Microsoft Azure AI Fundamentals: Explore natural language processing

Analyze Text with the Language Service

Used to describe solutions that involve extracting information from large volumes of unstructured data.

Analyzing text is a process to evaluate different aspects of a document or phrase, to gain insights about that text.

Text Analytics Techniques

Interpret words like “power”, “powered”, and “powerful” as the same word.

Convert to tree like structures (Noun phrases)

Often used for sentiment analysis

Determine the language of a document or text

Perform sentiment analysis (positive or negative)

Extract key phrases from text to indicate key talking points

Identify and categorize entities (places, people, organizations, etc)

Get started with Text analysis

Language name

ISO 6391 language code

Score as a level of confidence n the language returned.

Evaluates text to return a sentiment score and labels for each sentence

Useful for detecting positive or negative sentiment

Classification is between 0 to 1 with 1 being most positive

A score of 0.5 is indeterminate sentiment.

The phrase doesn’t have sufficient information to determine the sentiment.

Mixing language content with the language you tell it will return 0.5 also

Key Phrase extraction

Used to determine the main talking points of a text or a document

Depending on the volume this can take longer, so you can use the key phrase extraction capabilities of the Language Service to summarize main points.

Key phrase extraction can provide context about the document or text

Entity Recognition

Person

Location

OrganizationQuantity

DateTime

URL

Email

US-based phone number

IP address

Recognize and Synthesize Speech

Acoustic model - converts audio signal to phonemes (representation of specific sounds)

Language model - maps the phonemes to words using a statistical algorithm to predict the most probably sequence of words based on the phonemes

ability to generate spoken output

Usually converting text to speech

This process tokenizes the set to break it down into individual words, assign phonetic sounds to each word

It then breaks the phonetic transcription to prosodic units to create phonemes for the audio

Get started with speech on Azure

Use this for demos, presentations, or scenarios where a person is speaking

In real time it can translate to many lunges as it processes

Audio files with Shared access signature (SAS) URI can be used and results are received asynchronously.

Jobs will start executing within minutes, but no estimate is provided for when the job changes to running state

Used to convert text to speech

Voices can be selected that will vocalize the text

Custom voices can be developed

Voices are trained using neural networks to overcome limitations in speech synthesis with regards to intonation.

Translate Text and Speech

Where each word is translated to the corresponding word in the target language

This approach has issues. For example, a direct word to word translation may not exist or the literal translation may not be the correct meaning of the phrase

Machine learning has to also understand the semantic context of the translation.

This provides more accurate translation of the input phrase or phrases

Grammar, formal versus informal, colloquialism all need to be considered

Text and speech translation

Profanity filtering - remove or do not translate profamity

Selective translation - tag content that isn’t to be translated (brand names, code names, etc)

Speech to text - transcribe speech from an audio source to text format.

Text to speech - used to generate spoken audio from a text source

Speech translation - translate speech in one language to text or speech in another

Create a language model with Conversational language Understanding

A None intent exists.

This should be used when no intent has been identified and should provide a message to a user.

Getting started with Conversational Language Understanding

Authoring the model - Defining entities, intents, and utterances to use to train the model

Entity Prediction - using the model after it is published.

Define intents based on actions a user would want to perform

Each intent should include a variety of utterances as examples of how a user may express the intent

If the intent can be applied to multiple entities, include sample utterances for each potential entity.

Machine-Learned - learned by the model during training from context in the sample utterances you provide

List - Defined as a hierarchy of lists and sublists

RegEx - regular expression patterns

Pattern.any - entities used with patterns to define complex entities that may be hard to extract from sample utterances

After intents and entities are created you train the model.

Training is the process of using your sample utterances to teach the model to match natural language expressions that a user may say to probable intents and entities.

Training and testing are iterative processes

If the model does not match correctly, you create more utterances, retrain, and test.

When results are satisfactory, you can publish the model.

Client applications can use the model by using and endpoint for the prediction resource

Build a bot with the Language Service and Azure Bot Service

Knowledge base of question and answer pairs. Usually some built-in natural language processing model to enable questions and can understand the semantic meaning

Bot service - to provide an interface to the knowledge base through one or more channels

Microsoft Azure AI Fundamentals: Explore knowledge mining

Used to describe solutions that involve extracting information from large volumes of unstructured data.

It has a services in Cognitive services to create a user-managed index.

The index can b meant for internal use only or shared with the public.

It can use other Cognitive Services capabilities to extract the information

What is Azure Cognitive Search?

Provides a programmable search engine build on Apache Lucene

Highly available platform with 99.9% uptime SLA for cloud and on-premise assets

Data from any source - accepts data form any source provided in JSON format with auto crawling support for selected data sources in Azure

Full text search and analysis - Offers full text search capabilities supporting both simple query and full Lucene query syntax

AI Powered search - has Cognitive AI capabilities built in for image and text analysis from raw content

Multi-lingual - offers linguistic analysis for 56 langues

Geo-enabled - supports geo-search filtered based on proximity to a physical location

Configurable user experience - it includes capabilities to improve the user experience (autocomplete, autosuggest, pagination, hit highlighting, etc)

Identify elements of a search solution

Folders with files,

Text in a database

Etc

Use a skillset to Define an enrichment pipeline

Key Phrase Extraction - uses a pre-trained model to detect important phrases based on term placement, linguistic rules, proximity to terms

Text Translation - pre-trained model to translate the input text into various languages for normalization or localization use cases

Image Analysis Skills - uses an image detection algorithm to identify the content of an image an generate a text description

Optical Character Recognition Skills - extract printed or handwritten text from images, photos, videos

Understand indexes

Index schema - index includes a definition of the structure of the data in the documents to read.

Index attributes - Each field in a document the index stores its name, the data type, supported behaviors (searchable, sortable, etc)

Best indexes use only the features that are required/needed

Use an indexer to build an index

Push method - JSON data is pushed into a search index via a REST API or a .NET SDK. Most flexible and with least restrictions

Pull method - Search service indexer pulls from popular Azure data sources and if necessary exports the Tinto JSON if its not already in that format

Use the pull method to load data with an indexer

Azure Cognitive search’s indexer is a crawler that extracts searchable text and metadata form an external Azure data source an populates a search index using field-to-field mapping between the data and the index.

Data import monitoring and verification

Indexers only import new or updated documents. It is normal to see zero documents indexed

Health information is displayed in a dashboard.

You can monitor the progress of the indexing

Making changes to an index

You need to drop and recreate indexes if you need to make changes to the field definitions

An approach to update your index without impacting your users is to create a new index with a new name

After importing data, switch to the new index.

Persist enriched data in a knowledge store

A knowledge store is persistent storage of enriched content.

Azure Machine Learning Studio

You can use the workspace to develop solutions with the Azure ML service on the web portal or with developer tools

Web portal for ML solutions in Sure

Capabilities for preparing data, training models, publishing and monitoring a service.

First step assign a workspace to a studio.

Compute targets are cloud-based resources which can run model training and data exploration processes

Compute Instances - Development workstations that data scientists can use to work with data and models

Compute Clusters - Scalable clusters of VMs for on demand processing of experiment code

Inference Clusters - Deployment targets for predictive services that use your trained models

Attached Compute - Links to existing Azure compute resources like VMs or Azure data brick clusters

What is Azure Automated Machine Learning

Jobs have multiple settings

Provide information needed to specify your training scripts, compute target and Azure ML environment and run a training job

Understand the AutoML Process

ML model must be trained with existing data

Data scientists spend lots of time pre-processing and selecting data

This is time consuming and often makes inefficient use of expensive compute hardware

In Azure ML data for model training and other operations are encapsulated in a data set.

You create your own dataset.

Classification (predicting categories or classes)

Regression (predicting numeric values)

Time series forecasting (predicting numeric values at a future point in time)

After part of the data is used to train a model, then the rest of the data is used to iteratively test or cross validate the model

The metric is calculated by comparing the actual known label or value with the predicted one

Difference between the actual known and predicted is known as residuals; they indicate amount of error in the model.

Root Mean Squared Error (RMSE) is a performance metric. The smaller the value, the more accurate the model’s prediction is

Normalized root mean squared error (NRMSE) standardizes the metric to be used between models which have different scales.

Shows the frequency of residual value ranges.

Residuals represents variance between predicted and true values that can’t be explained by the model, errors

Most frequently occurring residual values (errors) should be clustered around zero.

You want small errors with fewer errors at the extreme ends of the sale

Should show a diagonal trend where the predicted value correlates closely with the true value

Dotted line shows a perfect model’s performance

The closer to the line of your model’s average predicted value to the dotted, the better.

Services can be deployed as an Azure Container Instance (ACI) or to a Azure Kubernetes Service (AKS) cluster

For production AKS is recommended.

Identify regression machine learning scenarios

Regression is a form of ML

Understands the relationships between variables to predict a desired outcome

Predicts a numeric label or outcome base on variables (features)

Regression is an example of supervised ML

What is Azure Machine Learning designer

Allow you to organize, manage, and reuse complex ML workflows across projects and users

Pipelines start with the dataset you want to use to train the model

Each time you run a pipelines, the context(history) is stored as a pipeline job

Encapsulates one step in a machine learning pipeline.

Like a function in programming

In a pipeline project, you access data assets and components from the Asset Library tab

You can create data assets on the data tab from local files, web files, open at a sets, and a datastore

Data assets appear in the Asset Library

Azure ML job executes a task against a specified compute  target.

Jobs allow systematic tracking of your ML experiments and workflows.

Understand steps for regression

To train a regression model, your data set needs to include historic features and known label values.

Use the designer’s Score Model component to generate the predicted class label value

Connect all the components that will run in the experiment

Average difference between predicted and true values

It is based on the same unit as the label

The lower the value is the better the model is predicting

The square root of the mean squared difference between predicted and true values

Metric based on the same unit as the label.

A larger difference indicates greater variance in the individual  label errors

Relative metric between 0 and 1 on the square based on the square of the differences between predicted and true values

Closer to 0 means the better the model is performing.

Since the value is relative, it can compare different models with different label units

Relative metric between 0 and 1 on the square based on the absolute of the differences between predicted and true values

Closer to 0 means the better the model is performing.

Can be used to compare models where the labels are in different units

Also known as R-squared

Summarizes how much variance exists between predicted and true values

Closer to 1 means the model is performing better

Remove training components form your data and replace it with a web service inputs and outputs to handle the web requests

It does the same data transformations as the first pipeline for new data

It then uses trained model to infer/predict label values based on the features.

Create a classification model with Azure ML designer

Classification is a form of ML used to predict which category an item belongs to

Like regression this is a supervised ML technique.

Understand steps for classification

True Positive - Model predicts the label and the label is correct

False Positive - Model predicts wrong label and the data has the label

False Negative - Model predicts the wrong label, and the data does have the label

True Negative - Model predicts the label correctly and the data has the label

For multi-class classification, same approach is used. A model with 3 possible results would have a 3x3 matrix.

Diagonal lien of cells were the predicted and actual labels match

Number of cases classified as positive that are actually positive

True positives divided by (true positives + false positives)

Fraction of positive cases correctly identified

Number of true positives divided by (true positives + false negatives)

Overall metric that essentially combines precision and recall

Classification models predict probability for each possible class

For binary classification models, the probability is between 0 and 1

Setting the threshold can define when a value is interpreted as 0 or 1.  If its set to 0.5 then 0.5-1.0 is 1 and 0.0-0.4 is 0

Recall also known as True Positive Rate

Has a corresponding False Positive Rate

Plotting these two metrics on a graph for all values between 0 and 1 provides information.

Receiver Operating Characteristic (ROC) is the curve.

In a perfect model, this curve would be high to the top left

Area under the curve (AUC).

Remove training components form your data and replace it with a web service inputs and outputs to handle the web requests

It does the same data transformations as the first pipeline for new data

It then uses trained model to infer/predict label values based on the features.

Create a Clustering model with Azure ML designer

Clustering is used to group similar objects together based on features.

Clustering is an example of unsupervised learning, you train a model to just separate items based on their features.

Understanding steps for clustering

Prebuilt components exist that allow you to clean the data, normalize it, join tables and more

Requires a dataset that includes multiple observations of the items you want to cluster

Requires numeric features that can be used to determine similarities between individual cases

Initializing K coordinates as randomly selected points called centroids in an n-dimensional space (n is the number of dimensions in the feature vectors)

Plotting feature vectors as points in the same space and assigns a value how close they are to the closes centroid

Moving the centroids to the middle points allocated to it (mean distance)

Reassigning to the closes centroids after the move

Repeating the last two steps until tone.

Maximum distances between each point and the centroid of that point’s cluster.

If the value is high it can mean that cluster is widely dispersed.

With the Average Distance to Closer Center, we can determine how spread out the cluster is

Remove training components form your data and replace it with a web service inputs and outputs to handle the web requests

It does the same data transformations as the first pipeline for new data