#voicecommands

Gemini Live API

Create live voice-driven agentic apps using Vertex AI Gemini Live API. All industries seek aggressive, effective solutions. Imagine frontline personnel using voice and visual instructions to diagnose issues, retrieve essential information, and initiate processes in real time. A new agentic industrial app may be created with the Gemini 2.0 Flash Live API.

This API extends these capabilities to complex industrial processes. Instead of using one data type, it uses text, audio, and visual in a continuous livestream. This allows intelligent assistants to understand and meet the demands of manufacturing, healthcare, energy, and logistics experts.

The Gemini 2.0 Flash Live API was used for industrial condition monitoring, notably motor maintenance. Live API allows low-latency phone and video communication with Gemini. This API lets users have natural, human-like audio chats and halt the model's answers with voice commands. The model processes text, audio, and video input and outputs text and audio. This application shows how APIs outperform traditional AI and may be used for strategic alliances.

Multimodal intelligence condition monitoring use case

Presentation uses Gemini 2.0 Flash Live API-powered live, bi-directional, multimodal streaming backend. It can interpret audio and visual input in real time for complex reasoning and lifelike speech. Google Cloud services and the API's agentic and function calling capabilities enable powerful live multimodal systems with a simplified, mobile-optimized user experience for factory floor operators. An obviously flawed motor anchors the presentation.

A condensed smartphone flow:

Gemini points the camera at motors for real-time visual identification. It then quickly summaries relevant handbook material, providing users with equipment details.

Real-time visual defect detection: Gemini listens to a verbal command like “Inspect this motor for visual defects,” analyses live video, finds the issue, and explains its source.

When it finds an issue, the system immediately prepares and sends an email with the highlighted defect image and part details to start the repair process.

Real-time audio defect identification: Gemini uses pre-recorded audio of healthy and faulty motors to reliably identify the issue one based on its sound profile and explain its results.

Multimodal QA on operations: Operators can ask complex motor questions by pointing the camera at certain sections. Gemini effectively combines motor manual with visual context for accurate voice-based replies.

The tech architecture

The demonstration uses Google Cloud Vertex AI's Gemini Multimodal Livestreaming API. The API controls workflow and agentic function calls while the normal Gemini API extracts visual and auditory features.

A procedure includes:

Function calling by agents: The API decodes audio and visual input to determine intent.

The system gathers motor sounds with the user's consent, saves them in GCS, and then begins a function that employs a prompt with examples of healthy and faulty noises. The Gemini Flash 2.0 API examines sounds to assess motor health.

The Gemini Flash 2.0 API's geographical knowledge is used to detect and highlight errors by recognising the intent to detect visual defects, taking photographs, and invoking a method that performs zero-shot detection with a text prompt.

Multimodal QA: The API recognises the objective of information retrieval when users ask questions, applies RAG to the motor manual, incorporates multimodal context, and uses the Gemini API to provide exact replies.

After recognising the intention to repair and extracting the component number and defect image using a template, the API sends a repair order via email.

Key characteristics and commercial benefits from cross-sector usage cases

This presentation highlights the Gemini Multimodal Livestreaming API's core capabilities and revolutionary industrial benefits:

Real-time multimodal processing: The API can evaluate live audio and video feeds simultaneously, providing rapid insights in dynamic circumstances and preventing downtime.

Use case: A remote medical assistant might instruct a field paramedic utilising live voice and video to provide emergency medical aid by monitoring vital signs and visual data.

Gemini's superior visual and auditory reasoning deciphers minute aural hints and complex visual settings to provide exact diagnoses.

Utilising equipment noises and visuals, AI can predict failures and eliminate manufacturing disruptions.

Agentic function invoking workflow automation: Intelligent assistants can start reports and procedures proactively due to the API's agentic character, simplifying workflows.

Use case: A voice command and visual confirmation of damaged goods can start an automated claim procedure and notify the required parties in logistics.

Scalability and seamless integration: Vertex AI-based API interfaces with other Google Cloud services ensure scalability and reliability for large deployments.

Use case: Drones with cameras and microphones may send real-time data to the API for bug identification and crop health analysis across huge farms.

The mobile-first design ensures that frontline staff may utilise their familiar devices to interact with the AI assistant as needed.

Store personnel may use speech and image recognition to find items, check stocks, and get product information for consumers on the store floor.

Real-time condition monitoring helps industries switch from reactive to predictive maintenance. This will reduce downtime, maximise asset use, and improve sectoral efficiency.

Use case: Energy industry field technicians may use the API to diagnose faults with remote equipment like wind turbines without costly and time-consuming site visits by leveraging live audio and video feeds.

Start now

Copilot remembers the specifics that are most useful in every circumstance and recognizes the context of your life while protecting your privacy, data, and security. It provides you with access to an infinite reservoir of knowledge, streamlining and organizing the daily onslaught of data while providing encouragement and assistance when needed.

Furthermore, it is integrating cutting-edge features like Copilot voice and vision to enhance its use and naturalness. These modalities transform its relationship with technology, allowing users to collaborate, reflect, and learn with their Copilot companions in a seamless way.

The next stage of Copilot

Microsoft is starting to release the updated Copilot today, which has the following new and improved features:

Copilot Voice: With Copilot Voice, connecting with your buddy is now simpler than ever. This is the most logical and effortless method for quick question-asking, brainstorming on-the-go, or even just venting after a difficult day. You will be able to customize your partner with four different voice options.

Copilot Daily: With additional choices including notifications of upcoming events, Copilot Daily helps you start your morning with a summary of the news and weather, all read in your preferred Copilot Voice. It acts as a counterbalance to that well-known sensation of information overload. straightforward, clear, and straightforward to process. Copilot Daily will only extract information from approved sources. It intend to gradually add more sources to the list of partners with which it is collaborating, including Reuters, Axel Springer, Hearst Magazines, USA TODAY Network, and Financial Times. Over time, it plan to incorporate more customization options and controls into Copilot Daily.

Copilot Discover: Uncertain about where to begin? Copilot Discover makes it easier than ever to get started by providing a helpful overview of its capabilities as well as discussion ideas. These starting points are tailored based on your experiences with other Microsoft services and will be further tuned over time depending on your discussion history, all with your consent.

Microsoft Edge: Copilot is a feature that comes pre-installed on your Microsoft Edge browser. It may be used to swiftly translate text, rephrase sentences, summarize page information, and answer questions. By just entering @copilot into the address box, it is now even simpler to access Copilot straight from the Microsoft Edge browser.

Copilot Labs: It allow users to test out its experimental features, which are still under development. It’s an opportunity to provide input and influence the experiences it design. Initially, it will be introducing Copilot Vision and Think Deeper to Labs.

Copilot Vision: It is a completely novel approach to computer interaction. Copilot Vision can see what you see and has real-time communication capabilities. It can answer inquiries about the content of the webpage you’re watching, recommend next steps, and assist you without interfering with your job. It comprehends both the text and the visuals on the website. If you’re trying to decorate a new apartment, Copilot Vision can assist you with furniture searches, color palette selection, consideration of alternatives for anything from throws to carpets, and even organizing ideas for the items you’re looking at.

Here, security and safety are first:

Copilot Vision sessions are transient and fully opt-in. As soon as your session ends, all of the stuff Copilot Vision interacts with is permanently deleted and is not saved or used for training.

Since it is made significant measures to restrict the kinds of websites that Copilot Vision may interact with, the experience won’t function on every website. Starting with a small selection of well-known websites, hope to make sure that everyone has a safe and secure experience.

For this preview, Copilot Vision will not function on sensitive or paywalled content. It designed it with the interests of users and artists in mind.

Neither AI training nor any particular processing of the content of a website you are reading occurs. All Copilot Vision does is read and understand what it sees on the page for the first time, right alongside you.

Additionally, it is ensuring sure Copilot is accessible on each of its platforms. It is making Copilot easily accessible on Copilot+ PCs and Windows, opening up new ways to engage with your computer with just a click.

Think Deeper: Copilot is now capable of handling trickier inquiries. Think Deeper takes longer to react, which enables Copilot to provide comprehensive, step-by-step responses to difficult inquiries. It is intended to be useful for a wide range of real-world and practical problems, such as side-by-side comparisons of two complicated solutions. Since it’s still in its early stages of development, it has tested and gathered feedback on it in experimental Copilot Labs.

Today, the updated Copilot is available for Windows, iOS, and Android users via the Copilot website at copilot.microsoft.com. Microsoft also thrilled to begin introducing Copilot to WhatsApp, which will enable users to have more organic and interesting interactions with Copilot.

Observations:

Initially, Australia, Canada, New Zealand, the United Kingdom, and the United States will have access to Copilot Voice in English. Soon, Copilot Voice will be available in more languages and areas.

Beginning today, Copilot Daily will be available in the US and the UK, with additional nations to follow shortly.

Users of Copilot can always choose not to have personalization, which is compliant with Microsoft Privacy Statement, by going to Settings. It is currently finalizing its options for providing users in the UK and the EU with customization.

A restricted number of Copilot Pro subscribers in the US will be able to use Copilot Vision when it shortly makes its way to Copilot Labs.

This week, a select group of Copilot Pro users in Australia, Canada, New Zealand, the United Kingdom, and the United States will be able to access Think Deeper with Copilot Labs.