#geminimodels

Google Magic Mirror

The new “Google Magic Mirror” showcases the JavaScript GenAI SDK and Gemini API's interactivity. A mirror, a common object, becomes a discussion interface in this idea.

The Google Magic Mirror optimises real-time communication. Interactivity relies on the Live API, which allows real-time voice interactions. The mirror participates in a genuine back-and-forth discussion in text or voice by digesting speech as you speak, unlike systems that merely listen for one order.

The Live API powers bidirectional, real-time audio streaming and communication. One of Live API's most dynamic features is speech detection during playback. This interruption can dynamically change the tale and dialogue, enabling text and aural dialogue, depending on the user's actions.

Google Magic Mirror can be a “enchanted storyteller” and a communication tool. This feature uses the Gemini model's advanced generating capabilities. The storytelling component can be customised by delivering system commands that affect the AI's tone and conversational style. By modifying speech configurations during initialisation, the AI can respond with different voices, accents, dialects, and other traits. AI language and voice are changed by speech setting.

The project combines the model's real-world connection for contemporary information seekers. The Google Magic Mirror may use Grounding with Google Search to provide real-time, grounded news. This ensures that the mirror's responses are not limited to its training material. Starting with Google Search ensures current, reliable information.

Image generation by the mirror adds a touch of “visual alchemy” to the experience. Function Calling from the Gemini API lets the mirror create images from user descriptions. This strengthens the engagement and deepens storytelling. The Gemini model determines whether a user request creates an image and triggers a function based on provided features.

The picture production service receives a detailed prompt from the user's spoken words. Function Calling is a more extensive feature that allows Gemini models to speak with publically accessible external tools and services, such as picture generation or bespoke actions, based on the discussion.

The user experience hides the technology intricacies, while strong Gemini model aspects provide this “magical experience” in the background. Among these technical traits:

Live API controls bidirectional, real-time audio streaming and communication.

Gemini models can call functions from external tools and services like picture production or bespoke actions based on the discussion.

Using Google Search for current, accurate information.

System directives shape AI tone and conversation.

Speech configuration changes AI responses' tone and vocabulary.

Modality control lets the Gemini API anticipate output modalities or respond in text or voice.

The inventors say their Gemini-enabled Google Magic Mirror is more than a gimmick. It shows how advanced AI may be blended into real life to create helpful, fascinating, and even miraculous interactions. Flexibility allows the Gemini API to enable many more applications. Immersive entertainment platforms, dynamic educational tools, and personalised assistants are possible applications.

The Google Magic Mirror's code is available on GitHub for those interested in its technical operation. Hackster.io also provides a detailed build tutorial. The founders of X and LinkedIn want the community to imagine what their Google magic mirror could do and contribute ideas and other Gemini-enabled products.

Supervised Fine Tuning

Supervised Fine-Tuning (SFT) enables robust models to be customized for specific tasks, domains, and even subtle stylistic differences. Questions about when to utilize SFT and how it stacks up versus alternatives like RAG, in-context learning, and prompt engineering are common among developers.

This article explores the definition of SFT, when to use it, and how it stacks up against other techniques for output optimization.

What is SFT?

Large language model (LLM) development frequently starts with pre-training. The model gains general language comprehension at this stage by reading vast volumes of unlabeled material. Pre-training’s main goal is to give the model a wide range of language understanding abilities. These previously trained LLMs have performed remarkably well on a variety of activities. The performance of this pre-trained model can then be improved for downstream use cases, such summarizing financial papers, that can call for a more in-depth understanding.

Using a task-specific annotated dataset, the pre-trained model is refined to make it suitable for certain use cases. This dataset contains examples of intended outputs (like a summary) correlating to input instances (like an earnings report). The model gains knowledge of how to carry out a particular task by linking inputs with their appropriate outputs. It refer to fine-tuning using an annotated dataset as supervised fine-tuning (SFT).

The main tool for modifying the behavior of the model is its parameters, which are the numerical values that are acquired during training. There are two popular supervised fine-tuning methods, albeit the number of model parameters it update during fine-tuning can vary:

Full fine-tuning: modifies every parameter in the model. However, comprehensive fine-tuning results in greater total costs because it requires more computer resources for both tuning and serving.

Parameter-Efficient Fine-Tuning (PEFT): In order to facilitate quicker and more resource-efficient fine-tuning, a class of techniques known as Parameter-Efficient Fine-Tuning (PEFT) freezes the initial model and only modifies a small number of newly introduced extra parameters. When dealing with huge models or limited computational resources, PEFT is especially helpful.

Although both PEFT and full fine-tuning are supervised learning techniques, they differ in how much parameter updating they carry out, thus one might be better suited for your situation. An example of a PEFT technique is LoRA (Low-Rank Adaptation), which is used in Supervised fine-tuning for Gemini models on Vertex AI.

When should one use supervised fine-tuning?

If you have access to a dataset of well-annotated samples and your objective is to improve the model’s performance on a particular, well-defined task, you should think about utilizing Supervised Fine Tuning. When the task is in line with the initial pre-training data, supervised fine-tuning is very useful for effectively activating and honing the pertinent knowledge that is already contained within the pre-trained model. Here are some situations in which Supervised Fine Tuning excels:

Domain expertise: Give your model specific knowledge to make it an authority on a certain topic, such as finance, law, or medical.

Customize the format: Make your model’s output conform to particular structures or formats.

Task-specific proficiency: Fine-tune the model for certain tasks, including brief summaries.

Edge cases: Boost the model’s capacity to manage particular edge cases or unusual situations.

Behavior Control: Direct the actions of the model, including when to give succinct or thorough answers.

One of SFT’s advantages is that it can produce gains even with a small quantity of excellent training data, which often makes it a more affordable option than complete fine-tuning. Moreover, refined models are typically more user-friendly. Supervised Fine Tuning helps the model become proficient at the job, which minimizes the need for long and intricate cues during inference. This results in decreased expenses and delayed inference.

SFT is an excellent tool for consolidating prior information, but it is not a panacea. In situations where information is dynamic or ever-changing, like when real-time data is involved, it might not be the best option. Let’s go over these additional possibilities as they may be more appropriate in some situations.

LLM Supervised Fine Tuning

Supervised Fine Tuning isn’t necessarily the only or best option for adjusting an LLM’s output, despite its strength. Effective methods for changing the behavior of the model can be found in a number of other approaches, each with advantages and disadvantages.

Prompt engineering is affordable, accessible, and simple to use for controlling outputs. For managing intricate or subtle jobs, it could be less dependable and necessitates experience and trial.

Like prompt engineering, In-Context Learning (ICL) is simple to use and makes use of examples found within the prompt to direct the behavior of the LLM. ICL, sometimes known as few-shot prompting, can be affected by the prompt’s examples and the sequence in which they are given. It might also not generalize well.

In order to increase quality and accuracy, Retrieval Augmented Generation (RAG) gathers pertinent data from Google search and other sources and gives it to the LLM. A strong knowledge base is necessary for this, and the extra step increases complexity and delay.

The capacity of a language model to recognize when external systems are required to respond to a user request and provide structured function calls in order to communicate with these tools and increase their capabilities is known as function calling. It might increase complexity and delay when employed.

Where to start?

You may be asking yourself, “What’s the right path now?” It’s critical to realize that the best course of action is determined by your particular requirements, available resources, and use case goals. These methods can be combined and are not exclusive to one another. Let’s examine a framework that can direct your choice:Image Credit to Google Cloud

If you want to be sure the model can understand the subtleties of your particular domain, you can begin by investigating prompt engineering and few-shot in-context learning. Here, Gemini’s huge context box opens up a world of options. You can experiment using Retrieval Augmented Generation (RAG) and/or Supervised Fine-Tuning (SFT) after you’ve perfected your prompting technique for even more refining. A lot of the most recent methods are shown in this graphic, although generative AI is a rapidly evolving subject.

Supervised Fine-Tuning on Vertex AI with Gemini

When you have a specific objective in mind and have labeled data to assist the model, Supervised Fine-Tuning (SFT) is the best option. Supervised Fine Tuning can be combined with other methods you may already be attempting to create more effective models, which could reduce expenses and speed up response times.

BigQuery vector search

Google Cloud announce today that BigQuery vector search is now generally available (GA), allowing users to search for vector similarity on BigQuery data. This feature, which is also known as approximate nearest-neighbor search, is essential for enabling a wide range of new data and AI use cases, including retrieval-augmented generation (RAG) using a large language model (LLM), similarity detection, and semantic search.

BigQuery vector search was first revealed in February and offers a serverless and integrated vector-analytics solution for use cases like anomaly detection, multi-modal search, product recommendations, drug discovery, and more. It does this by integrating the creation, management, and search of embeddings within the data platform.

- Advertisement -

Furthermore, the BigQuery vector search inverted file index (IVF) index is now publicly accessible. With the use of an inverted row locator and a k-means clustering method, this index creates a two-piece index that effectively searches for similar embedding representations of your data. Since being revealed in preview, IVF has undergone the following new improvements:

Increased scalability: 10 billion embeddings can now be indexed, opening the door to large-scale applications.

Managed index with guaranteed correctness: Vector indexes are automatically updated using the current k-means model if the underlying data changes. Even before the system has completed re-indexing the updated data, vector search consistently yields accurate results based on the most recent mutations of the data.

Stored columns: To save money on costly joins when obtaining more data for the search result, you may now save frequently used columns in the index. The situations where this optimization produces the most visible performance gains are those with high result-set cardinality, such as when you require a high top_k or when your query data comprises a large batch of embeddings. For example, using vector indexes with stored columns results in delivering the 1,000 most comparable candidates for an embedding ~4x faster with ~200x less slots than using vector indexes without stored columns for a table containing 1 billion 96-dimensional embeddings.

Pre-filters: By changing the base table statement into a query with filters, vector search results can be pre-filtered in conjunction with stored columns. Pre-filtering increases search quality, reduces the chance of missing results, and optimizes query performance in contrast to post-filtering, which adds WHERE clauses after the VECTOR_SEARCH() function.

Palo Alto Networks and other clients have used BigQuery vector search to uncover comparable, frequently asked queries, which has sped up the time to insight.

Furthermore, even for large-scale workloads like drug discovery, which Vilya has been working on, prototyping with BigQuery vector search and moving to production is straightforward. Moreover, the transition to capacity-based billing models has been easy with tools for budget evaluation and on-demand pricing.

Using an example to build

Let’s say you want to ask a question in an internal Q&A forum, but you want to see whether there are any already answered queries that are semantically related to yours. It will assume that it has created embeddings for the questions and put them in the database in order to illustrate. After that is finished, you may make a vector index and, for optimal efficiency, store the frequently used columns such as title, content, and tags in the index.

CREATE OR REPLACE VECTOR INDEX <index_name> ON <my_posts_questions>(embedding) STORING (title, content, tags) OPTIONS(distance_type=’COSINE’, index_type=’IVF’)

Even without a vector index, VECTOR_SEARCH() functions, however establishing an index usually improves query performance. When everything is ready, you can use ML.GENERATE_EMBEDDING in conjunction with VECTOR_SEARCH() to look up topics like “Android app using RSS crashing,” among others. You can use a pre-filter on the “tags” column to narrow the search space in order to more effectively refine the results.

SELECT query.query, base.title, base.content, distance FROM VECTOR_SEARCH( ( SELECT * FROM WHERE SEARCH(tags, ‘android’) ), ’embedding’, ( SELECT ml_generate_embedding_result AS embedding, content AS query FROM ML.GENERATE_EMBEDDING( MODEL , (SELECT ‘Android app using RSS crashing’ AS content) ) ))

Additionally, it just revealed a new index type that can even further enhance search performance. It is based on the ScaNN developed by Google and is available in preview. BigQuery vector search is becoming an essential part of a multi-modal Retrieval Augmentation Generation (RAG) solution, built on top of a fully functional BigQuery knowledge base that includes multimodal, structured, and unstructured data, and powered by cutting-edge Gemini models.

Start now

Beginning with fast and affordable searching on those embeddings, vector embeddings and machine learning have the potential to completely transform what you can do with the data kept in your BigQuery business data warehouses.

The first open models in the world, DataGemma establish LLMs in the massive, real-world statistical datasets of Google’s Data Commons, thereby helping to overcome the problems associated with hallucinations.

The large language models (LLMs) that drive the advances in AI today are getting more and more complex. These models are capable of sifting through enormous volumes of text to produce summaries, new creative directions, and even code drafts. Despite how amazing these talents are, LLMs occasionally provide false information with confidence. One of the main issues with generative AI is this behavior, called “hallucination.”

Today, Google is presenting exciting new study findings that directly address this problem by grounding LLMs in empirical statistical data, which helps lessen hallucinations. In addition to these academic advances, it is pleased to present DataGemma, the first publicly available models created to link LLMs with large amounts of real-world data taken from Google’s Data Commons.

Google Data Commons

Data Commons: An extensive collection of reliable data that is accessible to the public

Across hundreds of thousands of statistical variables, Data Commons is a publicly accessible knowledge network with over 240 billion rich data points. It obtains this public data from reliable sources such as the World Health Organization (WHO), the United Nations (UN), the Centers for Disease Control and Prevention (CDC), and Census Bureaus. Bringing these datasets together into a single, cohesive collection of tools and AI models allows researchers, policymakers, and organizations looking for precise insights.

Consider Data Commons to be an enormous, ever-growing library of trustworthy, publicly available data on a variety of subjects, from economics and health to demographics and the environment, that you can engage with using Google’s artificial intelligence (AI)-powered natural language interface. For instance, you can investigate which African nations have seen the biggest increases in access to electricity, how income and diabetes are correlated in US states, or you can ask your own data-driven question.

How to combat hallucinations with Data Commons

Google’s goal is to anchor the growing use of generative AI by incorporating Data Commons into Gemma, its family of cutting-edge, lightweight open models that are constructed using the same technology and research as the Gemini models. Researchers and developers can begin using these DataGemma models immediately.

By utilizing the expertise of Data Commons to improve LLM factuality and reasoning through two unique methods, DataGemma will increase the capabilities of Gemma models:

By proactively searching reliable sources and fact-checking against data in Data Commons, RIG (Retrieval-Interleaved Generation) improves the capabilities of its language model, Gemma 2. When DataGemma receives a prompt to provide an answer, the model is configured to find examples of statistical data and obtain the response from Data Commons. Although the RIG methodology is not new, the way it is used within the DataGemma framework is distinct.

Language models can now incorporate relevant information beyond their training data, take in additional context, and provide more thorough and informative outputs thanks to RAG (Retrieval-Augmented Generation). This was made possible with DataGemma by utilizing the extended context window of Gemini 1.5 Pro. In order to reduce the possibility of hallucinations and improve answer accuracy, DataGemma gets pertinent contextual information from Data Commons prior to the model starting to generate responses.

Future directions and promising outcomes

Although early, Google’s initial RIG and RAG results are promising. Notable improvements have been shown in its language models’ accuracy when processing numerical information. According to this, users may have fewer hallucinations when using it for research, making decisions, or just sating their curiosity.

Its study is still ongoing, and they are dedicated to improving these approaches even more as they ramp up this work, put it through a rigorous testing schedule, and eventually incorporate this improved capability into the Gemma and Gemini models, first in stages with restricted access.

We hope that Google can encourage a wider adoption of these Data Commons-led methods for firmly establishing LLMs in factual data by disseminating its findings and reopening this most recent Gemma model version as an “open” model. Building a future where AI provides humans with accurate information, encourages educated decision-making, and fosters a deeper awareness of the world around us will require making LLMs more dependable and trustworthy.

These quickstart notebooks for both the RIG and RAG techniques can be used by researchers and developers to get started with DataGemma.

Google Ask Photos

One of the first products Google developed with AI at its core was Google Photos, which lets you search through your images and videos for people, animals, locations, and more. Now that Gemini, Google’s most powerful AI model, is available, Google is significantly improving Google Photos. Google’s newest experimental feature, Ask Photos, will be available over the next few months and will make it even simpler for you to search for a particular memory or retrieve information from your gallery.

Locate the appropriate stuff more easily

Every day, more than 6 billion images are added to Google Photos. Even with the correct set of keywords, searching through pages of images and videos can get tedious as people’s galleries fill up. You can ask natural questions using Ask Photos, such as “Show me the best photo from each national park I’ve visited.” Google Photos can save you time and scrolling by displaying the information you need.

Pose inquiries and receive beneficial responses

Google take pictures all the time to record memories or save significant information, but the intricacies in those pictures sometimes get lost in the mix. When you ask questions about your life, such as where you camped last year or when your vouchers expire, Ask Photos finds that information for you. Gemini’s multimodal skills enable it to extract details from photographs by comprehending the subject and context. “What themes have we had for Lena’s birthday parties?” someone could ask. for instance. Ask Photos can provide you with the answer by understanding specifics, such as the decorations on the birthday cake or in the backdrop.

Obtain further task assistance beyond search

With the aid of Gemini models, completing activities in Google Photos is also simpler. For instance, selecting the best photos you took to send to friends and family after a long journey can be intimidating. You can more readily make a vacation highlight with the aid of Ask Photos. Just ask, and it will recommend the best photos for you to publish on social media along with a customized caption.

The inner workings of Ask Photos

Having interpreted your query: After comprehending your question, Ask Photos devises a strategy to locate the solution. In addition to finding pertinent keywords like locations, individuals, and dates, it also conducts an advanced search on your behalf and returns natural language notions like “themed birthday party.”

Formulating the answer: Examining the search results to determine which are most pertinent and appear to be what you’re looking for is the next step. With the use of Gemini’s multimodal skills, one can accurately decipher every picture and, if necessary, even comprehend text contained inside them. After that, Ask Photos selects whatever pictures and videos to return and writes a helpful reply.

Ensuring safety and remembering corrections: Ask Photos uses layers of protection and AI models to help make sure responses are acceptable and safe, even if it’s still experimental and may not get everything perfect. Ask Photos can also retain details for future use if you update an answer or add more material.

How Google safeguard your personal information

Google take seriously their obligation to secure the sensitive information contained in your images, which may contain very personal information. Google Photos never uses your personal information for advertising. Additionally, other users won’t be able to access your Ask Photos chats or personal information unless it’s necessary to report abuse or other damage. Additionally, Google don’t use this personal data to train any other generative AI product such as other Gemini models or products apart from Google Photos. As usual, Google’s industry-leading security features safeguard all of your data on Google Photos.

Ask Photos Google

Nearly half a billion people use Google Photos’ search tool each month to find images of people, pets, locations, and other subjects in your photo galleries. You can now search for photos and videos in Photos using more detailed queries, instead of just basic keywords, to obtain more relevant results. Additionally, Google is giving a limited number of American users early access to Ask Photos, their conversational search engine powered by Gemini.

An enhanced method for finding photos and videos

When photo libraries get bigger and bigger, it can often take more detailed searches to get what you’re looking for. You can now use common language to find what you’re seeking for as of right now. You can, for instance, use natural language to search for certain images, such as “Alice and me laughing,” “Kayaking on a lake surrounded by mountains,” or “Emma painting in the backyard.”

You can select to arrange your search results by relevance or date, which will help you discover the exact image you’re looking for even faster in addition to receiving better results. All users of Android and iOS are now able to enjoy the enhanced search experience in English, and over the next few weeks, more languages will be added.

Ask Photos early access for a limited number of users

As part of Google Labs, early access to Ask Photos is now available to a limited number of US users. This experimental experience surpasses Google’s enhanced search functionality. Ask Photos uses the most recent Gemini models to comprehend the context of your photo collection, including the people who matter most to you, your interests, and your favorite meals. It then extracts pertinent details to assist you in locating particular memories and learning more about your life.

For instance, you may inquire, “Where did we camp last time we went to Yosemite?” when making plans for your upcoming camping vacation and hoping to revisit a favorite location. Alternatively, you may just inquire, “What did we eat at the hotel in Stanley?” if you wanted to return to a restaurant you liked and place the same order. Ask Photos can provide you with the answer by understanding specifics, such as the location of the restaurant where you took pictures with your camping gear or the type of food that is on the table in your image. Additionally, because Ask Photos is conversational, you may give it more hints or facts to guide it in the right path if it doesn’t come up with the answer straight away.

Thanks to Gemini models, Ask Photos can also make chores easier for you to do. You could ask for help curating an album of your greatest birthday party images or compiling a list of the must-do activities from your most recent vacation to share with a buddy.

Google’s meticulous construction of Ask Photos

With Ask Photos, Google is introducing it responsibly, adhering to their AI Principles. Google take extremely seriously their responsibility to protect your data and privacy, even as they work hard to continuously improve their products based on real-world usage. As usual, Google Photos protects your data with industry-leading security safeguards and never uses it for advertisements. In order to enhance Ask Photos, user queries might be examined by humans; however, this will only happen once your Google Account has been disconnected to safeguard your privacy. Unless you provide feedback or in rare instances to address abuse or harm, Ask photographs’ answers including your photographs and videos are not vetted by humans.

Tabs Compare

Chrome for desktop users now has enhanced history search, Tabs compare, and Google Lens thanks to AI advancements. Using the most recent iterations of the Google AI and Gemini models, Google has introduced three new AI features for Chrome. These enhancements expand upon earlier generative AI capabilities, which were unveiled in January and comprised theme generation, tab organization, and writing support.

The following are the new features

Chrome Desktop with Google Lens

Google Lens functionality will be immediately integrated into the desktop browser with the most recent Chrome release.

Without switching tabs, users may choose, look up, and ask enquiries about anything on the internet. Users can highlight any region of interest by utilising the right-click menu or by clicking the Google Lens icon in the URL bar.

The side panel will display visual matches and results. Multi search options let you narrow down your search by brand, colour, or other specifics. Based on the user’s questions, AI Overview replies will offer pertinent information from all over the internet.

Compare Items Using Tab Contrast

Tabs compare is a new feature coming soon to Chrome that creates a comparison table from many tabs to make online purchasing easier.

By combining product details, features, costs, and reviews into a single tab, this feature will do away with the need to switch between tabs.

Users will be able to view all relevant information in one location, for example, when purchasing a new Bluetooth portable speaker, which will facilitate more informed decision-making.

Discover Again Perusing History

Google is using AI to improve Chrome’s history feature so that users can search for previously visited websites in a more conversational and natural way.

Queries such as “What was that ice cream shop I looked at last week?” can be typed by users. to obtain pertinent webpages from their previous surfing sessions.

This feature will remove any browsing data from incognito mode. It is optional and can be turned on or off in the settings.

Accessible

Google Lens on Chrome Desktop: Over the coming days, the most recent Chrome upgrade will make Google Lens available.

Tabs Compare: Coming soon to the United States in the coming weeks.

Rediscovered Browsing History: Coming soon to the United States.

Chrome desktop users get experimental generative AI features

Google has released Chrome (M121) with interesting new features, including experimental generative AI capabilities that will improve your browsing experience without sacrificing personalisation.

Organizing Smart Tabs

Multitasking is more convenient with Chrome’s new Tab Organiser function, which builds tab groupings based on the tabs you have open.

To organise similar Tabs Compare, either right-click on a tab and choose it, or use the drop-down arrow next to your tabs. For convenience, Chrome even suggests names and emoticons for these tab groups.

Artificial Intelligence-Powered Browser Themes

Easily customise your browser with themes created by AI. With Chrome, you may create custom themes based on subjects, emotions, visual styles, and colours thanks to its integration of a text-to-image diffusion paradigm.

Go to the “Customise Chrome” side menu, choose “Create with AI” under “Change theme.”

AI-Powered Help for Writing on the Web

Google plans to deploy a feature driven by artificial intelligence (AI) in the next version of Chrome that will help you write on the web with confidence.

To get AI to start writing for things like leaving reviews, creating RSVPs, or sending formal enquiries, right-click on a text box, select “Help me write.”

Accessible

In the next days, Chrome for Macs and Windows PCs will get these new features, first in the United States. Go to the “Experimental AI” page after logging into Chrome and choosing “Settings” from the three-dot menu to access them.

Please take note that at this time, enterprise and educational accounts are not eligible for these capabilities.

Chrome desktop browser security and performance have been improved by Google

Google has also said that it will be releasing more intelligent and beneficial AI-based features, driven by the recently revealed Gemini AI model, in addition to these new enhancements.

With the most recent version, the Security Check function will scan the system automatically in the background for any threats. Security Check actively searches for malicious extensions, compromised passwords, and site permissions that need to be fixed.

Through Chrome’s three-dot menu, users will be notified of potential security issues and prompted to take appropriate action. Moreover, it makes it simple to remove site rights.

Last year, Google added performance options, such as the Memory Saver mode, to Chrome for desktop users. With Memory Saver mode enabled, the most recent update provides more information on the amount of memory used by tabs.

One useful feature for multitasking in Microsoft Edge is tab grouping, which you may already be familiar with. It assists you in organizing and clearing off your tabs.

Now, Chrome for desktop users may also group tabs. The tab group can have whatever name you choose, and in a later version, you will be able to store it and easily continue on another desktop.

Forrester Wave 2024

Google Cloud are pleased to declare that Google has achieved the highest scores among all vendors assessed in the Current Offering and Strategy categories, making it a Leader in The 2024 Forrester Wave: AI Foundation Models for Language, Q2 2024.

“Gemini stands out in the market primarily due to its multi modality and context length, and it also maintains interoperability with a wider range of complementary cloud services.” – Q2 of 2024 edition of The Forrester Wave: AI Foundation Models for Language

The Forrester Wave

The Forrester Wave: AI Foundation Models for Language, Q2 2024 is available for free download.

Google Cloud’s interactions with Forrester Wave are changing as a result of generative AI. Developers now have the ability to leverage powerful managed models to create creative new applications, experiences, and agents for end users. With only 1% of the data required in the past, tuning models is now easier than ever. The advancement of Gen AI is accelerating across the board.

A long history of AI research and invention at Google, which includes the Transformer architecture, diffusion models, and other ground-breaking projects that are essential to the next generation of AI applications.

Google Cloud’s multi modal family of models, called Gemini, is the product of extensive teamwork from Google departments including Google Deep Mind and Google Research. Gemini models, which were created from the bottom up to seamlessly blend and comprehend text, code, graphics, audio, and video, are assisting developers in the creation of state-of-the-art AI agents for use in almost every industry.

Customers can access Gemini through Vertex AI, a unified, fully-managed platform for creating, deploying, and tracking machine learning models at scale offered by Google Cloud. Customers can tweak and implement Gemini and other AI models with enterprise-ready tuning, grounding, monitoring, and inference capabilities, along with industry-leading AI infrastructure and user-friendly tooling to create AI agents, thanks to Vertex AI’s ability to support both generative and predictive AI models.

Cutting-edge performance

Enterprise clients can purchase the following Gemini models from Vertex AI:

Gemini 1.5 Pro

With only a single prompt, Google Cloud’s customers can now accurately parse massive documents, codebases, or entire films thanks to the industry-leading and groundbreaking context window of 1 million tokens that Google Cloud introduced earlier this year with Gemini 1.5 Pro, which is now generally available. Customers will soon be able to test Gemini 1.5 Pro with up to a 2 million token context window for use cases that demand an even larger context window, such as analysing very large code bases or extensive document libraries (join the waitlist for the 2 million token context window by signing up here).

Flash Gemini 1.5

Additionally, it is now GA and provides Google Cloud’s ground-breaking 1 million token context window. Unlike 1.5 Pro, it is lightweight and made to function quickly and scaleably for jobs like chat applications.

Gemini 1.0 Pro

Developed to manage multiturn text and code chat, code creation, and natural language jobs. A fresh version with supervised tuning capabilities and reduced latency and better quality is usually released.

Pro Vision Gemini 1.0

accepts prompts in multiple modes. You can receive text or code responses to your prompt requests, and you can include text, photos, and video.

Vertex AI

Gemini can now be deployed and customised thanks to Vertex AI, enabling developers to create novel and distinctive apps that handle data from text, code, photos, and video. Developers using Vertex AI can:

Explore and utilise Gemini, or choose from a carefully compiled roster of over 130 models from Google, open-source, and other sources that satisfy Google’s exacting enterprise security and excellence criteria. Models are available to developers as simple-to-use APIs so they can include them into apps rapidly.

Tailor the behaviour of the model to a particular topic or company’s expertise. You may even modify the model weights as needed by using tuning tools to enhance training knowledge. Many tuning methods are offered by Vertex AI, such as distillation, adapter-based tuning like Low Rank Adaptation (LoRA), and prompt design. Furthermore, Google Cloud enable reinforcement learning from human feedback (RLHF) so that a model can be enhanced by obtaining user feedback.

Add tools to models to assist in customizing Gemini Pro for particular scenarios or use cases. Vertex AI Extensions and connectors enable developers to call functions within codebases, access data from external sources, and integrate Gemini Pro to external APIs for transactions and other tasks. Additionally, Vertex AI enables businesses to validate foundation model results against their own data sources, which enhances the precision and applicability of a model’s conclusions. Google Cloud provide businesses with the option to use both grounding with Google Search technology and grounding against their organised and unstructured data.

Utilise technologies designed specifically to manage and scale models in production, ensuring that applications can be readily deployed and maintained after they are produced. Automatic Side by Side (Auto SxS) is an automated tool that is available on demand for customers to compare models. In comparison to manual model evaluation, auto SxS is more economical and quicker. It may also be tailored to different job specifications to accommodate novel generative AI use cases.

Create AI bots in a low- or no-code setting. Developers of all machine learning skill levels may utilise Vertex AI Agent Builder to leverage Gemini models to produce compelling, production-ready AI agents in a matter of hours or days, as opposed to weeks or months.

Use Vertex AI’s safety filters, content moderation APIs, and other responsible AI tools to help developers make sure their models don’t produce offensive content. This will help you deliver innovation in a responsible manner.

With Google Cloud’s integrated data governance and privacy policies, you can help protect data. Consumer data is still in their control, and Google never utilises consumer data for model training. Vertex AI offers several tools, such as Customer Managed Encryption Keys and VPC Service Controls, to give clients complete control over their data.

Current developments

With a focus on scale and enterprise readiness, Vertex AI’s ongoing innovation aims to deliver the finest models available from Google and the industry, combined with an end-to-end model development platform and the ability to construct and deploy agents more quickly. Among the most recent product breakthroughs are:

Sending a huge number of non-latency sensitive text prompt requests in an incredibly quick manner is possible with batch API, which supports use cases like sentiment and classification analysis, data extraction, and description creation. It allows several prompts to be given to models in a single request, which speeds up developer operations and lowers expenses.

Customers may actively manage and utilize cached context data with context caching, which is in public preview this month. Putting long-context apps into production can be costly since processing costs rise with context length. Customers can save a lot of money by using Vertex AI context caching and its cached data.

Customers can specify Gemini model outputs in accordance with particular formats or schemas with controlled generation, which will be available for public preview later this month. Even under explicit directions, the majority of models are unable to ensure the syntax and format of their outputs. Customers can provide custom formats or select from pre-built alternatives like XML and YAML when using Vertex AI driven generation to determine the appropriate output format. JSON is available as a pre-built alternative.

The retrieval augmented generation (RAG) process is made simpler by LlamaIndex on Vertex AI, starting with data acquisition and transformation and continuing through embedding, indexing, retrieval, and generation. Customers of Vertex AI may now connect bespoke data sources to generative models by utilising LlamaIndex’s open-source, simple, and flexible data framework in conjunction with Google’s models and AI-optimized infrastructure.

Firebase just unveiled Genkit, an open-source Typescript/JavaScript framework made to make it easier to create, implement, and keep an eye on production-ready AI agents. Firebase developers may now benefit from text embeddings and Google models like Gemini and Imagen 2, made possible via the Vertex AI plugin.

Grounding with Google Search, which is now widely accessible, enables you to link models with current online material, a broad range of themes, or world knowledge. By integrating Google Search with Gemini models, Google Cloud provide users with access to the most recent foundation models from Google in addition to new, high-quality data, which greatly enhances answer accuracy and completeness.

The latest model in Google Cloud’s family of open models, Gemma 2, was developed using the same technology as Gemini for a wide range of AI developer use cases. Vertex AI Model Garden will soon have Gemma 2 models available.

Imagen 3, Google Cloud’s best text-to-image generation model to date, will soon be available at Vertex AI. It can generate images that are lifelike and photorealistic while maintaining an amazing level of detail.

How users are utilising Gemini models to innovate

API requests increased by almost 6X from H1 to H2 of last year, indicating that Vertex AI has been widely adopted. The remarkable things that customers are accomplishing with Gemini models truly astound them, especially considering how multimodal and adept they are at handling sophisticated thinking.

Samsung: The company just revealed that the Galaxy S24 series is the first to feature Gemini models in a smartphone. Customers can utilise summary tools in Notes and Voice Recorder, starting with Samsung-native applications. Samsung is certain that Vertex AI’s integrated security, safety, and privacy will safeguard its end consumers.

Jasper: Jasper uses Gemini models to swiftly produce marketing campaign content for its clients. Jasper is an AI marketing platform that helps enterprise marketing teams develop on-brand content and campaigns at scale. Teams may now work more quickly without sacrificing the quality of their material, as long as it still follows marketing and brand voice criteria.

Quora: The well-known question-and-answer website Quora is utilising Gemini to support creator monetization on Poe, their AI chat platform, which allows users to explore an extensive selection of AI-powered bots. Poe developers can now create unique bots for a range of use cases using Gemini, like as code generation, writing help, individualised learning, and more.

It gives them great pleasure to be listed as a Leader in both the The Forrester Wave: AI Infrastructure Solutions, Q1 2024 and The Forrester Wave: AI Foundation Models for Language, Q2 2024 reports. To the advantage of Google Cloud’s clients, Google Cloud have applied decades’ worth of AI R&D experience to developing models, ultra-scale infrastructure, and Vertex AI capabilities. Google Cloud are dedicated to continuing AI research and innovation.

Google has massive AI infrastructure, a large pool of AI researchers, and a growing number of enterprise customers to lead the AI market Google Cloud.” – Q2 2024 edition of The Forrester Wave: AI Foundation Models for Language

SAP Datasphere

According to Google Cloud and SAP, the cloud should provide more than just incredibly dependable, secure, and performant infrastructure. Additionally, it must to offer a multi-cloud, open, and comprehensive platform that lets your company leverage generative AI and analytics to extract the most value from its data. This is particularly relevant to important data that is kept on SAP systems, which include supply chains, finance, human resources, and more.

The Google Cloud alliance enables your company realise the full potential of its information to create continuous innovation, take on your most difficult business challenges, and make more informed decisions by combining Google Cloud’s generative AI and analytics capabilities with SAP data.

Google Cloud is thrilled to present new developments in AI and infrastructure at the SAP Sapphire conference today, enabling SAP clients to innovate and handle data more efficiently. These developments include:

A solution driven by AI to make supply networks more robust. The system combines SAP Joule and Integrated Business Planning (IBP) for Supply Planning with Google Cloud Cortex Framework and Gemini models.

A combined analytics solution with SAP Datasphere is currently offered on the Google Cloud Marketplace.

The release of memory-optimized X4 instances from Google Cloud, which offer the biggest SAP-certified compute instances available in the cloud and can support workloads up to 32TB SAP HANA.

Regional RISE extensions utilising SAP, SAP BTP, and SAP Datasphere in addition to assistance with RISE’s EU access.

Multiple Gemini updates that let SAP users use SAP GenAI Hub to leverage Google Cloud’s big language models and provide more gen AI capabilities to SAP workloads.

Accelerating resilient supply chains enabled by AI

Google Cloud is delving further into business-critical use cases that will provide clients a competitive edge as they collaborate with SAP on new innovations. It makes sense to use Google Cloud’s combined data and AI technologies to improve supply chain management and lower supply chain risks.

Organisations may accelerate insights from sensed events, threats, and opportunities to allow faster, more intelligent supply chain reactions by combining Cortex Framework and Gemini models with SAP’s generative AI assistant Joule and Integral Business Planning (IBP) for Supply Planning.

You may also assist in improving demand forecasting and supply chain risk mitigation using an AI-powered resilient supply chain, including use cases such as:

Improved demand forecasting with the incorporation of many data sources, including advertising efforts.

Supply chain risk mitigation using proactive disruption detection and alarms, supported by other data sources including meteorological and event data.

Enhanced productivity through options for automation, the capacity to manage ideal inventory levels, and more accurate planning.

You can now get SAP Datasphere on the Google Cloud Marketplace

Real-time access to your SAP and non-SAP data is still made possible by the SAP Datasphere integration with BigQuery. This includes access to publically accessible datasets within BigQuery as well as additional integrated data, such as Google Ads, that is accelerated by the Cortex Framework. Bidirectional replication and federation with BigQuery are now possible with SAP Datasphere’s Replication Flow, allowing for a petabyte-scale analytics and data science platform in addition to providing a basis for machine learning and gen AI with Vertex AI and Gemini. With SAP Datasphere for Google Cloud, which is offered on the Google Cloud Marketplace, all of this is accessible “out of the box.”

Customers using SAP Datasphere for Google Cloud have already produced impressive outcomes; one such customer was a large consumer electronics company that updated its sales reporting procedure. In order to replace their outdated PDF reports and update their entire reporting process, they developed a real-time sales reporting dashboard using SAP Datasphere, BigQuery, and Google Dataform for sales pipeline automation. They have consequently enhanced operating speed, saved a substantial amount of money, and produced greater business insights on business sales.

Presenting X4

More memory- and compute-intensive systems are needed to handle massive workloads as more multinational corporations move to SAP S/4HANA on the cloud. With 16TB, 24TB, and 32TB instances created especially for SAP HANA OLTP and OLAP workloads to run with optimal performance and dependability, Google Cloud is excited to present the X4 machine family. The X4 family leads the industry in several areas, including:

32TB instances certified by SAP HANA initially

Due to Google Cloud’s Titanium off-load technology, it offers 1920 vCPUs, more than double the amount of other approved hyperscalers, and industry-leading compute and Hyperdisk block storage speed (up to 10,000 MBps throughput).

With the industry-highest Compute Engine Memory-Optimized Single Instance SLA of 99.95%, the X4 Instance family guarantees outstanding dependability for SAP applications.

These systems also gain from the infrastructure differentiators offered by Google Cloud, which include the widest network footprint that provides low latency for SAP workloads and strong security features like proactive threat detection and inbuilt encryption.

According to Deloitte’s US Chief Technology Officer Shawn Lund, “our SAP HANA systems have seen significant data growth in the past few years with an increasing need for higher performance.” Google Cloud’s 24TB X4 servers and Hyperdisk storage have raised the bar for our data growth and performance. Also, Google’s X4 machines are cloud native, allowing system management and operations automation.”

Developing with SAP Clients for RISE

One example of how Google Cloud is investing to be the most dependable and scalable cloud provider for RISE with SAP workloads is the new high-performance X4 Instance family.

“SAP and Google Cloud have a deep co-innovation partnership, and Google Cloud is continuously launching new, differentiated solutions to benefit RISE with SAP customers,” says Lalit Patil, CTO of RISE with SAP.

Google Cloud is pleased to introduce two new products at SAP Sapphire:

New X4 Systems for SAP RISE

The most recent 16, 24, and 32TB X4 systems from Google Cloud are now supported by SAP RISE. This co-engineered solution will provide the performance and dependability required for even the biggest and most demanding SAP workloads, as proven by Google Cloud’s extensive testing through customer POCs.

Enhanced Compliance for Users in the EU

Google Cloud is happy to announce that EU Access for RISE is now available on Google Cloud. This takes care of Google Cloud’s European customers’ regulatory and compliance needs.

Asian Paints is a new RISE customer on Google Cloud that works with SAP. Aashish Kshetry, CIO of Asian Paints, states, “We chose Google Cloud for RISE with SAP because of their strong partnerships with SAP and their ability to manage large installations in the region.” “We are eager to implement SAP S/4HANA on their most recent platforms and infrastructure on Google Cloud in order to support our business innovations and long-term growth.”

Other recent developments that further establish Google Cloud’s standing as the best cloud partner for RISE with SAP clients are as follows:

The extension of SAP BTP and SAP Datasphere, with new launch areas scheduled for this quarter in Israel and Japan and later this year in Saudi Arabia, Australia, and Brazil.

The addition of SAP BTP to the ABAP SDK for Google Cloud. With the release of a fully supported version of the ABAP SDK on SAP BTP, Google Cloud becomes the first hyperscaler to assist customers in integrating AI, analytics, and automation into their SAP business processes through the use of 55 Google Cloud APIs.

Through SAP’s Customer Database Platform (CDP), organisations can directly integrate Vertex AI, bringing predictability to business operations, enhancing customer experiences, and spurring growth.

With the help of a natural language interface, SAP administrators may manage SAP deployments with the help of Gemini Assist for SAP, a part of Google Cloud Workload Manager (WLM).

Google Gemini LLMs are available in SAP GenAI Hub to enable SAP Joule and embedded generative AI capabilities across SAP’s product line. Ariba, SuccessFactors, S/4, and other workloads can all be improved by Google Cloud models. The most recent models from Google Cloud, such as Gemini Flash, which was unveiled at Google I/O in May, will remain available to SAP clients.