#alloydb

AlloyDB Omni boosts performance with vector search, analytics, and faster transactions.

With its latest release AlloyDB Omni version 15.7.0, AlloyDB Omni is back and is significantly improving your PostgreSQL workflows. These improvements include:

Quicker performance

A brand-new, lightning-fast disk cache

A better columnar engine

The widespread use of ScANN vector indexing

The AlloyDB Omni Kubernetes operator has been updated.

In your data center, on the edge, on your laptop, in any cloud, and with 100% PostgreSQL compatibility, this update offers on all fronts, from transactional and analytical workloads to state-of-the-art vector search.

AlloyDB Omni version 15.7.0 is now broadly accessible (GA). The following updates and features are included in version AlloyDB Omni version 15.7.0:

AlloyDB Version 15.7 of PostgreSQL is supported by Omni.

Previously known as postgres_scann, the alloydb_scann extension is now generally available (GA).

There is generally available (GA) support for Red Hat Enterprise Linux (RHEL) 8.

You can preview the AlloyDB Omni columnar engine on ARM.

Because disk cache and columnar storage cache speed up data access for AlloyDB Omni in a container and on a Kubernetes cluster, they can enhance AlloyDB Omni performance.

It has applied security updates for CVE-2023-50387 and CVE-2024-7348.

The documentation for the AlloyDB Omni Reference is accessible. This comprises AlloyDB Omni 15.7.0 metrics, database flags, model endpoint management reference, and extension documentation.

AlloyDB The pg_ivm extension, which offers incremental view maintenance for materialized views, is compatible with Omni.

Numerous efficiency enhancements and bug fixes.

Let’s get started.

Improved performance

When compared to regular PostgreSQL, many workloads already experience an improvement. For transactional workloads, AlloyDB Omni outperforms regular PostgreSQL by more than two times in performance testing. The majority of the tuning is done automatically for you without the need for additional setups. The memory agent that maximizes shared buffers while preventing out-of-memory issues is one of the main benefits. AlloyDB Omni generally runs better with more memory configured because it can serve more queries from the shared buffers and eliminate the need for disk calls, which can be significantly slower than memory, especially when utilizing durable network storage.

An extremely fast disk cache

The introduction of an ultra-fast disk cache also made the trade-off between memory and disk storage more flexible. As an extension of Postgres’ buffer cache, it enables you to set up a quick, local, and perhaps brittle storage device. AlloyDB Omni can store a copy of not-quite-hot data in the disk cache, where it can be accessed more quickly than from the permanent disk, rather than aging out of memory to create room for new data.

Improved columnar engine

The analytics accelerator from AlloyDB Omni is revolutionizing mixed workloads. Because it eliminates the need to manage additional data pipelines or databases, developers are finding it helpful for extracting real-time analytical insights from their transactional data. To speed up queries, you can instead activate the columnar engine, allocate a piece of your memory to it, and let AlloyDB Omni to choose which tables or columns to load in the columnar engine. The columnar engine outperforms regular PostgreSQL by up to 100x in our benchmarks for analytical queries.

The amount of RAM you can allocate to the columnar engine dictates the analytics accelerator’s practical size limit. The ability to set up a quick local storage device for the columnar engine to spill to is a new feature. This expands the amount of data on which you may do analytical queries.

SCaNN becomes GA

Finally, AlloyDB Omni already provides excellent performance with pgvector utilizing either the ivf or hnsw indexes for vector database use cases. Vector indexes, however, can be slow to build and reload even though they are a terrific method to speed up queries. It added the ScaNN index as an additional index type at Google Cloud Next 2024. The ScaNN index from AlloyDB AI provides up to 4 times faster vector queries than the HNSW index used in ordinary PostgreSQL. ScaNN offers substantial benefits for practical applications beyond only speed:

Rapid indexing: With noticeably quicker index build times, you may expedite development and remove bottlenecks in large-scale deployments.

Optimized memory usage: Cut memory usage by three to four times as compared to PostgreSQL’s HNSW index. This improves performance for a variety of hybrid applications and enables larger workloads to operate on smaller hardware.

In general, AlloyDB AI ScANN indexing is accessible as of AlloyDB Omni version 15.7.0.

A fresh Kubernetes administrator

Google Cloud has published version 1.2.0 of the AlloyDB Omni Kubernetes operator in addition to the latest version of AlloyDB Omni. With this release, you can now configure high availability to be enabled when a disaster recovery secondary cluster is promoted to primary, add more configuration options for health checks when high availability is enabled, and use log rotation to help manage the storage space used by PostgreSQL log files.

Version 1.2.0 of the AlloyDB Omni Kubernetes operator is now broadly accessible (GA). The following new features are included in version 1.2.0:

The interval between health checks can be set in seconds using the healthcheckPeriodSeconds option.

You can keep an eye on your database container’s performance with the following metrics. These measurements are all type gauge.

A database container’s memory limit is displayed by alloydb_omni_memory_limit_byte.

All replicas connected to the AlloyDB Omni primary node are shown in alloydb_omni_instance_postgresql_replication_state.

The database container’s memory usage is displayed in bytes via alloydb_omni_memory_used_byte.

When the following is true, a problem that briefly disrupted all database clusters has been resolved:

The AlloyDB Omni Kubernetes operator version 1.1.1 is being upgraded to a more recent version.

Version 15.5.5 or higher of the AlloyDB Omni database is what you’re using.

AI for AlloyDB is not activated.

Once promoted, high availability is supported on a secondary database cluster.

Model endpoint management can be enabled or disabled using Kubernetes manifests.

By setting thresholds depending on the size of the log files, the amount of time since the log file last rotated, or both, you may control when logs rotate.

To examine and troubleshoot the memory performance of the AlloyDB Omni Kubernetes operator, you can take a snapshot of its memory heap.

Note: Parameterized view features were accessible via the alloydb_ai_nl extension of AlloyDB Omni versions 15.5.5 and earlier. The parameterized_views extension, which you must develop before using parameterized views, contains the parameterized view features starting in AlloyDB Omni version 15.7.0. The associated function, google_exec_param_query, has also been renamed to execute_parameterized_query and is accessible through the parameterized_views extension as of AlloyDB Omni version 15.7.0.

AlloyDB ScaNN index

When developing a general intelligence or search application, you require real-time, high-quality results based on the most recent data for your application. Additionally, you need solutions that are affordable to run at scale and simple for developers to create and maintain.

Google Cloud is excited to announce today the general availability of the ScaNN for AlloyDB index, which will enable businesses to create scalable, accurate, and high-performing gen AI and search applications. For demanding enterprise workloads, including 4x faster transactional workloads and up to 100x faster analytical queries than regular PostgreSQL, AlloyDB for PostgreSQL is Google Cloud fully managed PostgreSQL-compatible solution. The ScaNN index now supercharges vector database workloads in AlloyDB for PostgreSQL by bringing 12 years of Google research, the same technology that powers some of Google’s billion+ user services.

With the help of the ScaNN index, AlloyDB combines the business features, multi-workload capacity, and query flexibility of a relational database with the efficiency of a vector-optimized database, providing precise results in record time and scalability to accommodate over a billion vectors.

ScaNN for AlloyDB index

With its ability to accommodate over a billion vectors and retain state-of-the-art query performance across the board, the ScaNN for AlloyDB index is the first PostgreSQL-compatible index that offers high performance at <25ms p95 latency at 1B vectors with 95% recall. In comparison to the HNSW index in regular PostgreSQL, it offers up to 4 times faster vector queries. Furthermore, AlloyDB’s ScaNN index has significant advantages for real-world workloads, including quick index creation times that boost developer productivity and cut down on costly bottlenecks for heavy lifting. Additionally, it uses memory efficiently, usually 3–4 times less than the regular PostgreSQL HNSW index.

This allows larger workloads to operate on smaller computers and enhances performance for hybrid workloads. Ultimately, AlloyDB enables users to appropriately scale for their unique workloads by supporting indices that are significantly larger than what can fit in main memory. When combined, these enhancements significantly increase the range of vector database workloads that AlloyDB can handle effectively and minimize the requirement for specialist tools.

AlloyDB has all the advantages of a relational database of commercial caliber while yet being affordable. With full PostgreSQL compatibility, developers may query vectors and scalar data simultaneously using an easy-to-use and versatile query language that supports a wide range of operations, from straightforward filters to intricate joins to hybrid searches on a single database. High write throughput and complete transactional consistency are supported by AlloyDB.

With ScaNN for AlloyDB index having up to 10 times the write throughput of normal PostgreSQL’s HNSW index, users with extremely high update rates may be guaranteed that their writes will be performant and easily queryable. Lastly, AlloyDB offers a 99.99% uptime SLA that is pleasant to customers and covers maintenance in addition to essential enterprise features like high availability, disaster recovery, data security, and more.

Professional developers who now utilize PostgreSQL, the most widely used open source database, may now easily use this technology with the new ScaNN index for AlloyDB, which is fully compatible with both PostgreSQL and pgvector.

Create indexes

Before you begin

The following requirements must be fulfilled before you can begin building indexes.

Within your AlloyDB database, an embedded vector is added to a table.

Installed is the pgvector extension, which Google enhanced for AlloyDB and is at least version 0.5.0.

CREATE EXTENSION IF NOT EXISTS vector;

Install the alloydb_scann extension in addition to the pgvector extension in order to produce ScaNN indexes.

CREATE EXTENSION IF NOT EXISTS alloydb_scann;

Create ScaNN index

Alloy DB Google’s alloydb_scann is a PostgreSQL modification that uses the ScaNN algorithm to create an extremely effective nearest-neighbor index.

For the approximate closest neighbor search, the ScaNN index is a quantization index based on trees. Compared to HNSW, it offers a shorter index creation time and a smaller memory footprint. On the basis of workload, it also offers faster QPS in contrast to HNSW.

Two-level ScaNN index tree

Use the following DDL query to apply a two-level tree index using the ScaNN algorithm to a column holding stored vector embeddings.CREATE INDEX INDEX_NAME ON TABLE   USING scann (EMBEDDING_COLUMN DISTANCE_FUNCTION)   WITH (num_leaves=NUM_LEAVES_VALUE); Replace the following: INDEX_NAME: the name of the index you want to create for example, my-scann-index. The index names are shared across your database. Ensure that each index name is unique to each table in your database. TABLE: the table to add the index to. EMBEDDING_COLUMN: a column that stores vector data. DISTANCE_FUNCTION: the distance function to use with this index. Choose one of the following: L2 distance: l2 Dot product: dot_product Cosine distance: cosine NUM_LEAVES_VALUE: the number of partitions to apply to this index. Set to any value between 1 to 1048576. For more information about how to decide this value, see Tune a ScaNN index.

Three-level ScaNN index tree

Use the following DDL query to generate a three-level tree index to a column containing stored vector embeddings using the ScaNN algorithm:CREATE INDEX INDEX_NAME ON TABLE   USING scann (EMBEDDING_COLUMN DISTANCE_FUNCTION)   WITH (num_leaves=NUM_LEAVES_VALUE, max_num_levels = MAX_NUM_LEVELS); Replace the following: MAX_NUM_LEVELS: the maximum number of levels of the K-means clustering tree. Set to (default) for two-level tree-based quantization and to for three-level tree-based quantization.

Once the index is created, you can use it to execute nearest-neighbor search queries by following the guidelines in Make a nearest-neighbor query using specified text.

A proper balance between recall and QPS must be achieved while setting the index parameters. See Tune a ScaNN index for additional details on adjusting the ScaNN index.

Cast the column into the vector data type in order to build this index on an embedding column that utilizes the real data type rather than vector:CREATE INDEX INDEX_NAME ON TABLE   USING scann (CAST(EMBEDDING_COLUMN AS vector(DIMENSIONS)) DISTANCE_FUNCTION)   WITH (num_leaves=NUM_LEAVES_VALUE, max_num_levels = MAX_NUM_LEVELS);

Substitute DIMENSIONS with the embedding column’s dimensional width. See the vector_dims function in Vector functions for additional details on how to determine the dimensions.

To view the indexing progress, use the pg_stat_progress_create_index view:SELECT * FROM pg_stat_progress_create_index;

The phase column displays the stage at which your index is currently being built. After the index is created, the tree training phase ends.

Aiven’s relationship with AlloyDB Omni makes multi-cloud, managed PostgreSQL possible.

Businesses now want a database that not only offers outstanding performance but also supports open standards and stays away from vendor lock-in, given the fast changing technological world. Let introduce you to AlloyDB Omni, a high-performance, 100% PostgreSQL-compatible downloadable database from Google Cloud that offers state-of-the-art features like vector processing, near real-time analytics, and scalable online transaction processing (OLTP) architecture.

And in a development that further broadens the availability and reach of AlloyDB Omni, Google Cloud has partnered with Aiven, a top supplier of multi-cloud data infrastructure. With the integration of AlloyDB Omni’s robust features into Aiven’s platform, developers and enterprises can now manage and deploy PostgreSQL workloads with even more flexibility.

AlloyDB Omni can be deployed, scaled, and managed more easily and securely on Google Cloud, Amazon Web Services (AWS), and Microsoft Azure with the help of Aiven for AlloyDB Omni, a managed cloud database solution that offers integrated multi-cloud operations. via this agreement, you can execute transactional, analytical, and vector workloads on any cloud, all managed via a single platform, and benefit from the flexibility of managed, multi-cloud deployments with AlloyDB Omni.

Freedom to choose, adapt, and use several clouds

For developers, this collaboration between Aiven and Google Cloud opens more options and freedom. Now, you can use Aiven’s extensive array of managed data services to leverage the capabilities of AlloyDB Omni across all major cloud providers. Aiven offers a standardized control plane that improves operational effectiveness, streamlines administration, and fortifies security and governance.

This gives you the ability to update apps and realize the full potential of your data, wherever it may be stored. This translates to software companies being able to supply apps in a multi-cloud environment by building once and deploying anywhere. Aiven’s dedication to a multi-cloud approach gives you the flexibility to choose the ideal cloud provider and location for your unique requirements while avoiding vendor lock-in.

Important attributes of Aiven and AlloyDB Omni

Support heavy, complicated workloads: AlloyDB Omni offers up to 100 times quicker analytical queries (OLAP) than ordinary PostgreSQL, and two times faster transactional performance (OLTP) for your important applications.

Prepared for the creation of a Gen AI application: Using 12 years of Google’s vector search expertise, AlloyDB provides quick, scalable, accurate, and economical vector searches. In addition to providing up to 4 times quicker vector queries and often using 3–4 times less memory than the HNSW index in standard PostgreSQL, AlloyDB AI delivers quick index creation times to increase development productivity. This enables you to differentiate yourself from the competition by creating a highly responsive and inventive consumer experience.

Unmatched deployment options: A multi-cloud approach is necessary in the current scenario. Companies must have the flexibility to choose the most suitable cloud for every task, stay away from vendor lock-in, adhere to data sovereignty regulations, and improve resilience. While AlloyDB Omni’s integration with the Aiven managed service reinforces the argument for multi-cloud and improves your multi-cloud posture by enabling you to utilize best-of-breed solutions from other providers, it is also 100% compatible with PostgreSQL.

Simplified administration: AlloyDB Omni setup and operation are made easier by Aiven’s user-friendly platform and powerful management tools, which lower operating overhead and free up developers to concentrate on creating and inventing. Aiven for AlloyDB Omni greatly streamlines PostgreSQL workload management when used with AlloyDB capabilities like adaptive vacuum, index advisors, and more.

Improved ecosystem: With this cooperation, AlloyDB Omni is now part of Aiven’s extensive network of open-source data services, which also includes OpenSearch, Grafana, Apache Kafka, and Valkey. This offers an all-inclusive framework for developing and overseeing contemporary data-driven applications.

For AlloyDB, Aiven Omni for your contemporary applications

AlloyDB Omni and Aiven together provide a strong platform for developing and executing contemporary applications. Together with Aiven’s user-friendly interface and extensive data services, AlloyDB Omni’s excellent speed, scalability, and PostgreSQL compatibility enable developers to design creative solutions that generate revenue.

Galxe Quest

The best platform for creating web3 communities is called Galxe Quest. Galxe connects projects with millions of consumers through reward-based loyalty programs while providing a straightforward, no-code solution. Optimism, Arbitrum, Base, and other prominent players in the market trust Galxe as their portal to the largest onchain distribution network on web3.

Create the community of your dreams

Accurate User Acquisition

Turn on autopilot mode for your product marketing and user growth, from on-chain KPIs to off-chain engagements.

Simplified Onboarding of Users

ZK technology is used to encrypt and store your identification data in off-chain vaults so that Galxe never has access to unencrypted data.

More People, Fewer Robots

Your information is private to you alone, and you have complete control over it. As you see appropriate, choose when to provide or deny access to third parties.

Boost Recognition of Your Brand

Your universal web3 identification is Galxe Passport, which removes the need for numerous sign-ups and verifications for various services and apps.

Recognize Faithful Users

Your information is private to you alone, and you have complete control over it. As you see appropriate, choose when to provide or deny access to third parties.

Boost User Retention

Your universal web3 identification is Galxe Passport, which removes the need for numerous sign-ups and verifications for various services and apps.

Plug-and-play, no code solution

Select one of the many Quest templates and easily customize with a few clicks.

Provide Your Own Information

Add your own Google Sheets, CSV files, and API connections to the BYOD (Bring Your Own Data) experience.

Chain-wide Imprint

To automatically track on-chain footprint, connect an API or subgraph.

Twitter Interaction

Keep tabs on interactions like following, liking, retweeting, and attending Twitter Space.

Activities on Discord

Using the Galxe Discord Bot, you can keep tabs on active members, AMA attendees, and Discord roles.

Contributions on GitHub

Recognize the top developers who worked on your product.

Participation in Events

Geo-fenced QR codes facilitate the easy tracking of attendees at both offline and online events.

Status of DAO Voting

With a few clicks, import every voter from your Snapshot.org proposal.

Confirmed People

Your community is shielded from bothersome bots and sybil attacks by using sybil prevention credentials.

Web3 is made of community. Your community and contributors can be encouraged with a range of rewards, such as tokens, loyalty points, reputation, and accomplishments. Learn about other features that Galxe Quest provides.

The next wave of internet development based on blockchain technology is referred to as Web3. It seeks to safeguard self-sovereign identification information, facilitate transactional freedom, decentralize online ownership, and accomplish other goals. Web3, however, is a quickly growing industry with its own set of difficulties. For both developers and end users, there are significant entrance barriers and onboarding friction due to the lack, if not outright absence, of a uniform infrastructure and user experience.

In order to address these current issues, Galxe has created an ecosystem of goods with the goal of bringing the entire planet online. Galxe Quest, the biggest on-chain distribution platform for Web3, is essential to this ecosystem since it links Web3 communities and companies through gamified learn-to-earn opportunities. Galxe Identity Protocol, Google Cloud’s permissionless self-sovereign identity infrastructure, which standardizes the creation, validation, and distribution of on-chain credentials using Zero-Knowledge Proofs, is the foundation upon which this platform is based. Additionally, they introduced Gravity, a Layer-1 blockchain intended for widespread use that eliminates the technical difficulties associated with multichain interactions. This enables developers to leverage Galxe’s 26 million users to produce new Web3 products.

AlloyDB for PostgreSQL earns Google Cloud confidence

Galxe Quest, with more than 26 million active users, requires scalable solutions as well as reliable and substantial data access. A key component in scaling the platform to satisfy the needs of Google expanding user base has been AlloyDB for PostgreSQL.

Because AlloyDB is fully compatible with PostgreSQL and offers high performance for fast online transaction processing (OLTP) workloads, Google Cloud may transition to it with little difficulty and at a reduced cost. After switching to AlloyDB from their previous solution, Amazon Aurora for PostgreSQL, which had much higher costs owing to read and write operations, Google Cloud is able to reduce database expenditures by 40%.

Google Cloud has faith in AlloyDB because of its exceptional performance, almost little downtime, and adaptability. AlloyDB securely stores millions of on-chain and off-chain user data records, serving as their only source of truth. Google Cloud developers may now access detailed datasets to construct blockchain-based loyalty schemes.

Everything we require for faith in Web3

Galxe’s Web3 projects are powered by multiple Google Cloud services. Depending on the requirements of each job, Google Cloud employ Google Kubernetes Engine (GKE) for containerized workloads and AlloyDB and BigQuery for data analysis and storing. Setting up services that continually replicate data from Amazon Aurora to AlloyDB with the goal of minimizing downtime during the transfer was made simple by Google Cloud’s serverless Database transfer Service. Additionally, they used Datastream, a user-friendly change data capture tool that reads and copies AlloyDB data automatically to BigQuery for analysis.

Google Cloud’s network, with its worldwide coverage, low latency, and strong stability offered by premium service tiers, supports Google Cloud’s objective to onboard everyone on the Web 3. With an unmatched user experience, Google Cloud’s consumers worldwide may now easily utilize Galxe’s solutions to explore Web3. Galxe’s architecture also heavily relies on memorystore, which acts as a caching mechanism to manage the substantially higher volume of read operations than write operations in their situations.

Spearheading the upcoming Web3 innovation wave

Galxe collaboration with Google Cloud has yielded several advantages, including improved scalability and stability at a markedly reduced cost. Their ability to grow and support the global Web3 enterprises that are just getting started has shown how important Google Cloud is. Furthermore, AlloyDB’s machine-type scaling and nearly 0% downtime for maintenance windows provide for seamless, uninterrupted service and quick scalability for their clients, allowing them to innovate and expand without hindrance.

SEEBURGER BIS

The SEEBURGER BIS(Business Integration System) on AlloyDB is a scalable and secure platform for business transactions.

As cloud-first databases continue to lead the way in technology, businesses creating software to address difficult, mission-critical problems are increasingly turning to Google Cloud as their platform of choice. That’s because of its unparalleled scalability, capacity to manage erratic demand, exceptional dependability, and ongoing innovation in cutting-edge technologies like artificial intelligence (AI).

A good example is the SEEBURGER BIS Platform. With billions of dollars in trade depending on a dependable and scalable data infrastructure, it enables smooth application connectivity and process automation in sectors including manufacturing, automotive, banking, finance, and logistics. Understanding that ongoing innovation was necessary to spur future growth and streamline operations, SEEBURGER set out to optimise the advantages of cloud-first technology.

This approach is in line with a significant trend in  cloud computing, which is the strategic use of cloud services to modify fundamental business processes rather than merely moving workloads to the cloud. New approaches are needed to meet the demands of trends like edge computing, multi-cloud strategies, Artificial Intelligence and ML, and an unwavering emphasis on security. Integration is one area where this is evidently the case; in order to address these changing needs, integration Platform-as-a-Service (iPaaS) solutions that are scalable, secure, and flexible are quickly ruling the industry.

This blog traces the steps that SEEBURGER took to integrate AlloyDB for PostgreSQL into its product, from the tangible benefits realised to the advanced features of AlloyDB that it utilised. It also highlights the stringent standards that went into choosing AlloyDB as the cloud database of choice.

BIS SEEBURGER

The digital transformation best practice approach

With dynamic deployment methods, such as iPaaS and fully managed services, the SEEBURGER BIS Platform integrates people, apps, and processes across cloud, hybrid, and on-premises settings harmoniously. For more than three decades, SEEBURGER has provided integration features like managed file transfer, application integration, B2B/EDI, API management, and more through its BIS Platform. With the help of this robust integration platform, customers may boost their digital ecosystems by designing, automating, and managing their own integrations. Each day, more than 14,000 clients depend on SEEBURGER’s integration expertise.

SEEBURGER made the decision to optimise the BIS Platform for industry-leading hyperscalers like Google Cloud after realising the cloud’s explosive expansion.

With its business goals in mind, SEEBURGER was eager to choose a database that would empower developers. Below are the five main factors the business took into account:

Enterprise-grade features: For mission-critical applications like BIS, SEEBURGER needs robust capabilities that remove the need for intricate workarounds in the code, while also improving reliability and scalability and removing operational toil.

Workload flexibility: The database should be able to move beyond the capabilities of regular PostgreSQL to handle workloads that are both transactional and analytical.

High availability: The smooth functioning of the system depends on it.

Developer productivity: The most efficient way to maximise developer efficiency is to have a fully managed, highly scalable database that adheres to an open standard.

Flexibility and affordability: SEEBURGER looks for affordable solutions that assist deployment across various cloud providers, on-premises, or hybrid settings and prevent vendor lock-in.

During the evaluation of AlloyDB for PostgreSQL, SEEBURGER had strict specifications. It was outstanding in several crucial areas, such as:

Ensuring a seamless adoption process, seamless PostgreSQL compatibility removed the requirement for code changes.

Superior scalability for bigger deployments is made possible by AlloyDB’s clever use of disaggregated computing and storage.

Minimised operational overhead: Functionalities such as enhanced vacuuming, drastically decreased replication latency, and flexible resource allocation greatly simplify and boost dependability of operations.

Flexibility in deployment: As a provider of enterprise apps, AlloyDB Omni maximises its flexibility by offering the essential choice for non-Google Cloud deployments (on-premises or in cloud environments).

Image credit to Google cloud

A typical BIS Platform architectural configuration utilising Google Cloud and AlloyDB is shown in the above diagram. It illustrates the several BIS components secure data transfer, data transformation, business and technical monitoring, and integration with other services that are put into practice to process and manage workload in order to achieve maximum throughput.

An effective customer use case for logistics

Using the SEEBURGER iPaaS solution, which is driven by the BIS Platform running on AlloyDB, a worldwide logistics and shipping network hub serving over 120 countries was able to optimise operations and speed global services.

Google Cloud and AlloyDB were chosen by SEEBURGER and the logistics customer following a consideration of architectural recommendations and preliminary performance test results. It was simpler to adopt AlloyDB because the BIS Platform already supported PostgreSQL. The scalable nature of AlloyDB and its features like analytics, tiered caches, data tiering, and integrated machine learning for performance optimisation and complex workload management along with its excellent compatibility with the agile, secure, and scalable SEEBURGER iPaaS solution further reinforced SEEBURGER’s decision. The setup and test ran without a hitch or modifications, and no changes to the code were necessary.

An inventive, nimble, safe, and expanding alliance

The SEEBURGER BIS Platform, in conjunction with Google Cloud‘s powerful data services, offers businesses the essential infrastructure they require. In today’s world, data drives business. SEEBURGER’s BIS platform has been strengthened by the utilisation of AlloyDB’s sophisticated functionalities, which guarantees unmatched security, outstanding scalability, and extraordinary agility. Through this partnership, organisations will receive a resilient and future-ready solution that easily adjusts to changing needs, demonstrating a shared commitment to innovation. The combination of AlloyDB and SEEBURGER BIS enables enterprises to boldly traverse the challenges of contemporary corporate integration, thereby boosting operational effectiveness and opening up fresh avenues for expansion.

Looking ahead, the collaboration between SEEBURGER BIS and AlloyDB is a prime illustration of how innovative technology may enable companies to prosper in a world that is becoming more interconnected by the day. See SEEBURGER’s collaboration with Google Cloud or read up on AlloyDB for additional details.

What is iEEG

Mountain View-based NeuroPace, Inc. makes the FDA-approved RNS System responsive neurostimulator device for people with refractory focal onset epilepsy. A lead with four electrode connections can be connected to a neurostimulator in up to two ways. Both the delivery of electrical pulses and the detection of aberrant patterns unique to each patient are controlled by physicians. Intracranial electroencephalograms, or iEEGs, are recorded by the device and span four channels with a sample rate of 250 Hz each. Typically, the recordings contain roughly 90 seconds of data. Almost 5,000 patients have provided over 16 million iEEG files to date.Image Credit to Google Cloud

Determining efficient stimulation patterns for the reduction of seizures is a primary research objective at NeuroPace. A theory posits that treatment settings that have proven successful for patients with comparable iEEG activity may also work for newly diagnosed patients or patients requiring modifications to their current stimulation regimen, thus potentially enhancing the traditional trial-and-error method of determining stimulation programming settings.

Physicians can swiftly identify similar patient profiles based on chosen iEEG files if large-scale iEEG data is searched for similar brain activity patterns. This is necessary for such data-driven procedures to be feasible. Previously, locating similar iEEG files among patients necessitated a convoluted processing pipeline that included grouping iEEG data within patients, locating cluster centres, and locating approximate nearest neighbours (ANN) from other patients using dimensionality reduction techniques like PCA and t-SNE. The flexibility and practical utility of the programme were limited because the approximate nearest neighbours were only computed once every several months using a small sample of new patient iEEG recordings.

Positively, vector databases may now be queried directly for comparable vector embeddings thanks to recent developments. Without having to complete clustering procedures beforehand, this invention may allow doctors to choose any iEEG file from a patient and locate comparable cross-patient iEEG data. As fresh iEEG files become available, all that is needed is to keep the vector database updated. Improved scalability could result from this reduction, which would make it much easier to query comparable iEEG files over millions of entries.

Using embedding models to turn iEEG data into vectors, the NeuroPace  AI team and Google Cloud developers carried out a proof-of-concept study. The vector data was then stored in a vector database called AlloyDB for PostgresQL. AlloyDB is a fully managed database that performs vector similarity searches based on the pgvector extension and is compatible with PostgreSQL, making it ideal for heavy transaction workloads.

AlloyDB Omni

AlloyDB Omni, a version of the database that can be downloaded and used anywhere, further permits on-premises hosting of the database, preserving the data inside the confines of an on-prem HIPAA-compliant environment. Reducing reliance on external network connectivity by having the database on-premise also lessens the chance of application outages that might arise from hosting the database externally while the remainder of the application is hosted on-premise.

Google Cloud handled almost 1.2 million iEEG files from 414 clinical trial participants in this proof-of-concept project. 20 patients’ worth of data were used for testing, and 394 patients’ data were added to the AlloyDB cloud service. A unique embedding model created by the NeuroPace  AI team converted each iEEG file into a spectrogram image and then into vectors. After that, 50 randomly chosen iEEG files from the test cohort were utilised to query the AlloyDB vector database, which now contained these vectors (Figure 2).Image credit to Google Cloud

AlloyDB with PGvector offers three distinct index types (Hierarchical Navigable Small World (HNSW), IVFFLAT, and IVF) that help reduce latency while conducting similarity searches in comparison to a brute-force search:

Using a graph-based technique, HNSW creates several layers of interconnected nodes to create more effective search pathways even for big datasets.

The {IVFFLAT} index balances speed and accuracy by first grouping vectors into coarse groups using a tree-based clustering technique, and then conducting a more thorough search inside the most comparable clusters.

Google AlloyDB Omni

A recent addition to AlloyDB  AI enhancements is the new “IVF” index, which increases the total number of dimensions supported per vector and dramatically reduces query time by utilising deeper integrations with AlloyDB query processing in addition to Google quantization techniques.

In actuality, distinct indices (as well as the corresponding algorithms) can perform very differently under very diverse use situations. Google  Cloud conducted a detailed benchmarking across IVF and HNSW indexes for the NeuroPace use case of locating similar cross-patient iEEGs. Recall, or the percentage of results in brute-force queries, and latency, or how quickly the query could finish, were both measured.

An analysis of the performance of these two approximate nearest neighbour (ANN) algorithms shows that IVF has high recall rates (~0.9) and a median query latency of roughly 60 ms, while HNSW indexing performed slightly worse (0.8) and was slower (median query latency of 160 ms) than IVF. To balance speed and latency, both indexes provide a variety of characteristics.

While brute force took about 14.7 seconds to find comparable iEEG data, both approaches performed noticeably better in terms of query time. The histogram of the recall and query latency for the two distinct indexing strategies in comparison to brute force. The output of a single sample iEEG query file from a test patient is displayed .

These results excite NeuroPace, since they could further the research towards effectively navigating large amounts of iEEG data. The development of algorithms that help determine the best programming settings for the RNS System may be made possible by this breakthrough. The new ScaNN index from AlloyDB may potentially help to further enhance usability and performance, and Google Cloud is eager to test it out.

Replication pipelines are vulnerable to failure in the intricate and dynamic realm of data replication. Restarting replication while minimising data integrity problems requires a number of manual processes, and determining the reason and timing of failure can be challenging.

Cloud Datastream

In circumstances like database failover or protracted network outages, you may immediately resume your data replication with little to no data loss thanks to Datastream’s new stream recovery capability.Image Credit to Google Cloud

Datastream’s unique methodology

Relational database data streaming

To load data into BigQuery,  Cloud SQL, Cloud Storage, and Spanner, Datastream reads and delivers each change, insert, update, and delete from your MySQL, PostgreSQL, AlloyDB, SQL Server, and Oracle databases. It consistently feeds every event in real time and is native to Google. It is agentless. Every month, Datastream handles more than 500 billion events.

Dependable pipes with sophisticated recovery

Costly disruptions can occur unexpectedly. Because of Datastream’s strong stream recovery, you can minimise data loss and downtime while still maintaining vital business processes and making choices based on continuous data streams.

Resolution of Schema Drift

Datastream has the capability to quickly and smoothly resolve schema drift when source schemas change. With each schema update, Datastream rotates files, generating a new file in the target bucket. A current, versioned Schema Registry can be accessed via an API request, providing access to original source data types.

Safe by design

Datastream offers a variety of private, secure connectivity options to safeguard data while it’s being transported. You may also feel secure knowing that your data is secure as it streams because it is encrypted both in transit and at rest.

Google Datastream

Read change events (inserts, updates, and deletes) from source databases and write them to a data destination are the methods used by change data capture to integrate data and enable action. Real-time analytics, database replication, and other use cases are made possible by Datastream’s support for change streams into BigQuery, Cloud SQL,  Cloud Storage, and Spanner from Oracle and MySQL databases.

Take the example of a financial institution that replicates transaction data from its operational database to BigQuery for analytics using Datastream. The primary database instance unexpectedly switches to a replica due to a hardware problem. Since the original source is no longer accessible, Datastream’s replication pipeline is malfunctioning. Replication from the failover database instance can continue thanks to stream recovery, which makes sure no transaction data is lost.

Consider an online shop that replicates consumer input using Datastream and then uses BigQuery ML for sentiment analysis. The link to the source database is broken by an extended network outage. Some of the modifications on the database server are gone by the time network connectivity is restored. Stream recovery in this scenario enables the user to rapidly resume replication from the first log position that becomes available. The merchant values having the most recent data for ongoing sentiment analysis and trend spotting, even though some feedback can be missed.

Stream recovery’s advantages

Stream recovery has several advantages, such as:

Diminished data loss

Recover from data lost as a result of unintentional deletion of log files, database instance failovers, and other situations.

Reduced downtime

Minimise downtime by promptly restoring your stream and starting up continuous CDC ingestion.

Recovery made easier

Recover your stream with ease using an easy-to-use UI.

Using Stream Recovery

You have a few possibilities with stream recovery, based on the particular failure circumstance and the availability of recent log information. You have three options for MySQL and Oracle: skip the current position and stream from the next available position, skip the current position and stream from the most recent position, or retry from the current log position. at order to ensure that no data is lost or duplicated at the destination, you can additionally specify a specific log position, such as the Log Sequence Number (LSN) or Change Sequence Number (CSN), from which you want the stream to resume.

You can instruct Datastream to resume streaming from the newly created replication slot for PostgreSQL sources by creating a new replication slot in your PostgreSQL database.

Launching a stream from a given location

There are other situations in which you might need to begin or continue a stream from a particular log location in addition to stream recovery. For instance, when historical data already exists in the destination and you would like to merge CDC from a particular point in time (when the historical data terminates), or when the source database is upgraded or transferred. In these situations, you can set a beginning position before initiating the stream using the stream recovery API.

Overview of stream recovery

A streaming stream may experience unrecoverable problems and get permanently marked as FAILED. These issues stop the stream from running and may result in lost data.

Rather than rebuilding the stream and backfilling the previous data, you can recover a permanently failed stream by instructing it to disregard the error and continue reading the ongoing events. Resetting the replication to begin reading from a new replication position will revive a stream that has permanently failed.

Start now

Generally speaking, stream recovery is now accessible for all available Datastream sources across all Google  Cloud regions using the Google  Cloud dashboard and API.

Google Cloud Databases

The folks at Loyal Guru are aware of the difficulties merchants have in fostering consumer engagement and loyalty in the face of quickly changing consumer demands. Retail clients can use Loyal Guru’s loyalty and offer personalization platform to analyse customer data and dynamically customise promotions to match specific contexts and preferences. The objective is to assist shops in building enduring relationships by making them more relevant.

Innovation is essential to us because the retail industry is constantly changing and neither should they. Loyal Guru was aware that a strong platform will enable the rapid and large-scale delivery of fully personalised experiences. For the purpose of managing Loyal Guru’s databases and infrastructure, Loyal Guru teamed with Google Cloud. AlloyDB offers the performance and agility required to continuously improve the features of Loyal Guru’s platform and assist us in seeing new patterns rather than just responding to them. These technologies work well along with other Google  Cloud databases offerings like Firestore and BigQuery.

Enabling customisation with high-performance database systems

With customised offers, promotions, and brand experiences, merchants can connect customers on a deep level with the aid of Loyal Guru. However, sophisticated data processing and analytics capabilities are needed to deliver this at scale. Retailers who want to achieve precise audience segmentation and dynamic offer changes that maximise relevance must absorb and act upon customers’ behavioural data in real time across various touchpoints.

These vital personalisation features require a scalable, high-performance database to power. Customers of Loyal Guru want immediate access to transactional records, offer specifics, customer data, and more for updating and querying during customer interactions. Database overload causes sluggish replies and unsatisfactory client experiences. For this reason, Loyal Guru decided to use Google  Cloud databases as the foundation for their platform.

A scalable, safe database approach

Firestore, BigQuery, and AlloyDB for PostgreSQL are the three potent Google  Cloud databases technologies at the centre of Loyal Guru’s architecture.

Transactional workloads using AlloyDB

Loyal Guru’s relational database, AlloyDB, provides better performance and faster analytical capabilities than they had anticipated. Loyal Guru was aware that Google Cloud databases offer remarkable scalability and speed. AlloyDB powers all of Loyal Guru’s essential transactional functions, such as offer redemptions, user profiles, and loyalty programme details. It guarantees dependable, latency-free access to this vital data for Loyal Guru APIs. It also offers a solid platform on which to begin leveraging the capabilities of Gemini and AlloyDB to develop generative AI applications that will enable Loyal Guru to offer more services.

Large traffic surges from significant shopping events on Loyal Guru frequently resulted in bottlenecks prior to the use of AlloyDB. The proliferation of data was difficult for Loyal Guru to handle, pushing the boundaries of their storage capacity. These problems have been significantly reduced by AlloyDB’s autopilot storage management feature, which automatically modifies store capacity to satisfy Loyal Guru demands without the need for human involvement. In total, Loyal Guru optimised storage from 2TB to 1.3TB, a 35% improvement, and decreased query latency by 40–50%.

Firestore for reads with a high concurrency

For high-throughput read operations on comparatively static data, such loyalty point balances and product catalogues, Loyal Guru uses Firestore in Datastore Mode. Its capacity to manage high concurrency without sacrificing speed makes it perfect for workloads that require a lot of reading. Because of this, Loyal Guru’s platform was able to continue operating with great availability and performance even when stores utilising its services sent in the highest read loads.

BigQuery for data analysis

Loyal Guru processes massive behavioural datasets through BigQuery for analytics and business intelligence in order to maximise the value from its data assets. As the data warehouse solution for Loyal Guru, Loyal Guru uses BigQuery to execute intricate queries, show patterns, and provide insights that inform strategic choices. Based on the data of Loyal Guru customers, these analytics capabilities are essential for optimising marketing tactics, product roadmaps, and Loyal Guru personalization models. Their ability to process large datasets and intricate queries quickly is further enhanced by BigQuery’s scalable and fully managed architecture, which keeps us free from infrastructure bottlenecks.

To assist us deliver a safe service with high availability, Loyal Guru uses a number of Google  Cloud databases components in addition to these three data products. We can scale Loyal Guru APIs automatically thanks to Google App Engine’s flexible environment, and Loyal Guru uses Firebase Hosting to host their progressive web app (PWA) for campaign management. For cloud-native security and traffic distribution, Loyal Guru also uses  Cloud Armour and Cloud Load Balancing.

Loyal Guru may increase the scalability and dependability of their services through cloud balancing by enhancing rotation and escalation procedures. In addition, Loyal Guru can screen outgoing or invalid attacks before they reach the GAE Service. This works in tandem with Cloud Armour, enabling us to identify certain attack attempts such as XSS, SQLi, and other prevalent malevolent activities.

Loyal Guru is still using Google Cloud

Loyal Guru has enhanced security, raised database speed, and met Loyal Guru SLA promises to their clients by optimising their infrastructure with Google Cloud’s completely managed services. Their Loyal Guru APIs now have roughly 60% less latency when connecting to Google  Cloud databases.

Loyal Guru intends to use read replicas in the future for improved availability and dependability. Additionally, Loyal Guru wants to take use of AlloyDB’s impending horizontal scalability features to facilitate future growth. To get closer to real-time operational procedures, Loyal Guru also intends to shift a portion of its BigQuery workloads to AlloyDB.

Google  Cloud Database

For Loyal Guru, Google Cloud’s robust database services have changed the game by enabling us to reliably and scalable provide shops with relevant content. These Google Cloud databases will be essential for fostering innovation and exceeding client expectations as Loyal Guru keeps pushing the envelope.