#bigtable

Allen Ginsberg, Irving Rosenthal, Peter Orlovsky, Chicago, at the time of Big Table Magazine benefit reading, featuring Peter Orlovsky, Gregory Corso and Allen, January 1959. Irving Rosenthal passed away last weekend at the age of 91.

The Big Table, it's a big idea for Ireland and The UK as a games manufacturer's association.

It’s not everyday that you’re in a room where someone is selling a vision of a better future and how to transform an industry, but that was the experience I had on Saturday. A vision of the future Talking for over 30 minutes, with barely a pause to look at reference material, industry veteran James Wallis (White Dwarf, Games Workshop, Asmodee) outlined the necessity for, and his vision of, the…

BigQuery to Bigtable

AI and real-time data integration in today’s applications have brought data analytics platforms like BigQuery into operational systems, blurring the lines between databases and analytics. Customers prefer BigQuery for effortlessly integrating many data sources, enriching data with AI and ML, and directly manipulating warehouse data with Pandas. They also say they need to make BigQuery pre-processed data available for quick retrieval in an operational system that can handle big datasets with millisecond query performance.

The EXPORT DATA to Bigtable (reverse ETL) tool is now generally accessible to bridge analytics and operational systems and provide real-time query latency. Now, anyone who can write SQL can quickly translate their BigQuery analysis into Bigtable’s highly performant data format, access it with single-digit millisecond latency, high QPS, and replicate it globally to be closer to consumers.

Three architectures and use cases that benefit from automated on-demand BigQuery to Bigtable data exports are described in this blog:

Real-time application serving 

Enriched streaming data for ML

Backloading data sketches to build real-time metrics that rely on big data.

Real-time application serving 

Bigtable enhances BigQuery for real-time applications. BigQuery’s storage format optimizes counting and aggregation OLAP queries. BigQuery BI Engine intelligently caches your most frequently used data to speed up ad-hoc analysis for real-time applications. Text lookups using BigQuery search indexes can also find rows without keys that require text filtering, including JSON.

BigQuery, a diverse analytics platform, is not geared for real-time application serving like Bigtable. Multiple columns in a row or range of rows can be difficult to access with OLAP-based storage. Bigtable excels in data storage, making it ideal for operational applications.

If your application needs any of the following, use Bigtable as a serving layer:

Row lookups with constant and predictable response times in single-digit milliseconds

High query per second (linearly scales with nodes)

Application writes with low latency

Global installations (automatatic data replication near users)

Reverse ETL reduces query latency by effortlessly moving warehouse table data to real-time architecture.

Step 1: Set up Bigtable and service table

Follow the instructions to build a Bigtable instance, a container for Bigtable data. You must choose SSD or HDD storage while creating this instance. SSD is faster and best for production, while HDD can save money if you’re simply learning Bigtable. You create your first cluster when you create an instance. This cluster must be in the same region as the BigQuery dataset you’re loading. However, you can add clusters in other regions that automatically receive data from BigQuery’s writing cluster.

Create your Bigtable table, which is the BigQuery sink in the reverse ETL process, after your instance and cluster are ready. Choose Tables in the left navigation panel and Create Table from the top of the Tables screen from the console.

Simply name the Table ID BQ_SINK and hit create on the Create a Table page. The third step was to enable BigQuery Reverse ETL construct column families.

You can also connect to your instance via CLI and run cbt createtable BQ-SINK.

Step 2: Create a BigQuery Reverse ETL application profile

Bigtable app profiles manage request handling. Consider isolating BigQuery data export in its own app profile. Allow single-cluster routing in this profile to place your data in the same region as BigQuery. It should also be low priority to avoid disrupting your main Bigtable application flow.

This gcloud command creates a Bigtable App Profile with these settings:

gcloud bigtable app-profiles create BQ_APP_PROFILE \ –project=[PROJECT_ID] \ –instance=[INSTANCE_ID]\ –description=”Profile for BigQuery Reverse ETL” \ –route-to=[CLUSTER_IN_SAME_REGION_AS_BQ_DATASET] \ –transactional-writes \ –priority=PRIORITY_LOW

After running this command, Bigtable should show it under the Application profiles area.

Step 3: SQL-export application data

Let’s analyze BigQuery and format the results for its artwork application. BigQuery public datasets’ the_met.objects table will be used. This table contains structured metadata about each Met artwork. It want to create two main art application elements:

Artist profile: A succinct, structured object with artist information for fast retrieval in our program.

Gen AI artwork description: Gemini builds a narrative description of the artwork using metadata from the table and Google Search for context.

Gemini in BigQuery setup

For your first time utilizing Gemini with BigQuery, set up the integration. Start by connecting to Vertex AI using these steps. Use the following BigQuery statement to link a dataset model object to the distant Vertex connection:

CREATE MODEL [DATASET].model_cloud_ai_gemini_pro REMOTE WITH CONNECTION us.bqml_llm_connection OPTIONS(endpoint = ‘gemini-pro’);

Step 4: GoogleSQL query Bigtable’s low-latency serving table

Its mobile app can use pre-processed artwork data. The Bigtable console’s left-hand navigation menu offers Bigtable Studio and Editor. Use this SQL to test your application’s low-latency serving query.

select _key, artist_info, generated_description[‘ml_generate_text_llm_result’] as generated_description from BQ_SINK

This Bigtable SQL statement delivers an artist profile as a single object and a produced text description field, which your application needs. This serving table can be integrated using Bigtable client libraries for C++, C#, Go, Java, HBase, Node.js, PHP, Python, and Ruby.

Enriching streaming ML data using Dataflow and Bigtable

Another prominent use case for BigQuery-Bigtable Reverse ETL is feeding ML inference models historical data like consumer purchase history from Bigtable. BigQuery’s history data can be used to build models for recommendation systems, fraud detection, and more. Knowing a customer’s shopping cart or if they viewed similar items might add context to clickstream data used in a recommendation algorithm. Identification of a fraudulent in-store credit card transaction requires more information than the current transaction, such as the prior purchase’s location, recent transaction count, or travel notice status. Bigtable lets you add historical data to Kafka or PubSub event data in real time at high throughput.

Use Bigtable’s built-in Enrichment transform with Dataflow to do this. You can build these architectures with a few lines of code!

Data sketch backloading

A data sketch is a brief summary of a data aggregation that contains all the information needed to extract a result, continue it, or integrate it with another sketch for re-aggregate. Bigtable’s conflict-free replicated data types (CRDT) help count data across a distributed system in data drawings. This is essential for real-time event stream processing, analytics, and machine learning.

Traditional distributed system aggregations are difficult to manage since speed typically compromises accuracy and vice versa. Distributed counting is efficient and accurate with Bigtable aggregate data types. These customized column families allow each server to update its local counter independently without performance-hindering locks, employing mathematical features to ensure these updates converge to the correct final value regardless of order. These aggregation data types are necessary for fraud detection, personalization, and operational reporting.

These data types seamlessly connect with BigQuery’s EXPORT DATA capability and BigQuery Data Sketches (where the same sketch type is available in Bigtable). This is important if you wish to backload your first application with previous data or update a real-time counter with updates from a source other than streaming ingestion.

Just add an aggregate column family with a command and export the data to leverage this functionality. Sample code from app:

On Bigtable, you may add real-time updates to this batch update and execute the HLL_COUNT.EXTRACT SQL function on the data sketch to estimate artist counts using BigQuery’s historical data.

What next?

Reverse ETL between BigQuery and Bigtable reduces query latency in real-time systems, but more is needed! it is working on real-time architecture data freshness with continuous queries. Continuous queries enable you to duplicate BigQuery data into Bigtable and other sources while in preview. StreamingDataFrames can be used with Python transformations in BigFrames, ready for testing.

Google Axion Processors

Arm Based CPU

At Google, they continuously push the limits of computers to investigate what can be done for big problems like global video distribution, information retrieval, and, of course, generative AI. Rethinking systems design in close cooperation with service developers is necessary to achieve this. Their large investment in bespoke silicon is the outcome of this rethinking. Google is excited to present the most recent iteration of this effort today: Google Axion Processors, Google’s first specially made Arm-based CPUs intended for data centers. Later this year, Axion which offers performance and energy efficiency that leads the industry will be made accessible to Google Cloud users.

Axion is only the most recent model of customised silicon from Google. Google’s first Video Coding Unit (VCU) increased video transcoding efficiency by 33x in 2018. Five generations of Tensor Processing Units have been launched since 2015. Google invested in “system on a chip” (SoC) designs and released the first of three generations of mobile Tensor processors in 2021 to boost bespoke computing.

General-purpose computing is and will continue to be a vital component of their customers’ workloads, even if Google investments in compute accelerators have revolutionised their capabilities. Extensive computation power is needed for analytics, information retrieval, and machine learning training and providing. The pace at which CPUs are being improved has slowed lately, which has affected customers and users who want to satisfy sustainability objectives, save infrastructure costs, and maximise performance. According to Amdahl’s Law, unless Google make the corresponding expenditures to stay up, general purpose compute will dominate the cost and restrict the capabilities of their infrastructure as accelerators continue to advance.

Google BigTable

In order to deliver instances with up to 30% better performance than the fastest general-purpose Arm-based instances currently available in the cloud, as well as up to 50% better performance and up to 60% better energy-efficiency than comparable current-generation x86-based instances, Axion processors combine Google’s silicon expertise with Arm’s highest performing CPU cores. For this reason, on current generation Arm-based servers, Google have already begun implementing Google services such as BigTable, Google Spanner, BigQuery, Blobstore, Pub/Sub, Google Earth Engine, and the YouTube Ads platform. Google also have plans to deploy and expand these services, along with others, on Axion shortly.

Superior effectiveness and performance, supported by Titanium

Axion processors, which are constructed around the Arm Neoverse V2 CPU, offer massive performance gains for a variety of general-purpose workloads, including media processing, web and app servers, containerised microservices, open-source databases, in-memory caches, data analytics engines, and more.

Titanium, a system of specially designed silicon microcontrollers and tiered scale-out offloads, provides the foundation for Axion. Platform functions like networking and security are handled by titanium offloads, giving Axion processors more capacity and enhanced performance for workloads from customers. Titanium also transfers I/O processing for storage to Hyperdisk, Google’s recently launched block storage solution that can be dynamically supplied in real time and decouples performance from instance size.

Titanium

A system of specially designed silicon security microcontrollers and tiered scale-out offloads that enhances the dependability, security, life cycle management, and performance of infrastructure.

Google-powered Titanium

Titanium is a free platform that supports Hyperdisk block storage, networking, the newest compute instance types (C3, A3, and H3), and more on Google Cloud.

Included in the system are:

Titan security microcontrollers are specially designed to provide Google Cloud’s infrastructure a hardware root of trust.

Titanium adaptor: specialised offload card that offers hardware acceleration for virtualization services; frees up resources for workloads by offloading processing from the host CPU

Titanium offload processors (TOPs) are silicon devices placed across the data centre that are used as a scalable and adaptable method of offloading network and I/O operations from the host CPU.

Enhanced functionality of the infrastructure

Titanium offloads computation from the host hardware to provide additional compute and memory resources for your applications.

Hyperdisk Extreme block storage allows for up to 500k IOPS per instance, which is the greatest among top hyperscalers.

200 Gbps or more of network bandwidth

Full line rate network encryption that offers security without compromising speed

Consistent performance comparable to bare metal for the most delicate workloads

Smooth management of the infrastructure life cycle

Infrastructure changes are made easier by Titanium’s modular hardware and software, which also provide offloading capabilities and workload continuity and security.

Advanced maintenance controls for the most critical workloads and seamless upgrades for the majority of workloads

It is possible to start remote infrastructure upgrades from any location.

The Titanium adaptor’s dedicated domains for networking and storage enable for the autonomous upkeep and upgrades of individual services, keeping them apart from the host’s burden.

“Building on Google’s high-performance Arm Neoverse V2 platform, Google’s announcement of the new Axion CPU represents a significant milestone in the delivery of custom silicon optimised for Google’s infrastructure.” The greatest experience for consumers using Arm is guaranteed by decades of ecosystem investment, Google’s continuous innovation, and its contributions to open-source software.”

Customers want to accomplish their sustainability objectives and operate more effectively, not only perform better. In comparison to five years ago, Google Cloud data centres are now 1.5 times more efficient than the industry average and provide 3 times more processing power with the same amount of electrical power. Google lofty objectives include running their campuses, offices, and data centres entirely on carbon-free energy sources around-the-clock and providing resources to assist with carbon emission reporting. Customers may optimise for even greater energy efficiency using Axion processors.

Axion: Interoperability and compatibility with out-of-the-box applications

Additionally, Google has a long history of supporting the Arm ecosystem. They worked closely with Arm and industry partners to optimize Android, Kubernetes, Tensorflow, and the Go language for the Arm architecture. Google also constructed and made them open-sourced.

Armv9 architecture

The standard Armv9 architecture and instruction set serve as the foundation for Axion. Google have made contributions to the SystemReady Virtual Environment (VE) standard, which is designed to ensure that Arm-based servers and virtual machines (VMs) can run common operating systems and software packages. This standard makes it easier for customers to deploy Arm workloads on Google Cloud with minimal to no code rewrites. Google is gaining access to an ecosystem of tens of thousands of cloud users that are already using Arm-native software from hundreds of ISVs and open-source projects and deploying workloads thanks to Google’s partnership.

Axion will be available to users across a variety of Google Cloud services, such as Cloud Batch, Dataproc, Dataflow, Google Compute Engine, and Google Kubernetes Engine. The Google Cloud Marketplace now offers Arm-compatible apps and solutions, and Google just released preview support for the Migrate to Virtual Machines service, which allows you to migrate Arm-based instances.