#apacheiceberg

Large Lake Storage

Built open, high-performance, enterprise Big Lake storage Lakehouses iceberg native

Businesses may use Apache Iceberg to develop open, high-performance, enterprise-grade data lakehouses on Google Cloud with recent Big Lake storage engine improvements. Customers no longer have to choose between completely managed, enterprise-grade storage management and open formats like Apache Iceberg.

Businesses want adaptive, open, and interoperable architectures that let several engines work on a single copy of data while data management is revolutionised. Apache Iceberg is a popular open table style. The latest Big Lake storage development offers Apache Iceberg access to Google's infrastructure, enabling open data lakehouses.

Major advances include:

BigLake Metastore is normally available: BigLake Metastore, formerly BigQuery, is now public. This completely managed, serverless, and scalable solution simplifies runtime metadata maintenance and operations for BigQuery and other Iceberg-compatible engines. Use of Google's global metadata management infrastructure reduces the need to control proprietary metastore implementation. BigLake Metastore is necessary for open interoperability.

Iceberg REST Catalogue API Preview Introduction: To complement the GA Custom Iceberg Catalogue, the Iceberg REST Catalogue (Preview) provides a standard REST interface for interoperability. Users, including Spark users, can use the BigLake metastore as a serverless Iceberg catalogue. The Custom Iceberg Catalogue lets Spark and other open-source engines connect with Apache Iceberg and BigQuery BigLake tables.

Google Cloud is simplifying lakehouse upkeep using Apache Iceberg and Google Cloud Storage management. Cloud Storage features like auto-class tiering, encryption, and automatic table maintenance including compaction and trash collection are supported. This enhances Iceberg data management in Cloud Storage.

BigQuery usually has Apache Iceberg BigLake tables: These publicly available tables combine BigQuery's scalable, real-time metadata with Iceberg formats' transparency. This enables BigQuery's Write API's high-throughput streaming ingestion and zero-latency reads at tens of GiB/second. It also has automatic table management (compaction, garbage collection), native Vertex AI interface, auto-reclustering speed improvements, and future fine-grained DML and multi-table transactions (coming soon in preview). These tables maintain Iceberg's openness while providing controlled, enterprise-ready functionality. BigLake automatically creates and registers an Apache Iceberg V2 metadata snapshot in its metastore. This snapshot updates automatically after edits.

BigLake natively supports Dataplex Universal Catalogue for AI-Powered Governance. This interface provides consistent and fine-grained access restrictions to apply Dataplex governance standards across engines. Direct Cloud Storage access supports table-level access control, whereas BigQuery can use Storage API connectors for open-source engines for finer control. Dataplex integration improves BigQuery and BigLake Iceberg table governance with search, discovery, profiling, data quality checks, and end-to-end data lineage. Dataplex simplifies data discovery with AI-generated insights and semantic search. End-to-end governance benefits are automatic and don't require registration.

The BigLake metastore enables interoperability with BigQuery, AlloyDB (preview), Spark, and Flink. This increased compatibility allows AlloyDB users to easily consume analytical BigLake tables for Apache Iceberg from within AlloyDB (Preview). PostgreSQL users can link real-time AlloyDB transactional data with rich analytical data for operational and AI-driven use cases.

CME Group Executive Director Zenul Pomal noted, “We needed teams throughout the company to access data in a consistent and secure way – regardless of where it stored or what technologies they were using.” They used Google's BigLake. BigLake from Google was clear. The uniform layer for accessing data and a fully managed experience with enterprise capabilities via BigQuery are available without moving or duplicating data, whether the data is in traditional tables or open table formats like Apache Iceberg. Metadata quality is critical as it explores gen AI applications. BigLake Metastore and Data Catalogue help us preserve high-quality metadata.

At Google Cloud Next '25, Google Cloud announced support for change data capture, multi-statement transactions, and fine-grained DML in the coming months.

IBM Db2

Designed to handle mission-critical workloads worldwide.

What is IBM Db2?

IBM Db2 is a cloud-native database designed to support AI applications at scale, real-time analytics, and low-latency transactions. It offers database managers, corporate architects, and developers a single engine that is based on decades of innovation in data security, governance, scalability, and availability.

- Advertisement -

When moving to hybrid deployments, create the next generation of mission-critical apps that are available 24/7 and have no downtime across all clouds.

Support for all contemporary data formats, workloads, and programming languages will streamline development.

Support for open formats, including Apache Iceberg, allows teams to safely communicate data and information, facilitating quicker decision-making.

Utilize IBM Watsonx integration for generative artificial intelligence (AI) and integrated machine learning (ML) capabilities to implement AI at scale.

Use cases

Power next-gen AI assistants

Provide scalable, safe, and accessible data so that developers may create AI-powered assistants and apps.

Build new cloud-native apps for your business

Create cloud-native applications with low latency transactions, flexible scalability, high concurrency, and security that work on any cloud. Amazon Relational Database Service (RDS) now offers it.

Modernize mission-critical web and mobile apps

Utilize Db2 like-for-like compatibility in the cloud to modernize your vital apps for hybrid cloud deployments. Currently accessible via Amazon RDS.

Power real-time operational analytics and insights

Run in-memory processing, in-database analytics, business intelligence, and dashboards in real-time while continuously ingesting data.

Data sharing

With support for Apache Iceberg open table format, governance, and lineage, you can share and access all AI data from a single point of entry.

In-database machine learning

With SQL, Python, and R, you can create, train, assess, and implement machine learning models from inside the database engine without ever transferring your data.

Built for all your workloads

IBM Db2 Database

Db2 is the database designed to handle transactions of any size or complexity. Currently accessible via Amazon RDS.

IBM Db2 Warehouse

You can safely and economically conduct mission-critical analytical workloads on all kinds of data with IBM Db2 Warehouse. Watsonx.data integration allows you to grow AI workloads anywhere.

IBM Db2 Big SQL

IBM Db2 Big SQL is a high-performance, massively parallel SQL engine with sophisticated multimodal and multicloud features that lets you query data across Hadoop and cloud data lakes.

Deployment options

You require an on-premises, hybrid, or cloud database. Use Db2 to create a centralized business data platform that operates anywhere.

Cloud-managed service

Install Db2 on Amazon Web Services (AWS) and IBM Cloud as a fully managed service with SLA support, including RDS. Benefit from the cloud’s consumption-based charging, on-demand scalability, and ongoing improvements.

Cloud-managed container

Launch Db2 as a cloud container:integrated Db2 into your cloud solution and managed Red Hat OpenShift or Kubernetes services on AWS and Microsoft Azure.

Self-managed infrastructure or IaaS

Take control of your Db2 deployment by installing it as a conventional configuration on top of cloud-based infrastructure-as-a-service or on-premises infrastructure.

IBM Db2 Updates With AI-Powered Database Helper

Enterprise data is developing at an astonishing rate, and companies are having to deal with ever-more complicated data environments. Their database systems are under more strain than ever as a result of this. Version 12.1 of IBM’s renowned Db2 database, which is scheduled for general availability this week, attempts to address these demands. The latest version redefines database administration by embracing AI capabilities and building on Db2’s lengthy heritage.

The difficulties encountered by database administrators who must maintain performance, security, and uptime while managing massive (and quickly expanding) data quantities are covered in Db2 12.1. A crucial component of their strategy is IBM Watsonx’s generative AI-driven Database Assistant, which offers real-time monitoring, intelligent troubleshooting, and immediate replies.

Introducing The AI-Powered Database Assistant

By fixing problems instantly and averting interruptions, the new Database Assistant is intended to minimize downtime. Even for complicated queries, DBAs may communicate with the system in normal language to get prompt responses without consulting manuals.

The Database Assistant serves as a virtual coach in addition to its troubleshooting skills, speeding up DBA onboarding by offering solutions customized for each Db2 instance. This lowers training expenses and time. By enabling DBAs to address problems promptly and proactively, the database assistant should free them up to concentrate on strategic initiatives that improve the productivity and competitiveness of the company.

IBM Db2 Community Edition

Now available

Db2 12.1

No costs. No adware or credit card. Simply download a single, fully functional Db2 Community License, which you are free to use for as long as you wish.

What you can do when you download Db2

Install on a desktop or laptop and use almost anywhere. Join an active user community to discover events, code samples, and education, and test prototypes in a real-world setting by deploying them in a data center.

Limits of the Community License

Community license restrictions include an 8 GB memory limit and a 4 core constraint.

BigQuery tables

Optimized storage for the open lakehouse using BigQuery tables for Apache Iceberg. BigQuery native tables have been supporting enterprise-level data management features including streaming ingestion, ACID transactions, and automated storage optimizations for a number of years. Open-source file formats like Apache Parquet and table formats like Apache Iceberg are used by many BigQuery clients to store data in data lakes.

Google Cloud introduced BigLake tables in 2022 so that users may take advantage of BigQuery’s security and speed while keeping a single copy of their data. BigQuery clients must manually arrange data maintenance and conduct data changes using external query engines since BigLake tables are presently read-only. The “small files problem” during ingestion presents another difficulty. Table writes must be micro-batched due to cloud object storage’ inability to enable appends, necessitating trade-offs between data integrity and efficiency.

Google Cloud provides the first look at BigQuery tables for Apache Iceberg, a fully managed storage engine from BigQuery that works with Apache Iceberg and offers capabilities like clustering, high-throughput streaming ingestion, and autonomous storage optimizations. It provide the same feature set and user experience as BigQuery native tables, but they store data in customer-owned cloud storage buckets using the Apache Iceberg format. Google’s are bringing ten years of BigQuery developments to the lakehouse using BigQuery tables for Apache Iceberg.Image Credit To Google Cloud

BigQuery’s Write API allows for high-throughput streaming ingestion from open-source engines like Apache Spark, and BigQuery tables for Apache Iceberg may be written from BigQuery using the GoogleSQL data manipulation language (DML). This is an example of how to use clustering to build a table:

CREATE TABLE mydataset.taxi_trips CLUSTER BY vendor_id, pickup_datetime WITH CONNECTION us.myconnection OPTIONS ( storage_uri=’gs://mybucket/taxi_trips’, table_format=’ICEBERG’, file_format=’PARQUET’ ) AS SELECT * FROM bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2020;

Fully managed enterprise storage for the lakehouse

Drawbacks of BigQuery tables for Apache Iceberg

The drawbacks of open-source table formats are addressed by BigQuery tables for Apache Iceberg. BigQuery handles table-maintenance duties automatically without requiring client labor when using BigQuery tables for Apache Iceberg. BigQuery automatically re-clusters data, collects junk from files, and combines smaller files into appropriate file sizes to keep the table optimized.

For example, the size of the table is used to adaptively decide the ideal file sizes. BigQuery tables for Apache Iceberg take use of more than ten years of experience in successfully and economically managing automatic storage optimization for BigQuery native tables. OPTIMIZE and VACUUM do not need human execution.

BigQuery tables for Apache Iceberg use Vortex, an exabyte-scale structured storage system that drives the BigQuery storage write API, to provide high-throughput streaming ingestion. Recently ingested tuples are persistently stored in a row-oriented manner in BigQuery tables for Apache Iceberg, which regularly convert them to Parquet. The open-source Spark and Flink BigQuery connections provide parallel readings and high-throughput ingestion. You may avoid maintaining custom infrastructure by using Pub/Sub and Datastream to feed data into BigQuery tables for Apache Iceberg.

Advantages of using BigQuery tables for Apache Iceberg

Table metadata is stored in BigQuery’s scalable metadata management system for Apache Iceberg tables. BigQuery handles metadata via distributed query processing and data management strategies, and it saves fine-grained information. since of this, BigQuery tables for Apache Iceberg may have a greater rate of modifications than table formats since they are not limited by the need to commit the information to object storage. The table information is tamper-proof and has a trustworthy audit history since authors are unable to directly alter the transaction log.

While expanding support for governance policy management, data quality, and end-to-end lineage via Dataplex, BigQuery tables for Apache Iceberg still support the fine-grained security rules imposed by the storage APIs.Image Credit To Google Cloud

BigQuery tables for Apache Iceberg are used to export metadata into cloud storage Iceberg snapshots. BigQuery metastore, a serverless runtime metadata service that was revealed earlier this year, will shortly register the link to the most recent exported information. Any engine that can comprehend Iceberg may query the data straight from Cloud Storage with to Iceberg metadata outputs.

Find out more

Clients such as HCA Healthcare, one of the biggest healthcare organizations globally, recognize the benefits of using BigQuery tables for Apache Iceberg as their BigQuery storage layer that is compatible with Apache Iceberg, opening up new lakehouse use-cases. All Google Cloud regions now provide a preview of the BigQuery tables for Apache Iceberg.

Can other tools query data stored in BigQuery tables for Apache Iceberg?

Yes, metadata is exported from Apache Iceberg BigQuery tables into cloud storage Iceberg snapshots. This promotes interoperability within the open data ecosystem by enabling any engine that can comprehend the Iceberg format to query the data straight from Cloud Storage.

How secure are BigQuery tables for Apache Iceberg?

The strong security features of BigQuery, including as fine-grained security controls enforced by storage APIs, are carried over into BigQuery tables for Apache Iceberg. Additionally, end-to-end lineage tracking, data quality control, and extra governance policy management layers are made possible via interaction with Dataplex.