#azureblobstorage

Microsoft Azure Blob Storage

Scalable, extremely safe, and reasonably priced cloud object storage

Incredibly safe and scalable object storage for high-performance computing, archiving, data lakes, cloud-native workloads, and machine learning.

What is Azure Blob Storage?

Microsoft’s cloud-based object storage solution is called Blob Storage. Massive volumes of unstructured data are best stored in blob storage. Text and binary data are examples of unstructured data, which deviates from a certain data model or specification.

Scalable storage and retrieval of unstructured data

Azure Blob Storage offers storage for developing robust cloud-native and mobile apps, as well as assistance in creating data lakes for your analytics requirements. For your long-term data, use tiered storage to minimize expenses, and scale up flexibly for tasks including high-performance computing and machine learning.

Construct robust cloud-native apps

Azure Blob Storage was designed from the ground up to meet the demands of cloud-native, online, and mobile application developers in terms of availability, security, and scale. For serverless systems like Azure Functions, use it as a foundation. Blob storage is the only cloud storage solution that provides a premium, SSD-based object storage layer for low-latency and interactive applications, and it supports the most widely used development frameworks, such as Java,.NET, Python, and Node.js.

Save petabytes of data in an economical manner

Store enormous volumes of rarely viewed or infrequently accessed data in an economical manner with automated lifecycle management and numerous storage layers. Azure Blob Storage can take the place of your tape archives, and you won’t have to worry about switching between hardware generations.

Construct robust data lakes

One of the most affordable and scalable data lake options for big data analytics is Azure Data Lake Storage. It helps you accelerate your time to insight by fusing the strength of a high-performance file system with enormous scalability and economy. Data Lake Storage is tailored for analytics workloads and expands the possibilities of Azure Blob Storage.

Scale out for billions of IoT devices or scale up for HPC

Azure Blob Storage offers the volume required to enable storage for billions of data points coming in from IoT endpoints while also satisfying the rigorous, high-throughput needs of HPC applications.

Features

Scalable, robust, and accessible

With geo-replication and the capacity to scale as needed, the durability is designed to be sixteen nines.

Safe and sound

Role-based access control (RBAC), Microsoft Entra ID (previously Azure Active Directory) authentication, sophisticated threat protection, and encryption at rest.

Data lake-optimized

Multi-protocol access and file namespace facilitate analytics workloads for data insights.

Complete data administration

Immutable (WORM) storage, policy-based access control, and end-to-end lifecycle management.

Integrated security and conformance

Complete security and conformance, integrated

Every year, Microsoft spends about $1 billion on cybersecurity research and development.

Over 3,500 security professionals who are committed to data security and privacy work for it.

Azure boasts one of the biggest portfolios of compliance certifications in the sector.

Azure Blob storage cost

Documents, films, images, backups, and other unstructured text or binary data can all be streamed and stored using block blob storage.

The most recent features are accessible through blob storage accounts, however they do not allow page blobs, files, queues, or tables. For the majority of users, general-purpose v2 storage accounts are advised.

Block blob storage’s overall cost is determined by:

Monthly amount of data kept.

Number and kinds of activities carried out, as well as any expenses related to data transfer.

The option for data redundancy was chosen.

Adaptable costs with reserved alternatives to satisfy your needs for cloud storage

Depending on how frequently you anticipate accessing the data, you can select from its storage tiers. Keep regularly accessed data in Hot, seldom accessed data in Cool and Cold, performance-sensitive data in Premium, and rarely accessed data in Archive. Save a lot of money by setting up storage space.

To continue building with the same free features after your credit, switch to pay as you go. Only make a payment if your monthly usage exceeds your free amounts.

You will continue to receive over fifty-five services at no cost after a year, and you will only be charged for the services you utilize above your monthly allotment.

What is BigQuery Omni?

BigQuery Omni is a multi-cloud data analytics solution that lets you use BigLake tables to perform BigQuery analytics on data kept in Azure Blob Storage or Amazon Simple Storage Service (Amazon S3). It offers a single interface for analyzing data from several public clouds, removing the need to relocate data and allowing you to learn from your data no matter where it is stored.

Many businesses use several public clouds to store their data. Due to the difficulty of gaining insights from all of the data, this data frequently becomes siloed. To evaluate the data, you need a multi-cloud data tool that is quick, affordable, and doesn’t contribute to the costs of decentralized data governance. With a single interface, BigQuery Omni helps us to lower these frictions.

Connecting to Amazon S3 or Blob Storage is a must for performing BigQuery analytics on your external data. You would need to establish a BigLake table that refers data from Blob Storage or Amazon S3 in order to query external data.

Additionally, data can be moved between clouds using cross-cloud transfer, or it can be queried between clouds using cross-cloud joins. One cross-cloud analytics option that BigQuery Omni provides is the freedom to replicate data as needed and the ability to examine data where it resides.

Google BigQuery Omni

Operating hundreds of separate applications across multiple platforms is not unusual in today’s data-centric enterprises. The enormous amount of logs generated by these applications poses a serious problem for log analytics. Furthermore, accuracy and retrieval are made more difficult by the widespread use of multi-cloud solutions, since the dispersed nature of the logs may make it more difficult to derive valuable insights.

In contrast to a traditional strategy, BigQuery Omni was created to help solve this problem and lower overall expenditures. We’ll go over the specifics in this blog post.

Log analysis includes a number of steps, including:

Gathering log data: gathers log data from the applications and/or infrastructure of the enterprise. A popular method for gathering this data is to save it in an object storage program like Google Cloud Storage in JSONL file format. It can be prohibitively expensive to move raw log data between clouds in a multi-cloud setup.

Normalization of log data: Various infrastructures and applications produce distinct JSONL files. The fields in each file are specific to the program or infrastructure that produced it. To make data analysis easier, these disparate fields are combined into a single set, which enables data analysts to do thorough and effective studies throughout the environment.

Indexing and storage: To lower storage and query expenses and improve query performance, normalized data should be stored effectively. Logs are often stored in a compressed columnar file format, such as Parquet.

Querying and visualization: Enable enterprises to run analytics queries to find known threads, abnormalities, or anti-patterns in the log data through querying and visualization.

Data lifecycle: While storage expenses persist, the usefulness of log data declines with age. A data lifecycle procedure must be established in order to maximize costs. Archiving logs after a month (it is rare to query log data older than a month) and deleting them after a year are usual practices. This strategy ensures that crucial data is always available while efficiently controlling storage expenses.

A common Architecture

Many businesses use the following architecture to apply log analysis in a multi-cloud setting:Image credit to Google Cloud

This architecture has advantages and disadvantages.

On the Plus side:

Data lifecycle: By utilizing pre-existing functionality from object storage solutions, data lifecycle management may be implemented really easily. For instance, you can provide the following data lifecycle policy in Cloud Storage: You can use the following policies: (a) delete any item older than a week, which will remove any JSONL files that were available during the Collection process; (b) archive any object older than a month, which will also remove your Parquet files; and (c) delete any object older than a year, which will also remove your Parquet files.

Minimal egress costs: By storing the data locally, you can avoid transmitting large amounts of unprocessed information between cloud providers.

From the negative perspective:

Normalization of log data: You will code and manage an Apache Spark workload for every application with logs you gather. In a time when (a) engineers are in short supply and (b) the use of microservices is expanding quickly, it is wise to steer clear of this.

Querying: You can’t do as much analysis and visualization if you spread your data over several cloud providers.

Querying: Using WHERE clauses to prevent partitions with archived files requires human error and is not a simple solution for excluding archived files created earlier in the data lifetime. Managing the table’s manifest by adding and removing divisions as necessary is one way to work with Iceberg Table. However, it is difficult to play with the Iceberg Table manifest by hand, and using a third-party solution only makes things more expensive.

A better way to address all of these issues would be to use BigQuery Omni, which is shown in the architecture below.

This method’s primary advantage is the removal of several Spark workloads and the need for software engineers to code and maintain them. Having a single product (BigQuery) manage the entire process, aside from storage and visualization, is another advantage of this system. You gain from cost savings as well. Below, we’ll go into more detail about each of these points.

An streamlined procedure for normalizing

The ability of BigQuery to automatically determine the schema of JSONL files and generate an external table pointing to them is a valuable feature. This function is especially helpful when working with multiple log schema formats. The JSONL content of any application can be accessed by defining a simple CREATE TABLE declaration.

Once there, you may program BigQuery to export the JSONL external table into compressed Parquet files in Hive format that are divided into hourly segments. An EXPORT DATA statement that may be programmed to execute once every hour is shown in the query below. This query’s SELECT statement only records the log data that was ingested during the previous hour and transforms it into a Parquet file with columns that have been normalized.

DECLARE hour_ago_rounded_string STRING; DECLARE hour_ago_rounded_timestamp DEFAULT DATETIME_TRUNC(TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR), HOUR);

SET (hour_ago_rounded_string) = ( SELECT AS STRUCT FORMAT_TIMESTAMP(“%Y-%m-%dT%H:00:00Z”, hour_ago_rounded_timestamp, “UTC”) );

EXPORT DATA OPTIONS ( uri = CONCAT(‘[MY_BUCKET_FOR_PARQUET_FILES]/ingested_date=’, hour_ago_rounded_string, ‘/logs-*.parquet’), format = ‘PARQUET’, compression = ‘GZIP’, overwrite = true) AS ( SELECT [MY_NORMILIZED_FIELDS] EXCEPT(ingested_date) FROM [MY_JSONL_EXTERNAL_TABLE] as jsonl_table WHERE DATETIME_TRUNC(jsonl_table.timestamp, HOUR) = hour_ago_rounded_timestamp );

A uniform querying procedure for all cloud service providers

While using the same data warehouse platform across several cloud providers already improves querying, BigQuery Omni’s ability to perform cross-cloud joins is revolutionary for Log Analytics. Combining log data from many cloud providers was difficult prior to BigQuery Omni. Sending the raw data to a single master cloud provider results in large egress expenses due to the volume of data; yet, pre-processing and filtering it limits your capacity to do analytics on it. Cross-cloud joins allow you to execute a single query across several clouds and examine the outcomes.

Reduces TCO

This architecture’s ability to lower total cost of ownership (TCO) is its last and most significant advantage. There are 3 ways to measure this:

Decreased engineering resources: Apache Spark is eliminated from this procedure for two reasons. The first is that Spark code can be worked on and maintained without a software developer. By employing standard SQL queries, the log analytics team can complete the deployment process more quickly. The shared responsibility concept of BigQuery and BigQuery Omni, which are PaaS, is extended to data in AWS and Azure.

Lower compute resources: The most economical environment might not always be provided by Apache Spark. The application itself, the Apache Spark platform, and the virtual machine (VM) make up an Apache Spark solution. In comparison to Apache Spark, BigQuery uses slots (virtual CPUs, not virtual machines) and an export query that is transformed into C-compiled code during the export process can lead to quicker performance for this particular operation.

Lower egress expenses: BigQuery Omni eliminates the need to transfer raw data between cloud providers in order to get a consolidated view of the data by processing data in-situ and egressing only results through cross-cloud joins.

What is the best way to use BigQuery in this setting?

BigQuery has two compute pricing models for query execution:

On-demand pricing (per TiB): This pricing model charges you according to the quantity of bytes each query processes, but it does not charge you for the first 1 TiB of query data handled each month. Using this technique is not advised because log analytics tasks use a lot of data.

Capacity pricing (per slot-hour): Under this pricing model, you are billed for the amount of computing power that is utilized to execute queries over time, expressed in slots (virtual CPUs). This model utilizes editions of BigQuery. Slot commitments, which are dedicated capacity always available for your workloads, are less expensive than on-demand, and you can use the BigQuery autoscaler.

In order to conduct an empirical test, Google assigned 100 slots (baseline 0, maximum slots 100) to a project that aimed to export log JSONL data into a compressed Parquet format. This configuration allowed BigQuery to process 1PB of data daily without using up all 100 slots.

In order to enable the TCO reduction of Log Analytics workloads in a multi-cloud context, it proposed an architecture in this blog post that substitutes SQL queries running on BigQuery Omni for Apache Spark applications. Your particular data environment may benefit from this method’s ability to minimize overall DevOps complexity while lowering engineering, computation, and egress expenses.

BigQuery Omni pricing

Please refer to BigQuery Omni pricing for details on price and time-limited promotions.

The public preview of Azure Storage Actions, a fully managed platform that enables you to automate data management tasks for Azure Blob Storage and Azure Data Lake Storage, is being announced with great excitement.

Data management is becoming increasingly difficult for organizations as their data estates grow exponentially. For businesses to fully utilize their data assets, adhere to compliance requirements, cut expenses, and protect sensitive data, effective data management is crucial. Increasing resource investments to manage data at the same pace as the increase in data volumes is unsustainable, and the tools and methods available today to manage massive data assets are laborious. Customers using storage need an effective system to manage billions of objects across thousands of datasets, consistently and holistically, across all regions.

With a quicker time to value, Azure Storage Actions revolutionizes how you manage massive data assets in your object storage and data lakes. Without requiring any resource provisioning or management, its serverless architecture offers a dependable platform that grows to meet your data management requirements. Without the need for programming knowledge, you can define the conditional logic for processing objects using a no-code experience.

With a few clicks, the tasks you create can safely operate on several datasets with comparable requirements. By providing views that provide an overview of results at a glance, in addition to filters and drilldowns for more detail, monitoring overhead is reduced. For Azure Blob Storage and Azure Data Lake Storage, this release supports cost optimization, data protection, rehydration from archive, tagging, and a number of additional use cases.

The operation of Azure Storage Actions

You can quickly create, verify, and implement data management tasks by using Azure Storage Actions. These jobs can be set up to run on demand or according to a schedule.

You can create a condition that specifies the blobs and operations you want to perform on using the Azure portal interface. Without taking any action, you can safely verify the condition against your production data using the integrated validation experience, which displays the blobs that meet the condition and the operations that would be performed on them if the task were executed.

Any storage account within the same Microsoft Entra ID tenant can have tasks assigned to it to execute. When necessary, the service automatically sets up, scales, and optimizes the resources for either ongoing or one-time task execution. Aggregate metrics and dashboards provide a visual summary of operations and allow you to drill down into more in-depth reports with minimal intervention when and where needed.

REST APIs and the Azure SDK are additional programmatic means of controlling Azure Storage Actions. PowerShell, Azure Resource Manager (ARM) templates, and Azure Command-Line Interface (CLI) are all supported.

Supported operations: This release supports all built-in operations on Azure Blob Storage and Azure Data Lake Storage, such as adjusting tiers, controlling blob expiry, deleting or undeleting blobs, and setting time-based retention. Additional operations will be supported by the feature in future releases.

The rationale behind utilizing Azure Storage Actions

Utilizing Azure Storage Actions to automate your data management processes has the following benefits:

Reduces the amount of work needed to automate routine data management tasks, which increases productivity.

Reduces the overhead associated with managing or provisioning infrastructure.

Offers confidence through the no-code interface’s integrated validation experience for error-free application to your production data.

Makes reuse easier by allowing you to create a task once and quickly deploy it to any storage account.

Promotes the consistent application of metadata and blob tags in conditions and operations.

Use cases examples

Thousands of data sets with a variety of object types that are needed for different kinds of processing can be found in large data lakes. Individual objects within a blob container may need different tiering transitions, distinct labels for tagging, distinct retention or expiry periods, and other requirements based on their attributes. Tasks that scan billions of blobs, analyze each one based on dozens of properties (file extension, naming pattern, index tags, blob metadata, or system properties like creation time, content type, blob tier, and more), and decide how to handle each one can be defined with Azure Storage Actions.

This method can simplify a wide range of recurrent or one-time use cases, such as:

Depending on object tags, retention, and expiration: One of Azure’s international clients in the financial services industry uses Azure Blob Storage to ingest call recordings from customer support agents. These recordings contain blob tags that indicate when an order was placed for trading, when an account was updated, and other information. Depending on the type of call, these recordings have different retention requirements. Now, they can use Azure Storage Actions to create a task that uses a combination of blob tags and creation time to automatically manage the retention and expiry durations of ingested recordings.

Flexible data protection in datasets: Although blob versioning and snapshots are used by a prominent travel services company customer, the thousands of datasets in the storage account have varying data protection needs. Certain datasets must have a strict version history maintained, but others do not require this kind of security. It is prohibitively expensive to preserve the full blob version and snapshot history for every dataset in their storage account. They can now flexibly manage the appropriate retention and lifecycle of versions and snapshots for their datasets by using tags and metadata with Azure Storage Actions.

Cost optimization based on file types and naming conventions: A lot of Azure Storage users also need to control blob tiering, expiration, and retention according to file types, naming conventions, or path prefixes. To process the objects as desired, these attributes can be combined with blob properties like size, creation time, last modified or last accessed times, access tier, version counts, and more.

Processing blobs on demand at scale: Azure Storage Actions can be used for processing billions of objects on demand in addition to continuous data management tasks. For example, you can set up tasks to clean up redundant and outdated datasets, reset tags on a portion of a dataset when an analytic pipeline needs to be restarted, or initialize blob tags for a new or modified process. You can also define tasks to rehydrate large datasets from the archive tier.

How to begin using Azure Storage Actions

We cordially request that you check out Azure Storage Actions for object storage data management. During the preview, you can test the feature for free and only pay for the transactions that are initiated on your storage account. Before the feature’s widespread release, pricing details will be released. Please visit the feature support page to view the list of supported regions. Start by using the quickstart guide to quickly create and complete your first data management task. Please refer to the documentation for further information.