#meachinelearning

Palo Alto Networks’ suggestions on database conversion from Cassandra to Bigtable

In this blog post, we look at how Palo Alto Networks, a leading cybersecurity company worldwide, solved its scalability and performance issues by switching from Apache Cassandra to Bigtable, Google Cloud’s enterprise-grade, low-latency NoSQL database service. This allowed them to achieve 5x lower latency and cut their total cost of ownership in half. Please continue reading if you want to find out how they approached this migration.

Bigtable has been supporting both internal systems and external clients at Google. Google Cloud wants to tackle the most challenging use cases in the business and reach more developers with Bigtable. Significant progress has been made in that approach with recent Bigtable features:

High-performance, workload-isolated, on-demand analytical processing of transactional data is made possible by the innovative Bigtable Data Boost technology. Without interfering with your operational workloads, it enables you to run queries, ETL tasks, and train machine learning models directly and as often as necessary on your transactional data.

Several teams can safely use the same tables and exchange data from your databases thanks to the authorized views feature, which promotes cooperation and effective data use.

Distributed counters: This feature continuously and scalablely provides real-time operation metrics and machine learning features by aggregating data at write time to assist you in processing high-frequency event data, such as clickstreams, directly in your database.

SQL support: With more than 100 SQL functions now included into Bigtable, developers may use their current knowledge to take advantage of Bigtable’s scalability and performance.

For a number of business-critical workloads, including Advanced WildFire, Bigtable is the database of choice because to these improvements and its current features.

From Cassandra to Bigtable at Palo Alto Networks

Advanced WildFire from Palo Alto Networks is the biggest cloud-based malware protection engine in the business, evaluating more than 1 billion samples per month to shield enterprises from complex and cunning attacks. It leverages more than 22 distinct Google Cloud services in 21 different regions to do this. A NoSQL database is essential to processing massive volumes of data for Palo Alto Networks’ Global Verdict Service (GVS), a key component of WildFire, which must be highly available for service uptime. When creating Wildfire, Apache Cassandra first appeared to be a good fit. But when performance requirements and data volumes increased, a number of restrictions surfaced:

Performance bottlenecks: Usually caused by compaction procedures, high latency, frequent timeouts, and excessive CPU utilization affected user experience and performance.

Operational difficulty: Managing a sizable Cassandra cluster required a high level of overhead and specialized knowledge, which raised management expenses and complexity.

Challenges with replication: Low-latency replication across geographically separated regions was challenging to achieve, necessitating a sophisticated mesh architecture to reduce lag.

Scaling challenges: Node updates required a lot of work and downtime, and scaling Cassandra horizontally proved challenging and time-consuming.

To overcome these constraints, Palo Alto Networks made the decision to switch from GVS to Bigtable. Bigtable’s assurance of the following influenced this choice:

High availability: Bigtable guarantees nearly continuous operation and maximum uptime with an availability SLA of 99.999%.

Scalability: It can easily handle Palo Alto Networks’ constantly increasing data needs because to its horizontally scalable architecture, which offers nearly unlimited scalability.

Performance: Bigtable provides read and write latency of only a few milliseconds, which greatly enhances user experience and application responsiveness.

Cost-effectiveness: Bigtable’s completely managed solution lowers operating expenses in comparison to overseeing a sizable, intricate Cassandra cluster.

For Palo Alto Networks, the switch to Bigtable produced outstanding outcomes:

Five times less latency: The Bigtable migration resulted in a five times reduced latency, which significantly enhanced application responsiveness and user experience.

50% cheaper: Palo Alto Networks was able to cut costs by 50% because of Bigtable’s effective managed service strategy.

Improved availability: The availability increased from 99.95% to a remarkable 99.999%, guaranteeing almost continuous uptime and reducing interruptions to services.

Infrastructure became simpler and easier to manage as a result of the removal of the intricate mesh architecture needed for Cassandra replication.

Production problems were reduced by an astounding 95% as a result of the move, which led to more seamless operations and fewer interruptions.

Improved scalability: Bigtable offered 20 times the scale that their prior Cassandra configuration could accommodate, giving them plenty of space to expand.

Fortunately, switching from Cassandra to Bigtable can be a simple procedure. Continue reading to find out how.

The Cassandra to Bigtable migration

Palo Alto wanted to maintain business continuity and data integrity during the Cassandra to Bigtable migration. An outline of the several-month-long migration process’s steps is provided below:

The first data migration

To begin receiving the transferred data, create a Bigtable instance, clusters, and tables.

Data should be extracted from Cassandra and loaded into Bigtable for each table using the data migration tool. It is important to consider read requests while designing the row keys. It is generally accepted that a table’s Cassandra primary key and its Bigtable row key should match.

Make sure that the column families, data types, and columns in Bigtable correspond to those in Cassandra.

Write more data to the Cassandra cluster during this phase.

Verification of data integrity:

Using data validation tools or custom scripts, compare the Cassandra and Bigtable data to confirm that the migration was successful. Resolve any disparities or contradictions found in the data.

Enable dual writes:

Use Cassandra and dual writes to Bigtable for every table.

To route write requests to both databases, use application code.

Live checks for data integrity:

Using continuous scheduled scripts, do routine data integrity checks on live data to make sure that the data in Bigtable and Cassandra stays consistent.

Track the outcomes of the data integrity checks and look into any anomalies or problems found.

Redirect reads:

Switch read operations from Cassandra to Bigtable gradually by adding new endpoints to load balancers and/or changing the current application code.

Keep an eye on read operations’ performance and latency.

Cut off dual writes:

After redirecting all read operations to Bigtable, stop writing to Cassandra and make sure that Bigtable receives all write requests.

Decommission Cassandra:

Following the migration of all data and the redirection of read activities to Bigtable, safely terminate the Cassandra cluster.

Tools for migrating current data

The following tools were employed by Palo Alto Networks throughout the migration process:

‘dsbulk’ is a utility for dumping data. Data can be exported from Cassandra into CSV files using the ‘dsbulk’ tool. Cloud Storage buckets are filled with these files for later use.

To load data into Bigtable, create dataflow pipelines: The CSV files were loaded into Bigtable in a test environment using dataflow pipelines.

At the same time, Palo Alto decided to take a two-step method because data transfer is crucial: first, a dry-run migration, and then the final migration. This tactic assisted in risk reduction and process improvement.

A dry-run migration’s causes include:

Test impact: Determine how the ‘dsbulk’ tool affects the live Cassandra cluster, particularly when it is under load, and modify parameters as necessary.

Issue identification: Find and fix any possible problems related to the enormous amount of data (terabytes).

Calculate the estimated time needed for the migration in order to schedule live traffic handling for the final migration.

It then proceeded to the last migration when it was prepared.

Steps in the final migration:

Set up pipeline services:

Reading data from all MySQL servers and publishing it to a Google Cloud Pub/Sub topic is the function of the reader service.

Writer service: Converts a Pub/Sub topic into data that is written to Bigtable.

Cut-off time: Establish a cut-off time and carry out the data migration procedure once more.

Start services: Get the writer and reader services up and running.

Complete final checks: Verify accuracy and completeness by conducting thorough data integrity checks.

This methodical technique guarantees a seamless Cassandra to Bigtable migration, preserving data integrity and reducing interference with ongoing business processes. Palo Alto Networks was able to guarantee an efficient and dependable migration at every stage through careful planning.

Best procedures for migrations

Database system migrations are complicated processes that need to be carefully planned and carried out. Palo Alto used the following best practices for their Cassandra to Bigtable migration:

Data model mapping: Examine and convert your current Cassandra data model to a Bigtable schema that makes sense. Bigtable allows for efficient data representation by providing flexibility in schema construction.

Instruments for data migration: Reduce downtime and expedite the data transfer process by using data migration solutions such as the open-source “Bigtable cbt” tool.

Adjusting performance: To take full advantage of Bigtable’s capabilities and optimize performance, optimize your Bigtable schema and application code.

Modification of application code: Utilize the special features of Bigtable by modifying your application code to communicate with its API.

However, there are a few possible dangers to be aware of:

Schema mismatch: Verify that your Cassandra data model’s data structures and relationships are appropriately reflected in the Bigtable schema.

Consistency of data: To prevent data loss and guarantee consistency of data, carefully plan and oversee the data migration procedure.

Prepare for the Bigtable migration

Are you prepared to see for yourself the advantages of Bigtable? A smooth transition from Cassandra to Bigtable is now possible with Google Cloud, which uses Dataflow as the main dual-write tool. Your data replication pipeline’s setup and operation are made easier with this Apache Cassandra to Bigtable template. Begin your adventure now to realize the possibilities of an extremely scalable, efficient, and reasonably priced database system.

DARPA AI cyber challenge

Among the most obvious advantages of artificial intelligence (AI) is its capacity to improve cybersecurity for businesses and global internet users. This is particularly true since malicious actors are still focussing on and taking advantage of vital software and systems. These dynamics will change as  artificial intelligence develops, and if used properly,  AI may help move the sector towards a more secure architecture for the digital age.

Experience also teaches us that, in order to get there, there must be strong public-private sector partnerships and innovative possibilities such as DARPA’s AI Cyber Challenge (AIxCC) Semifinal Competition event, which will take place from August 8–11 at DEF CON 32 in Las Vegas. Researchers in cybersecurity and AI get together for this two-year challenge to create new AI tools that will aid in the security of significant open-source projects.

This competition is the first iteration of the challenge that DARPA AIxCC unveiled at Black Hat last year. Following the White House’s voluntary AI commitments, which saw business and government unite to promote ethical methods in AI development and application, is the Semifinal Competition. Today, we’re going to discuss how Google is assisting rivals in the AIxCC:

Google  Cloud resources: Up to $1 million in Google Cloud credits will be awarded by Google to each qualifying AIxCC team, in an effort to position rivals for success. The credits can be redeemed by participants for Gemini and other qualified Google Cloud services. As a result, during the challenge, competitors will have access to Google Cloud AI and machine learning models, which they can utilise and expand upon for a variety of purposes. Google is urge participants to benefit from Google incentive schemes, such as the Google for firms Program, which reimburses up to $350,000 in Google Cloud expenses for AI firms.

Experts in cybersecurity: For years, Google has been in the forefront of utilising AI in security, whether it is for sophisticated threat detection or shielding Gmail users from phishing scams. Google excited to share knowledge with AIxCC by making security specialists available throughout the challenge, they have witnessed the promise of AI for security. Google security specialists will be available to offer guidance on infrastructure and projects as well as assist in creating the standards by which competitors will be judged. Specifically, Google discuss recommended approaches to AIxCC in Google tutorial on how AI may improve Google’s open source OSS-Fuzz technology.

Experience AIxCC at DEF CON: We’ll be showcasing Google’s AI technology next week at the AIxCC Semifinal Experience at DEF CON, complete with an AI-focused hacker demo and interactive demonstrations. In order to demonstrate AI technologies, which includes Google Security Operations and SecLM, Google will also have Chromebooks at the Google exhibit. Google security specialists will be available to guests to engage in technical conversations and impart information.

AI Cyber Challenge DARPA

At Black Hat USA 2023, DARPA invited top computer scientists,  AI experts, software engineers, and others to attend the AI Cyber Challenge (AIxCC). The two-year challenge encourages AI-cybersecurity innovation to create new tools.

Software powers everything in Google ever-connected society, from public utilities to financial institutions. Software increases productivity and makes life easier for people nowadays, but it also gives bad actors a larger area to attack.

This surface includes vital infrastructure, which is particularly susceptible to cyberattacks, according to DARPA specialists, because there aren’t enough technologies to adequately secure systems on a large scale. Cyber defenders face a formidable attack surface, which has been evident in recent years as malevolent cyber actors take advantage of this state of affairs to pose risks to society. Notwithstanding these weaknesses, contemporary technological advancements might offer a way around them.

AIxCC DARPA

According to Perri Adams, DARPA’s AIxCC program manager, “AIxCC represents a first-of-its-kind collaboration between top AI companies, led by DARPA to create AI-driven systems to help address one of society’s biggest issue: cybersecurity.” Over the past decade, exciting AI-enabled capabilities have emerged. DARPA see a lot of potential for this technology to be used to important cybersecurity challenges when handled appropriately. DARPA can have the most influence on cybersecurity in the nation and the world by automatically safeguarding essential software at scale.Image credit to DARPA

Participation at AIxCC will be available on two tracks: the Funded Track and the Open Track. Competitors in the Funded Track will be chosen from submissions made in response to a request for proposals for Small Business Innovation Research. Funding for up to seven small enterprises to take part will be provided. Through the competition website, Open Track contestants will register with DARPA; they will not receive funds from DARPA.

During the semifinal phase, teams on all tracks will compete in a qualifying event where the best scoring teams (up to 20) will be invited to the semifinal tournament. The top five scoring teams from these will advance to the competition’s final stage and win cash prizes. Additional cash awards will be given to the top three contestants in the final competition.

DARPA AIxCC

Leading AI businesses come together at DARPA AIxCC, where they will collaborate with DARPA to make their state-of-the-art technology and knowledge available to rivals. DARPA will work with Anthropic, Google, Microsoft, and Open AI to allow rivals to create cutting-edge a cybersecurity solutions.

As part of the Linux Foundation, the Open Source Security Foundation (OpenSSF) will act as a challenge advisor to help teams develop AI systems that can tackle important cybersecurity problems, like the safety of a crucial infrastructure and software supply chains. The majority of software, and therefore the majority of the code that requires security, is open-source and frequently created by volunteers in the community. Approximately 80% of contemporary software stacks, which include everything from phones and cars to electricity grids, manufacturing facilities, etc., use open-source software, according to the Linux Foundation.

Lastly, there will be AIxCC competitions at DEF CON and additional events at Black Hat USA. These two globally renowned cybersecurity conferences bring tens of thousands of practitioners, experts, and fans to Las Vegas every August. There will be two stages of AIxCC: the semifinal stage and the final stage. DEF CON in Las Vegas will hold 2024 and 2025 semifinals and finals.

DEF CON 32’s AI Village

Google will keep supporting the  AI Village at DEF CON, which informs attendees about security and privacy concerns related to AI, in addition to work with the AIxCC. In addition to giving the Village a $10,000 donation this year, Google will be giving workshop participants Pixel books so they can acquire practical experience with the nexus of cybersecurity and AI. After a successful red teaming event in which Google participated and security experts worked on AI security concerns, the AI Village is back this year.

Google excited to see what fresh ideas emerge from the AIxCC Semifinal Competition and to discuss how they may be used to safeguard the software that Google all use. Meanwhile, the Secure AI Framework and the recently established Coalition for Secure AI provide additional information about their efforts in the field of AI security.

In the fast-paced world of technology, the interplay between Machine Learning (ML) and Artificial Intelligence (AI) stands as a pivotal force driving innovation. As we delve deeper into an era integrating AI into our daily lives, it’s the strides in ML that serve as the bedrock for this transformative journey.

Machine Learning forms the backbone of AI, enabling systems to learn, adapt, and evolve autonomously without explicit programming. This dynamic capability empowers AI algorithms to sift through colossal data sets, identify patterns, and make informed predictions or decisions based on acquired knowledge.

The progress of AI owes much to the strides in ML techniques, particularly in neural networks and deep learning. Inspired by the human brain’s neural structure, these methodologies empower AI models to comprehend intricate data structures, uncover correlations, and accomplish tasks once deemed beyond machine capabilities.

ML’s profound impact on AI’s evolution extends across diverse industries and applications. From revolutionizing healthcare through predictive analytics for disease diagnosis to reshaping transportation via autonomous vehicles, ML-driven AI is reshaping the world as we know it.

Moreover, the accessibility of ML tools and frameworks has democratized AI innovation. Developers and researchers now wield sophisticated ML libraries and platforms, fostering a vibrant community dedicated to pushing the boundaries of AI’s potential.