Uhmm

Hi. This is my blog where I freak out over VIRGINMEDIA and Hazbin hotel as a whole (mostly Alastor if I'm being honest but yeah). Feel free to call me Mutt, Fawn or Bambi if you'd like, I use it/it's pronouns only and will obliterate anyone using they/them on me. With exceptions of friends, they are allowed to know my other pronouns, but you strangers, you get it/it's only

That's all

— @virginmedia

How BigQuery Analytics Hub streamlined internal data exchange for Virgin Media O2

Data sharing made simple is now a vital tool for any company looking to make well-informed decisions. Nevertheless, a lot of businesses still find it difficult to share data in an efficient and legal manner. Uncertain governance, version control problems, data silos, access limitations, and a lack of data management expertise within the larger organization are common obstacles that data teams must overcome.

Virgin Media O2, a media and telecoms company, uses internal data sharing to drive strategy, enhance operations, and give decision makers more authority. The data team supports every department for timely and reliable information, from marketing to finance.

Virgin Media O2 required a system that would facilitate governance and data access amongst business units. Without it, it wouldn’t be able to achieve org-level visibility and efficiency and would be stuck without a centralized data-sharing method.

Overcoming obstacles to internal data-sharing

Strong version control was necessary to guarantee that the data was always correct, consistent, and up to date because teams were already working on Google Cloud-based projects. However, this frequently prolonged the time it required to generate new insights. In order to meet their enterprise and AI needs, Virgin Media O2 already had their corporate data in BigQuery. Therefore, one possible way to build on their current infrastructure was to use Analytics Hub, which is BigQuery’s data sharing feature.Image credit to Google Cloud

After learning about BigQuery Analytics Hub‘s scalability, self-service capabilities, and straightforward governance mechanism for data tagging and quality, the data platform team made the decision to trial the product. This final aspect, in particular, was in line with enhancing the implementation of privacy by design.

Following a successful trial, Virgin Media O2 had developed a well-defined onboarding and training procedure. Two owners were assigned to each new data exchange, and two owners were assigned to the subscriber side to facilitate the tracking of any actions within BigQuery. This method was expanded to 25 teams, over 50 exchanges, 100 postings, 500 tables, and around 300 daily customers over the course of the following nine months.

The absence of data duplication in BigQuery Analytics Hub saves on network and storage expenses, which is one of the main advantages the team discovered. It accomplishes this by generating a shareable real-time pointer to the underlying dataset, referred to as a Linked Dataset. As a result, any subscriber can access updated data instantly and it’s simple to audit, trace, and restore the original data source. By using this method, a safety net for catastrophe recovery is also included.

The complexity issues associated with building views are also resolved by BigQuery Analytics Hub, notably the fact that permitted views frequently cause original table metadata to be lost when accessing data. All table descriptions and columns are still visible to subscribers who have direct access to the original dataset.

Virgin Media O2 was able to save time, lower latency, and improve management and usability for publishers and subscribers by connecting data directly from the data publisher to a data subscriber. Additionally, the platform’s enhanced governance offered a centralized location for managing data access and quality.

BigQuery Analytics Hub minimizes human labor and mistakes by streamlining the data sharing process between teams and business divisions. Within Virgin Media O2, the platform has been especially helpful to software developers, analytics engineers, data scientists, data engineers, and analysts. It guarantees that everyone has instant access to the data they require for their different jobs.

Using data that is more readily available than ever to save time

The solution helped save up to 30 hours a week on time spent on training, support, pipelines difficulties with deployment, and communication overhead from squads using the old approach after BigQuery Analytics Hub was rolled out to about 25 squads. Due to the nearly nonexistent problems, the weekly time spent by all teams is now as little as thirty minutes. According to the team, this results in an effort savings of about 95%. Data is now widely accessible to the various departments that require it, as it is no longer stored in silos.

Without requiring users to use BigQuery Analytics Hub directly, the team was able to democratize data access for subscribers and their larger teams by creating a dashboard. While retaining a robust governance approach, self-service is made possible by allowing users to subscribe to datasets. Cutting out the intermediary simplifies and expedites this tedious procedure while preserving a strong governance framework.

Principal advantages of secure data exchange

Security and Integrity of Data

By guarding against metadata loss and unwanted access, secure, zero-copy sharing guarantees consistent data integrity across departments.

Economy of Cost and Streamlined Administration

The platform lowers operational overhead and long-term costs by doing away with data transfer, and data oversight may be efficiently handled by a small workforce.

Centralized Administration and Observation

Real-time control over data sharing operations is made possible by a single dashboard, which also enforces stringent access and authorization regulations and enables prompt issue detection.

The group’s future priorities include streamlining four important areas: data ownership, data catalog, data quality measurements, and more efficient sensitive data tagging. After data is certified, these four areas need to be precisely specified and strictly adhered to as a policy via BigQuery Analytics Hub in order to achieve the goal of automating the complete data activity.Image credit to Google Cloud

The aforementioned procedure, referred to as “data certification,” has two main advantages:

By utilizing data quality measurements (at the column level) and data lineage, it is possible to quickly identify uncertified data assets and track out problems with the quality of the data in a matter of minutes.

Real-time audit logs that allow for the proactive control of data privacy issues by identifying sensitive data in real-time and tracking its consumption.

BigQuery is the best place for users who are new to Google Cloud to begin their adventure. After BigQuery, Analytics Hub is the next best thing for users.