#bandarlog

By Alexey Lipodat, Project Engagement, Oath

One of the biggest problems with a typical Extract, Transform, Load (ETL) workflow is that stringing together multiple systems adds processing time that is often unmeasured. At Oath, we turned to open source for a solution.

When researching possible solutions for enabling process metrics within our ETL pipeline, we searched for tools focused on Apache Kafka, Athena, Hive, Presto, and Vertica. For Kafka, we discovered Burrow, a monitoring service that provides consumer lag metric, but no existing options were available for Hive and Vertica. Creating or integrating a different monitoring application for each component of the ETL pipeline would significantly increase the complexity to run and maintain the system.

In order to avoid unnecessary complexity and effort, we built Bandar-Log, a simple, standalone monitoring application based on typical ETL workflows. We published it as an open source project so that it can be a resource for other developers as well.

Meet Bandar-Log

Bandar-Log is a monitoring service that tracks three fundamental metrics in real-time: lag, incoming, and outgoing rates. It runs as a standalone application that tracks performance outside the monitored application and sends metrics to an associated monitoring system (e.g. Datadog).

A typical ETL assumes there will be some processing logic between data sources. This assumption adds some delay, or "resistance," which Bandar-Log can measure.

For example:

How many events is the Spark app processing per minute as compared to how many events are coming to Kafka topics?

What is the size of unprocessed events in Kafka topics at this exact moment?

How much time has passed since the last aggregation processed?

Bandar-Log makes it easy to answer these questions and is a great tool:

Simple to use — create your own Bandar-Log in 10 minutes by following the Start Bandar-Log in 3 steps doc, and easily extend or add custom data sources

Stable — tested extensively on real-time big data pipelines

Fully supported — new features are frequently added

Flexible — no modifications needed to existing apps for this external, standalone application to monitor metrics.

 How Bandar-Log works

To better understand how Bandar-Log works, let's review a single part of the ETL process and integrate it with Bandar-Log.

As an example, we’ll use the replication stage, the process of copying data from one data source to another. Let's assume we already have aggregated data in Presto/Hive storage and now we’ll replicate the data to Vertica and query business reports. We'll use a replicator app to duplicate data from one storage to another. This process copies the dedicated batch of data from Presto/Hive to Vertica.

The replication process is now configured, but how should we track the progress? We want to make sure that the replication works as expected and that there isn’t any delay. In order to confirm that the replication is working correctly, we will need to know:

How much unreplicated data do we have in Presto/Hive?

What's the incoming rate of Presto/Hive data?

How much data has already been replicated to Vertica?

What's the lag between unreplicated and already replicated data?

Inserting tracking logic directly in the replicator app or adding tracking logic in all the ETL pipeline components where we need the process status pose concerns, as these approaches are not scalable and if the replicator app fails, no information is received. A standalone application, like Bandar-Log, solves these issues by tracking status from outside the monitored application.

Let's integrate Bandar-Log with the replication pipeline. As mentioned above, Bandar-Log operates with three fundamental metrics:

IN - incoming rate

OUT - outgoing rate

LAG - lag rate (the difference between IN and OUT)

Every component independently interprets lag and incoming/outgoing rates. For instance, a Spark-driven app depends on Kafka, therefore, the lag and rates signify the number of unread messages, both producing and consuming rates. For our replicator app, incoming and outgoing rates mean how many batches of data arrived and were processed per some interval. ETL components use a dedicated column to mark and isolate a specific piece of processed data, which is called batch_id. The semantics of batch_id is a Unix time timestamp measured in milliseconds, which determines the time when the piece of data was processed.

Keeping this in mind, we’ll now use the Presto/Hive source to measure the input rate and our Vertica source to measure the output rate.  

Due to this approach, an input rate will be the last processed timestamp from Presto/Hive, and the output rate will be the last replicated timestamp from Vertica. Bandar-Log will fetch these metrics according to the rate interval, which can be configured to retrieve metrics based on preference. Fetched metrics will be pushed to Datadog, the default monitoring system.

Now that we have the input and output rate metrics. What about lag? How can we track the delay of our replication process?

This is quite simple: the LAG metric depends on IN and OUT metrics and is calculated as the difference between them. If we have both of these metrics, then we can easily calculate the LAG metric, which will show us the delay in the replication process between Presto/Hive and Vertica. However, if we are using the real-time system then we should provide data faster and track the status of this process. This metric can be called "Real-time Lag" and determines the delay between the current timestamp and the timestamp associated with the last processed or replicated batch of data.

To calculate the REALTIME_LAG metric, the IN metric from Presto/Hive isn’t needed; instead, we'll use the current timestamp and OUT metric from the final source, which is Vertica. After this step we'll receive four metrics (IN, OUT, LAG, REALTIME_LAG) in the monitoring system. We can now create appropriate dashboards and alert monitors.

This is just one of many examples showing how you can use Bandar-Log. Explore more examples in the Bandar-Log Readme doc. Download the code and if you’d like to contribute any code fixes or additional functionality, visit Bandar-Log on Github and become part of our open source community.