#apache flink

I’ve been committed to the Apache Spark ecosystem for as long as I’ve been doing data engineering. I’ve seen it’s adoption, been a fan of using the APIs and have even decided to write a book about Spark. In all of this experience, there has been a nagging sense that I need to try a “true” streaming framework like Twitter Heron, Apache Flink or Apache Apex. I’ve read articles and watched videos about the difference between a true streaming and a micro batch framework and been mostly unconvinced that it would make a difference for me. I started a project where Flink ended up being the go to streaming framework and I learned a great deal about Flink. This post is a combination of some notes and things I noticed while using Flink and hopefully anyone starting Flink can find it helpful.

Getting Started

Like Spark, Flink is fairly overwhelming to get started with. This is mostly because of installations and run-time configurations. After a bunch of searching around and I was able to put together a decent starter SBT config for Flink. I used Intellij to work with Flink because of it’s complex API, the type hinting and other niceties come in pretty handy.

For other ways using Flink (Docker, cluster, Mesos) I found the docs pretty helpful for that. This blog is fairly elementary, so I won’t cover those in this post.

Ergonomics / API

What I noticed early on using Flink is that it really was designed with streaming in mind first. This is evident by different APIs for a Key Stream, versus a DataStream versus a Window Stream. Once you work with streams for a little while, you’ll find yourself wanting an API that is robust and can handle streams in a sensible way. In Spark streaming, I found myself writing a bunch of boilerplate code aimed at accomplishing a lot of what Flink is designed to do out of the box. One of the ways this is most evident is the DataStream and it’s derivatives, I’ll walk through each.

DataStream

A Datastream is a typed stream of data is one-at-a-time. It’s what all of Flinks streaming architecture is built off of. You can use this stream to do whatever you wanted to. For example I could read in a stream and add two to each value:

Or, I could read in a stream and write it back out to the file system.

KeyedDataStream

A KeyedDataStream is a stream grouped by and evaluated by a key. Typically, this is the first step in a longer process of creating a window or other type of stream. The API is pretty straight forward, you just add a .keyBy("Name your key") to a DataStream. In and of itself it's not all that useful. It comes in handy with a Window.

WindowedDataStream

A WindowedDataStream is what you’d expect, it’s a DataStream over some window of time. This is how Spark Streaming works out of the box and how most introductory streams are introduced. Flinks API is very robust allowing you to configure streams based on number of occurances and windows of time. This kind of control makes it easy to work with out of the box versus custom code to do it.

You can also define what are known as TriggerPolicies and EvictionPolicies, in which you can define a specific trigger and end state for that trigger. These require a bit more orchestration but can be effective for a number of streaming situations.

Technically, Spark supports all 3 of these but not with the same ease of use. With the new Structured Streaming API most of these mechanics are there now, but it’s still not as intuitive as Flink in my opinion.

Another thing that is extremely nice about Flink is their dashboard. Spark has a notoriously bad dashboard but it’s not nearly as nice as this:

It’s quite easy to follow the flow of data in Flink and consume and create streaming sources without a ton of effort.

Batch Model

The real strength of Spark is in it’s batch model and in comparison Flink is actually pretty nice to use. Doing most of your batch related transformations is just as nice as it is to do in Spark. Here are a few examples:

Ecosystem

When your ecosystem bar is Spark, you’ll be hard pressed to meet it. Flink supports all of the major streaming technologies like Apache Kafka, AWS Kinesis and Debezium. Being the newer kid on the block, it’s just not as rich as what Spark has to offer. That being said, with full support of the Scala and Java ecosystem, I have yet to find a situation Flink couldn’t handle.

Summary

It’s clear that a good deal of smart people and companies are investing a lot in Apache Flink, but the question remains, should you? A lot is written and said about streaming, but in my experience, the majority of workloads are still batch. Spark is a much more mature ecosystem for batch workflows and a pretty mature one for streaming as well.

I’m a pretty big believer in streams and their potential. I’m not sure that streams are the wave of the future for Dashboards, reporting and applications that are frozen in time, but I think it’s already clear that some of the most impressive and useful apps are built off of real time data. These are the kinds of applications I want to build my career on, so I’ll be using Flink more often. I hope to write some useful beginner friendly tutorials on Flink in the coming months as well!

Notes

Introduction  

When building a recommendation system, the quality of your data pipeline determines the quality of your recommendations. After evaluating Cassandra’s Batch Processing Platform (BPP), we decided to build our Universal Recommendation System’s data ingestion pipeline on Stream Processing Platform (SPP) with Apache Flink. Six months and millions of processed messages later, here is why it was the right decision and what we learned along the way.  

Introduction to SPP and Apache Flink  

Stream Processing Platform (SPP) is an enterprise streaming platform that provides event-driven processing, automatic deployment, monitoring, and native integration with Event Bus (Kafka). Apache Flink is the streaming engine powering SPP, providing exactly-once semantics, stateful processing, and horizontal scalability.  

What Our Stream Processing Pipeline Delivers 

For personalisation use cases, stale data means missed opportunities — a recommendation system is only as good as the data feeding it. Our pipeline addresses this directly by keeping recommendation data in near-real-time sync with source systems. This means newly published content is available for recommendation within seconds, incoming events are dynamically enriched before reaching the recommender, and delivery remains reliable at scale — so users always receive contextually relevant, up-to-date recommendations rather than yesterday’s data. 

How SPP Helps Achieve the Above Use Cases  

SPP provides event-driven streaming with sub-second latency (vs. batch delays), horizontal auto-scaling based on message volume, native Kafka integration and tooling, automatic checkpointing and state management, and SDK support with local development tools. These capabilities directly support our goals of near-real-time sync, scalability, and faster iteration for personalisation.  

High-Level Stream Processing Pipeline Architecture  

The architecture follows an event-driven pattern: Kafka Source → JSON Parsing → Dynamic Pipeline Orchestration → Recommender Output. Failed messages are managed via fail-fast and filtering (logged and dropped so healthy traffic continues).  

Apache Flink serves as the streaming engine, providing exactly-once semantics, stateful processing, and horizontal scalability. SPP adds enterprise-grade features including automatic deployment, monitoring, and integration with the ecosystem.  

High-Level Architecture Diagram  

Explanation: Event Source: Two Kafka-backed flows: OICMS (content ingestion topic) and Campaign Manager (CM events topic).

Stream Processing — SPP on Apache Flink: shared ingest and JSON parsing; flow-specific validation (details in diagram for both OICMS and CM); flow-specific enrichment (OICMS: fetch external content; CM: recommender API + CM placement).

Destination: Both flows write to the Recommender System.  

Why Real-Time Stream Processing Matters  

Traditional batch processing systems collect data and deliver results hours or days later. For modern recommendation systems, this delay is unacceptable. Real-time processing enables immediate content availability, dynamic enrichment, and reliable delivery at scale. Understanding the right architecture for stream processing keeps engineering teams ahead of the scalability curve.  

The Journey: From BPP to SPP  

We started with a proof-of-concept using Cassandra’s Batch Processing Platform. While BPP is powerful for scheduled batch jobs, we quickly realised our recommendation system needed real-time, event-driven processing.  

Advantages of SPP Over Batch Processing  

Real-Time vs. Batch: SPP provides event-driven streaming with sub-second latency, while BPP processes data in scheduled intervals with hours of delay.  

Scalability: SPP offers horizontal auto-scaling based on message volume, whereas BPP relies on vertical scaling with manual capacity planning.  

Integration: SPP provides native Event Bus (Kafka) integration and extensive tooling, reducing development time significantly.  

Operations: SPP handles automatic checkpointing and state management, while BPP requires manual job scheduling and failure recovery.  

Developer Experience: SPP includes SDK support, local development tools, and comprehensive documentation, making it much easier for teams to be productive.  

Reliability: We achieved 99.9% uptime over six months in production.  

Cost Optimisation: 40% reduction in operational costs through efficient auto-scaling and resource utilisation.  

Our Architecture: Event-Driven Stream Processing  

We designed a multi-stage processing pipeline built on Apache Flink that ingests content from Kafka, processes it through configurable stages, and delivers enriched data to the recommendation system in real time.  

Key Architectural Decisions  

Several deliberate design choices shaped the reliability, maintainability, and performance of our pipeline. Here is what made the most difference. 

Configuration-Driven Pipeline Design  

One of our most impactful decisions was making the entire pipeline configuration-driven. Instead of hardcoding processing logic, we externalised all processing steps, validation rules, and service endpoints into HOCON configuration files. This approach delivers significant benefits: zero-code deployments for pipeline modifications, environment-specific configurations (local, QAL, production), easy experimentation with different processing strategies, and deployment time reduced from hours to minutes.  

Resilience and Error Handling Patterns  

We designed the system expecting failures in distributed environments. Key resilience patterns include: Retry Mechanisms with exponential backoff for external API calls and configurable retry attempts per service; fail-fast and filtering so failed messages are logged and dropped without blocking healthy traffic; Circuit Breakers to prevent cascading failures when external services become unavailable; and Monitoring & Observability with structured logging and Splunk integration for real-time visibility into pipeline health, performance metrics, and error tracking.  

Type-Safe Data Transformation  

We integrated MapStruct for compile-time code generation of data mappings between DTOs and domain models. This approach catches mapping errors during compilation rather than at runtime, eliminating an entire class of production bugs and making refactoring safer.  

Local Development Without Infrastructure  

Running a full Flink cluster with Kafka locally is complex. We built a Local Producer tool that executes the complete pipeline locally without infrastructure dependencies. This reduced the developer feedback loop from hours to seconds, dramatically improving productivity and code quality.  

Results in Production  

Since deploying to production, our pipeline has delivered:  

Performance: Processing 10,000+ messages per second with p99 latency under 100ms end-to-end. 

Reliability: 99.9% uptime over six months in production. 

Operational Efficiency: Reduced deployment time from 2 hours to 15 minutes through configuration updates. 

Cost Optimisation: 40% reduction in operational costs through efficient auto-scaling and resource utilisation. 

Developer Velocity: Faster iteration cycles and reduced time-to-market for new features.  

Lessons Learnt  

Validate Through POCs: Evaluating both BPP and SPP through proof-of-concepts saves months of potential rework. Hands-on experimentation enables confident architectural decisions.  

Configuration Over Code: Dynamic, configuration-driven systems are significantly easier to maintain, test, and evolve than hardcoded implementations.  

Design for Failure Early: Implementing comprehensive error handling patterns from day one prevented production incidents as the system scaled.

Invest in Developer Tools: Building tools that improve the inner development loop has an outsized impact on team velocity and code quality.  

Type Safety Reduces Risk: Compile-time validation caught bugs that would have been costly production failures.  

Observability is Critical: Without comprehensive monitoring and structured logging, operating distributed systems at scale becomes impossible.  

Conclusion  

Choosing Stream Processing Platform over Batch Processing Platform gives us the low latency, scalability, and reliability needed for modern recommendation systems. Our configuration-driven architecture on Apache Flink with SPP’s enterprise features created a maintainable foundation for near-real-time data synchronisation to support personalisation use cases. Validating technology choices through POCs, prioritising configuration over code, and designing for failure and observability from the start were key to our success.  

Want to build smarter pipelines with AI-first architecture? At Covalense Digital, we specialise in building intelligent, AI-first platforms that power real-time decisioning, personalised customer experiences, and scalable data pipelines — for telecoms, enterprises, and beyond. Whether you’re modernising a legacy BSS stack or engineering a next-generation data architecture, our expertise in AI/ML, iPaaS, and digital transformation can help you move faster and scale confidently. 

Let's connect at [email protected] or fill in a quick contact form.

Corraling Kafka: New ecosystem simplifies, democratizes event-streaming data for enterprises

Aiven’s open-source ecosystem for Apache Kafka helps enterprises consolidate their Kafka infrastructure using open-source components.Read More

Berlin-based Immerok picks up €17 million to harness the power of real-time stream processing

Founded by a team of Apache Flink experts, Immerok has just secured €17 million to enhance how businesses use Apache Flink to process data in real-time. The Berlin-based startup is now planning to launch its cloud service and expand the team.  In our digital world, data-driven decisions and insights are ruling the way businesses operate…. This content is for members only. Visit the site and log…