#payment data pipelines

Payment data pipelines fail in ways that ruin a payments engineer’s week, and the failures rhyme. The dashboards froze. Fraud scores arrived after the transaction had already cleared. Settlement reports came in stale. Nobody slept. The frustrating part is that the same data architecture had run fine for years. So, what changed?

The honest answer is that batch thinking does not survive contact with real-time payments. A lot of banks built their data foundations in an era when nightly jobs were good enough. Load the warehouse overnight, run the reports in the morning, move on. That rhythm worked when money moved slowly. It does not work when a customer expects an instant confirmation and a fraud engine has milliseconds to make a call.

Here is where things crack. Real-time payment rails push a constant stream of events instead of a tidy nightly dump. Your pipeline now has to ingest, transform, and serve data while transactions are still happening. Add ISO 20022 into the mix and the pressure climbs. ISO 20022 messages are rich. They carry far more structured detail than the old formats, which is wonderful for analytics and miserable for a pipeline that was never designed to parse that much context at speed. This is not a fringe concern either. Swift reported that by the time its MT/ISO 20022 coexistence period closed in November 2025, around 80% of daily traffic was already running on the ISO 20022 format, with more than 3.1 million of these messages exchanged every day. The rich-data era is the default now, not the roadmap.

Then there is the fraud-scoring window. Fraud models need fresh features. Account behaviour over the last few minutes, velocity checks, device signals. If your pipeline takes thirty seconds to surface that data, the fraud decision is already too late. You are essentially detecting fraud after the loss. 

That gap between when data is created and when it becomes usable is the silent killer in most payment systems. And the cost of getting it wrong runs in both directions. Javelin Strategy & Research estimated that wrongly declined transactions cost merchants roughly $118 billion in a single year, against about $9 billion in genuine card fraud over the same period. When your features arrive late, you do not just miss fraud, you also misjudge good customers, and the second mistake is the more expensive one.

So how do teams actually fix it? A few moves matter more than the rest.

First, separate the hot path from the cold path. Not every piece of data needs to move in real time. Fraud scoring and authorization need low latency. Quarterly trend analysis does not. Mixing both in one pipeline means the slow stuff drags down the fast stuff. Splitting them lets each run at its own pace.

Second, treat data domains as first-class citizens. Payments, cards, risk, and customer data each have their own shape and their own owners. When you map these domains deliberately instead of dumping everything into one lake, you get clarity, and clarity is what keeps latency predictable. This domain-driven approach is exactly the kind of foundation that strong data engineering services are built to deliver, especially in a payments context where the cost of a stale record is measured in real money.

Third, build observability in from the start. You cannot fix what you cannot see. Most pipeline failures are not sudden. They creep. A queue backs up a little, then a little more, until one Monday morning everything is an hour behind. Good observability catches the creep early, before it becomes a 2 a.m. incident.

Fourth, push governance into the pipeline rather than bolting it on afterward. In financial services, a fast pipeline that cannot prove its data lineage is a liability. Embedding governance and quality checks at the point of ingestion means you stay both quick and auditable. This is no longer just good hygiene. Under the EU’s Digital Operational Resilience Act, in force since 17 January 2025, financial entities can be fined up to 2% of global annual turnover for failures tied to operational resilience and data lineage. A pipeline that cannot show where a record came from and how it was transformed is now a regulatory exposure, not merely a technical one.

None of this is glamorous. There is no single product you buy that makes the latency problem disappear. It is architecture, discipline, and a willingness to stop pretending that batch habits scale into a real-time world.

The banks that get this right share one trait. They stopped treating data as something you store and report on later. They started treating it as something that flows, continuously, and engineered for that reality. The latency problem is rarely a hardware problem. It is a design problem. And design problems, unlike server crashes, can actually be solved for good.

Derek Francis