#spring batch

Spring Batch tip – Partitioning

Basically, Spring Batch Job runs in single thread. To increase the throughput, we need to parallelize the job by partitioning. The core architecture of Job partitioning is as follows.

Partition Step : The wrapper of common Step with a partitioner and grid size (or partition size)

Single Step : Common Step with a Reader, a Processor and a Writer

Partitioner : The core part of partitioning.…

Spring Batch – Read once, Write multi example

Sometimes, we need to read once, write multi ETL with Spring Batch. Spring Batch supports it with CompositeItemWriter (org.springframework.batch.item.support.CompositeItemWriter).

Sample table schema

CREATE TABLE "TB_SOURCE" ( "COL1" VARCHAR2(10 BYTE), "COL2" VARCHAR2(20 BYTE), "COL3" VARCHAR2(20 BYTE) ); CREATE TABLE "TB_TARGET1" ( "NEW_COL1" VARCHAR2(10 BYTE), "NEW_COL2" VARCHAR2(20…

More on Spring Batch Writer

Spring Batch Writer is the implementation of org.springframework.batch.item.ItemWriter. You can write a custom writer, but Spring Batch already has some useful implementations, such as

org.springframework.batch.item.amqp.AmqpItemWriter for AMQP brokers such as RabbitMQ

org.springframework.batch.item.file.FlatFileItemWriter for a file

org.springframework.batch.item.database.JpaItemWriterfor…

More on Spring batch Reader

Spring Batch Job is composed of Reader, Processor and Writer. This post shows some more details on Reader.

Reader is the implementation of org.springframework.batch.item.Reader. I recommend to use the following implementations.

AmqpItemReader (org.springframework.batch.item.amqp.AmqpItemReader) : to interact with queue such as RabbitMQ

FlatFileItemReader(org.springframework.batch.item.file.FlatFi…

Simple ETL with Spring Batch

Spring Batch is literally a batch framework based on Spring Framework. I usually use it to develop a simple ETL(Extraction, Transaformation and Loading) program.

In this post, I’ll show you how to write a simple ETL program. (This sample is tested on Spring Batch 3.0.10)

Prerequisites

Database (MySQL or Oracle)

Spring batch context database

Spring batch libraries (spring-batch-core, spring…

Spring Batch Admin override Components

Para sobrescribir componentes de spring batch admin, debemos situar nuestros .xml de configuración bajo la ruta: META-INF/spring/batch/override/resources

Dentro de esta carpeta podemos incluir tantos ficheros de configuración como necesitemos, poniéndoles el nombre que queramos (spring los “mappeara” todos).

En mi ejemplo, he incluido los siguientes ficheros con estas configuraciones:

database-c…

The technical aspects of a Spring Batch comprises of the Language and libraries used. Libraries consists of JobLauncher, Job, Step, JobRepository, ItemReader, ItemProcessor, ItemWriter.

A Job is launched by a JobLauncher. A Job consists of multiple steps. Each step has a Reader/Processor/Writer. Metadata about the process is stored on a JobRepository.

Job

Spring Job represents entire batch process. A job contains steps which are executed within a batch. It stands at the top of the table in the spring batch processing. A wrapper/container for steps to be executed one by one or simultaneously. 

Job can be defined in an xml file listing the steps to be executed. Job has a Name, Steps, Restartable (Yes/No). execute(...) method executes the Job.

Job Instance

Spring JobInstance represents one iteration of a Job. It's lifecycle consists of multiple failures and ends with one successful run. A successful Job can't be restarted. A failed Job may be restarted depending on other factors.                                        

Consider an example of a Cricket match. A match starts (JobInstance). Runs are scored, players get injured. And then it rains. Players go off the field (Job paused). Now either the rain subsides and same JonInstance starts again. If it rains a lot then No-Result (JobFailure). In this case we can start a match again next day. Either we continue from the score we left off (Start off the Failed Instance). or we start the match again (Start a new JobInstance).

Only one match (JobInstance) between two given teams should be played a given time.

getJobName() give the name of the Job.

getInstanceId() will give the Id for a given JobInstance.

JobParameters

JobParameters  are associated with a given JobInstance. They are the run time parameters passed to a Job. They identify an Job Instance uniquely. They hold the reference data required to run a Job.

In the above cricket match example. A match that starts today starts on day sat 'T'. If it restarts the next day, then it starts on day 'T+1'. 

These Parameters need to be persisted to be able to referenced again for restart etc. 

JobParameters is immutable/thread-safe class. It overrides hascode and equals method to be able to compare between two JobParameters.

JobExecution

JobExecutions contains what exactly happened during a run. For a give JobInstance, if it is run multiple times (with same JobParameters), a unique JobExecution takes placed. Thus JobInstance and JobExecution has one-many relationship.

JobExecution contains information such as StartTime, EndTime, Status, JobInstanceId etc. JobExecution can be stopped by stop().

A typical scenario looks like below,

S No.

JobExecutionId

JobInstanceId

StartTime

EndTime

Status

1

101

12

X

X+1

FAILED

2

102

12

X+2

X+3

COMPLETED

3

103

12

X+4

X+5

COMPLETED

Step

Spring Batch Step contains the configuration of a Step in Batch Processing. When a complete Job is to play a match. First Step can be Toss, 2nd Step can be the Powerplay overs and so on.

Job vs Step

Just like a Job, execute() starts a Step. You can consider it just a baby-step. In an application, first step can be to read the data from a database. Complex steps can be to perform computations on the data and process it. Simple ItemReader/ItemWriter/ItemProcessor to application specific functions are preformed in a Step.

StepExecution

StepExecution is just like JobExecution, but for a Step. Every time a Step is run, a new StepExecution is created. However it contains more details than a JobExecution as proceeding steps depend on this step (unlike JobExecution, when it comprises of entire Job). So it contains transactional data (commit/rollback etc).

ExecutionContext

ExecutionContext consists of the data which is persisted and is required by StepExecution/JobExecution for example, information required to restart.

In case of a failure, we need the state of the last step (or some preceding step). This is stored in format. Both key and values are user defined.

We can store the result of a particular step. For example as below

Consider Execution Context like a generic HashMap. However we have to be careful how much we persist, as it can bubble up Program Memory.

JobRepository

JobRepository is for persisting everything we discussed till now. At the starting, JobExecution is read from JobRepository and intermittently Step/JobExecutions are persisted to JobRepository. JobRepo is omnipresent across the lifecycle of Batch Execution. What is persistence infra for JobRepo depends on implementation. It can be a flat-file/database etc.

ItemReader/ItemWriter/ItemProcessor

Abstractions provided by Spring batch for reading/writing/processing data required for a Step/output from a Step. 

ItemReader has many implementations in Spring Batch to read from Files/Database/Queues etc. Need not be thread-safe as it just reads without modifying data.

ItemWriter has many implementations in Spring Batch to write to Files/Database/Queues etc. Implementing class needs to take care of serialization of the Objects written. 

ItemProcessor has many implementations in Spring Batch as per requirement.

Batch can be driven by incoming File trigger, Database update, JMS (or some other) incoming message. 

In an application I worked on file trigger/JMS message used to initiate and drive the batch at different steps.

1. File Trigger: On the basis of input file and instruction thereof, action can be taken. Spring batch initiates and drives on the basis of the file data.

2. Database update: An incoming Database updates defines what the applications needs to do and how the batch will proceed.