Spring Batch integration

[Arnold Galovics <ar...@apache.org>]

Dear all,

I'd like to introduce you to the idea of enhancing Fineract with the
capability to handle high-volume batch jobs along with a few other things.

As any financial system, Fineract also supports batch jobs but they're
quite limited on the data volume it can handle. Currently a batch job is a
single Java method with loading everything at once and processing that huge
chunk of data in a single execution step/transaction.

To make Fineract better, we need to alter this simplified framework a
little bit and introduce something that's more battle-tested: the Spring
Batch module is here to help.

With introducing Spring Batch into the mix, we'll finally have a chance to
decouple data reading, processing and data writing that'll finally bring us
the benefit of making our batch jobs:
- faster
- reliable
- being able to handle huge data volumes

And I'm not gonna just stop there, with Spring Batch in place, we'll be
able to distribute our batch job workloads into separate JVMs in a scenario
where we want to run a cluster of Fineract instances to cope with the load.

The next thing to the Spring Batch module integration is the introduction
of a new batch job, Loan Close Of Business; that'll be the starting point
of reworking the date handling in Fineract. This job will be simply
responsible to "close" Loans on a daily basis making sure that
interests/fees/etc are calculated properly. It's going to be the first
full-fledged job that'll be able to run in a partitioned mode on a cluster
of JVMs and multiple threads.

The job will be rolled out as disabled by default so it won't interfere
with any existing deployments but if somebody will need it's potential and
business functionality, they can certainly enable it. Also, there's an
extensive documentation I already wrote for this; right now it's in PR for
some review and I'm gonna do a few additional modifications but you can
check it out here: https://github.com/apache/fineract/pull/2326

I don't want to repeat everything that's in the PR so if you're curious be
sure to check it.

TL;DR; Spring Batch is going to be integrated into Fineract. A new batch
job called Loan Close Of Business will be created along with some
additional functionality that's described in the PR above.

Best,
Arnold

Re: Spring Batch integration

[Arnold Galovics <ar...@apache.org>]

Hi Frank,

Hard to tell since I don't have the target date for 1.8. I can already tell
this is gonna be a huge work and probably will not be finished during the
timeframe of a single release (until 1.8).

Best,
Arnold

On Mon, May 16, 2022 at 2:17 PM Frank Nkuyahaga <ha...@gmail.com> wrote:

> Thanks Arnold, sound interesting. Is this on the roadmap for Finearct 1.8?
>
> On 16 May 2022, at 15:04, Arnold Galovics <ar...@apache.org> wrote:
>
> Dear all,
>
> I'd like to introduce you to the idea of enhancing Fineract with the
> capability to handle high-volume batch jobs along with a few other things.
>
> As any financial system, Fineract also supports batch jobs but they're
> quite limited on the data volume it can handle. Currently a batch job is a
> single Java method with loading everything at once and processing that huge
> chunk of data in a single execution step/transaction.
>
> To make Fineract better, we need to alter this simplified framework a
> little bit and introduce something that's more battle-tested: the Spring
> Batch module is here to help.
>
> With introducing Spring Batch into the mix, we'll finally have a chance to
> decouple data reading, processing and data writing that'll finally bring us
> the benefit of making our batch jobs:
> - faster
> - reliable
> - being able to handle huge data volumes
>
> And I'm not gonna just stop there, with Spring Batch in place, we'll be
> able to distribute our batch job workloads into separate JVMs in a scenario
> where we want to run a cluster of Fineract instances to cope with the load.
>
> The next thing to the Spring Batch module integration is the introduction
> of a new batch job, Loan Close Of Business; that'll be the starting point
> of reworking the date handling in Fineract. This job will be simply
> responsible to "close" Loans on a daily basis making sure that
> interests/fees/etc are calculated properly. It's going to be the first
> full-fledged job that'll be able to run in a partitioned mode on a cluster
> of JVMs and multiple threads.
>
> The job will be rolled out as disabled by default so it won't interfere
> with any existing deployments but if somebody will need it's potential and
> business functionality, they can certainly enable it. Also, there's an
> extensive documentation I already wrote for this; right now it's in PR for
> some review and I'm gonna do a few additional modifications but you can
> check it out here: https://github.com/apache/fineract/pull/2326
>
> I don't want to repeat everything that's in the PR so if you're curious be
> sure to check it.
>
> TL;DR; Spring Batch is going to be integrated into Fineract. A new batch
> job called Loan Close Of Business will be created along with some
> additional functionality that's described in the PR above.
>
> Best,
> Arnold
>
>
>

Re: Spring Batch integration

[Frank Nkuyahaga <ha...@gmail.com>]

Thanks Arnold, sound interesting. Is this on the roadmap for Finearct 1.8?

> On 16 May 2022, at 15:04, Arnold Galovics <ar...@apache.org> wrote:
> 
> Dear all,
> 
> I'd like to introduce you to the idea of enhancing Fineract with the capability to handle high-volume batch jobs along with a few other things.
> 
> As any financial system, Fineract also supports batch jobs but they're quite limited on the data volume it can handle. Currently a batch job is a single Java method with loading everything at once and processing that huge chunk of data in a single execution step/transaction.
> 
> To make Fineract better, we need to alter this simplified framework a little bit and introduce something that's more battle-tested: the Spring Batch module is here to help.
> 
> With introducing Spring Batch into the mix, we'll finally have a chance to decouple data reading, processing and data writing that'll finally bring us the benefit of making our batch jobs:
> - faster
> - reliable
> - being able to handle huge data volumes
> 
> And I'm not gonna just stop there, with Spring Batch in place, we'll be able to distribute our batch job workloads into separate JVMs in a scenario where we want to run a cluster of Fineract instances to cope with the load.
> 
> The next thing to the Spring Batch module integration is the introduction of a new batch job, Loan Close Of Business; that'll be the starting point of reworking the date handling in Fineract. This job will be simply responsible to "close" Loans on a daily basis making sure that interests/fees/etc are calculated properly. It's going to be the first full-fledged job that'll be able to run in a partitioned mode on a cluster of JVMs and multiple threads.
> 
> The job will be rolled out as disabled by default so it won't interfere with any existing deployments but if somebody will need it's potential and business functionality, they can certainly enable it. Also, there's an extensive documentation I already wrote for this; right now it's in PR for some review and I'm gonna do a few additional modifications but you can check it out here: https://github.com/apache/fineract/pull/2326 <https://github.com/apache/fineract/pull/2326>
> 
> I don't want to repeat everything that's in the PR so if you're curious be sure to check it.
> 
> TL;DR; Spring Batch is going to be integrated into Fineract. A new batch job called Loan Close Of Business will be created along with some additional functionality that's described in the PR above.
> 
> Best,
> Arnold
> 

Re: Spring Batch integration

[Arnold Galovics <ar...@apache.org>]

Hi Pramod,

I see. That's part of the Quartz scheduling logic and we're not planning to
touch that at all so we should be good.

Best,
Arnold

On Tue, May 17, 2022 at 6:29 PM Pramod Nuthakki <pr...@finflux.co> wrote:

> @Arnold: I am talking about controlling the job execution within grouped
> jobs (ref scheduler_group column in job table). Where the system used to
> execute all the grouped jobs in a single thread as per execution time and
> priorities. So that all other jobs will wait in queue till the currently
> executing job finishes.
>
> Thanks,
> Pramod
>
> On Tue, May 17, 2022 at 9:13 PM Arnold Galovics <ga...@gmail.com>
> wrote:
>
>> Thanks Ed.
>>
>> @Pramod: when you talk about dependency, I assume you're talking about an
>> implicit dependency maintained by the cron expression that's attached to
>> the jobs, right? If so, then there's nothing to worry about. The Quartz
>> scheduling won't be touched and the schedules will remain as they are.
>>
>> Best,
>> Arnold
>>
>> On Tue, May 17, 2022 at 5:29 PM Ed Cable <ed...@mifos.org> wrote:
>>
>>> Arnold,
>>>
>>> Thank you for sharing the in progress design documentation. I have been
>>> forwarding it to some other individuals for their feedback.
>>>
>>> @Pramod Nuthakki <pr...@finflux.co> had the following question
>>> regarding jobs dependent on other jobs.
>>>
>>> "Currently  few scheduled jobs have dependency on other  jobs like NPA
>>> depends on arrears job, Accruals has dependency  on NPA jobs etc... Once we
>>> move the jobs to batch processing then how the above mentioned dependency
>>> will be addressed."
>>>
>>> Thanks,
>>>
>>> Ed
>>>
>>>
>>> On Mon, May 16, 2022, 05:04 Arnold Galovics <ar...@apache.org> wrote:
>>>
>>>> Dear all,
>>>>
>>>> I'd like to introduce you to the idea of enhancing Fineract with the
>>>> capability to handle high-volume batch jobs along with a few other things.
>>>>
>>>> As any financial system, Fineract also supports batch jobs but they're
>>>> quite limited on the data volume it can handle. Currently a batch job is a
>>>> single Java method with loading everything at once and processing that huge
>>>> chunk of data in a single execution step/transaction.
>>>>
>>>> To make Fineract better, we need to alter this simplified framework a
>>>> little bit and introduce something that's more battle-tested: the Spring
>>>> Batch module is here to help.
>>>>
>>>> With introducing Spring Batch into the mix, we'll finally have a chance
>>>> to decouple data reading, processing and data writing that'll finally bring
>>>> us the benefit of making our batch jobs:
>>>> - faster
>>>> - reliable
>>>> - being able to handle huge data volumes
>>>>
>>>> And I'm not gonna just stop there, with Spring Batch in place, we'll be
>>>> able to distribute our batch job workloads into separate JVMs in a scenario
>>>> where we want to run a cluster of Fineract instances to cope with the load.
>>>>
>>>> The next thing to the Spring Batch module integration is the
>>>> introduction of a new batch job, Loan Close Of Business; that'll be the
>>>> starting point of reworking the date handling in Fineract. This job will be
>>>> simply responsible to "close" Loans on a daily basis making sure that
>>>> interests/fees/etc are calculated properly. It's going to be the first
>>>> full-fledged job that'll be able to run in a partitioned mode on a cluster
>>>> of JVMs and multiple threads.
>>>>
>>>> The job will be rolled out as disabled by default so it won't interfere
>>>> with any existing deployments but if somebody will need it's potential and
>>>> business functionality, they can certainly enable it. Also, there's an
>>>> extensive documentation I already wrote for this; right now it's in PR for
>>>> some review and I'm gonna do a few additional modifications but you can
>>>> check it out here: https://github.com/apache/fineract/pull/2326
>>>>
>>>> I don't want to repeat everything that's in the PR so if you're curious
>>>> be sure to check it.
>>>>
>>>> TL;DR; Spring Batch is going to be integrated into Fineract. A new
>>>> batch job called Loan Close Of Business will be created along with some
>>>> additional functionality that's described in the PR above.
>>>>
>>>> Best,
>>>> Arnold
>>>>
>>>>

Re: Spring Batch integration

[Pramod Nuthakki <pr...@finflux.co>]

@Arnold: I am talking about controlling the job execution within grouped
jobs (ref scheduler_group column in job table). Where the system used to
execute all the grouped jobs in a single thread as per execution time and
priorities. So that all other jobs will wait in queue till the currently
executing job finishes.

Thanks,
Pramod

On Tue, May 17, 2022 at 9:13 PM Arnold Galovics <ga...@gmail.com>
wrote:

> Thanks Ed.
>
> @Pramod: when you talk about dependency, I assume you're talking about an
> implicit dependency maintained by the cron expression that's attached to
> the jobs, right? If so, then there's nothing to worry about. The Quartz
> scheduling won't be touched and the schedules will remain as they are.
>
> Best,
> Arnold
>
> On Tue, May 17, 2022 at 5:29 PM Ed Cable <ed...@mifos.org> wrote:
>
>> Arnold,
>>
>> Thank you for sharing the in progress design documentation. I have been
>> forwarding it to some other individuals for their feedback.
>>
>> @Pramod Nuthakki <pr...@finflux.co> had the following question
>> regarding jobs dependent on other jobs.
>>
>> "Currently  few scheduled jobs have dependency on other  jobs like NPA
>> depends on arrears job, Accruals has dependency  on NPA jobs etc... Once we
>> move the jobs to batch processing then how the above mentioned dependency
>> will be addressed."
>>
>> Thanks,
>>
>> Ed
>>
>>
>> On Mon, May 16, 2022, 05:04 Arnold Galovics <ar...@apache.org> wrote:
>>
>>> Dear all,
>>>
>>> I'd like to introduce you to the idea of enhancing Fineract with the
>>> capability to handle high-volume batch jobs along with a few other things.
>>>
>>> As any financial system, Fineract also supports batch jobs but they're
>>> quite limited on the data volume it can handle. Currently a batch job is a
>>> single Java method with loading everything at once and processing that huge
>>> chunk of data in a single execution step/transaction.
>>>
>>> To make Fineract better, we need to alter this simplified framework a
>>> little bit and introduce something that's more battle-tested: the Spring
>>> Batch module is here to help.
>>>
>>> With introducing Spring Batch into the mix, we'll finally have a chance
>>> to decouple data reading, processing and data writing that'll finally bring
>>> us the benefit of making our batch jobs:
>>> - faster
>>> - reliable
>>> - being able to handle huge data volumes
>>>
>>> And I'm not gonna just stop there, with Spring Batch in place, we'll be
>>> able to distribute our batch job workloads into separate JVMs in a scenario
>>> where we want to run a cluster of Fineract instances to cope with the load.
>>>
>>> The next thing to the Spring Batch module integration is the
>>> introduction of a new batch job, Loan Close Of Business; that'll be the
>>> starting point of reworking the date handling in Fineract. This job will be
>>> simply responsible to "close" Loans on a daily basis making sure that
>>> interests/fees/etc are calculated properly. It's going to be the first
>>> full-fledged job that'll be able to run in a partitioned mode on a cluster
>>> of JVMs and multiple threads.
>>>
>>> The job will be rolled out as disabled by default so it won't interfere
>>> with any existing deployments but if somebody will need it's potential and
>>> business functionality, they can certainly enable it. Also, there's an
>>> extensive documentation I already wrote for this; right now it's in PR for
>>> some review and I'm gonna do a few additional modifications but you can
>>> check it out here: https://github.com/apache/fineract/pull/2326
>>>
>>> I don't want to repeat everything that's in the PR so if you're curious
>>> be sure to check it.
>>>
>>> TL;DR; Spring Batch is going to be integrated into Fineract. A new batch
>>> job called Loan Close Of Business will be created along with some
>>> additional functionality that's described in the PR above.
>>>
>>> Best,
>>> Arnold
>>>
>>>

Re: Spring Batch integration

[Arnold Galovics <ga...@gmail.com>]

Thanks Ed.

@Pramod: when you talk about dependency, I assume you're talking about an
implicit dependency maintained by the cron expression that's attached to
the jobs, right? If so, then there's nothing to worry about. The Quartz
scheduling won't be touched and the schedules will remain as they are.

Best,
Arnold

On Tue, May 17, 2022 at 5:29 PM Ed Cable <ed...@mifos.org> wrote:

> Arnold,
>
> Thank you for sharing the in progress design documentation. I have been
> forwarding it to some other individuals for their feedback.
>
> @Pramod Nuthakki <pr...@finflux.co> had the following question regarding
> jobs dependent on other jobs.
>
> "Currently  few scheduled jobs have dependency on other  jobs like NPA
> depends on arrears job, Accruals has dependency  on NPA jobs etc... Once we
> move the jobs to batch processing then how the above mentioned dependency
> will be addressed."
>
> Thanks,
>
> Ed
>
>
> On Mon, May 16, 2022, 05:04 Arnold Galovics <ar...@apache.org> wrote:
>
>> Dear all,
>>
>> I'd like to introduce you to the idea of enhancing Fineract with the
>> capability to handle high-volume batch jobs along with a few other things.
>>
>> As any financial system, Fineract also supports batch jobs but they're
>> quite limited on the data volume it can handle. Currently a batch job is a
>> single Java method with loading everything at once and processing that huge
>> chunk of data in a single execution step/transaction.
>>
>> To make Fineract better, we need to alter this simplified framework a
>> little bit and introduce something that's more battle-tested: the Spring
>> Batch module is here to help.
>>
>> With introducing Spring Batch into the mix, we'll finally have a chance
>> to decouple data reading, processing and data writing that'll finally bring
>> us the benefit of making our batch jobs:
>> - faster
>> - reliable
>> - being able to handle huge data volumes
>>
>> And I'm not gonna just stop there, with Spring Batch in place, we'll be
>> able to distribute our batch job workloads into separate JVMs in a scenario
>> where we want to run a cluster of Fineract instances to cope with the load.
>>
>> The next thing to the Spring Batch module integration is the introduction
>> of a new batch job, Loan Close Of Business; that'll be the starting point
>> of reworking the date handling in Fineract. This job will be simply
>> responsible to "close" Loans on a daily basis making sure that
>> interests/fees/etc are calculated properly. It's going to be the first
>> full-fledged job that'll be able to run in a partitioned mode on a cluster
>> of JVMs and multiple threads.
>>
>> The job will be rolled out as disabled by default so it won't interfere
>> with any existing deployments but if somebody will need it's potential and
>> business functionality, they can certainly enable it. Also, there's an
>> extensive documentation I already wrote for this; right now it's in PR for
>> some review and I'm gonna do a few additional modifications but you can
>> check it out here: https://github.com/apache/fineract/pull/2326
>>
>> I don't want to repeat everything that's in the PR so if you're curious
>> be sure to check it.
>>
>> TL;DR; Spring Batch is going to be integrated into Fineract. A new batch
>> job called Loan Close Of Business will be created along with some
>> additional functionality that's described in the PR above.
>>
>> Best,
>> Arnold
>>
>>

Re: Spring Batch integration

[Ed Cable <ed...@mifos.org>]

Arnold,

Thank you for sharing the in progress design documentation. I have been
forwarding it to some other individuals for their feedback.

@Pramod Nuthakki <pr...@finflux.co> had the following question regarding
jobs dependent on other jobs.

"Currently  few scheduled jobs have dependency on other  jobs like NPA
depends on arrears job, Accruals has dependency  on NPA jobs etc... Once we
move the jobs to batch processing then how the above mentioned dependency
will be addressed."

Thanks,

Ed


On Mon, May 16, 2022, 05:04 Arnold Galovics <ar...@apache.org> wrote:

> Dear all,
>
> I'd like to introduce you to the idea of enhancing Fineract with the
> capability to handle high-volume batch jobs along with a few other things.
>
> As any financial system, Fineract also supports batch jobs but they're
> quite limited on the data volume it can handle. Currently a batch job is a
> single Java method with loading everything at once and processing that huge
> chunk of data in a single execution step/transaction.
>
> To make Fineract better, we need to alter this simplified framework a
> little bit and introduce something that's more battle-tested: the Spring
> Batch module is here to help.
>
> With introducing Spring Batch into the mix, we'll finally have a chance to
> decouple data reading, processing and data writing that'll finally bring us
> the benefit of making our batch jobs:
> - faster
> - reliable
> - being able to handle huge data volumes
>
> And I'm not gonna just stop there, with Spring Batch in place, we'll be
> able to distribute our batch job workloads into separate JVMs in a scenario
> where we want to run a cluster of Fineract instances to cope with the load.
>
> The next thing to the Spring Batch module integration is the introduction
> of a new batch job, Loan Close Of Business; that'll be the starting point
> of reworking the date handling in Fineract. This job will be simply
> responsible to "close" Loans on a daily basis making sure that
> interests/fees/etc are calculated properly. It's going to be the first
> full-fledged job that'll be able to run in a partitioned mode on a cluster
> of JVMs and multiple threads.
>
> The job will be rolled out as disabled by default so it won't interfere
> with any existing deployments but if somebody will need it's potential and
> business functionality, they can certainly enable it. Also, there's an
> extensive documentation I already wrote for this; right now it's in PR for
> some review and I'm gonna do a few additional modifications but you can
> check it out here: https://github.com/apache/fineract/pull/2326
>
> I don't want to repeat everything that's in the PR so if you're curious be
> sure to check it.
>
> TL;DR; Spring Batch is going to be integrated into Fineract. A new batch
> job called Loan Close Of Business will be created along with some
> additional functionality that's described in the PR above.
>
> Best,
> Arnold
>
>