You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fineract.apache.org by GitBox <gi...@apache.org> on 2021/11/18 12:24:40 UTC
[GitHub] [fineract] BLasan opened a new pull request #1972: Feat: Split Interest Posting Job
BLasan opened a new pull request #1972:
URL: https://github.com/apache/fineract/pull/1972
## Description
Split Interest Posting on multiple nodes in parallel.
## Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!
- [x] Write the commit message as per https://github.com/apache/fineract/#pull-requests
- [x] Acknowledge that we will not review PRs that are not passing the build _("green")_ - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.
- [ ] Create/update unit or integration tests for verifying the changes made.
- [x] Follow coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions.
- [x] Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/api-docs/apiLive.htm with details of any API changes
- [x] Submission is not a "code dump". (Large changes can be made "in repository" via a branch. Ask on the developer mailing list for guidance, if required.)
FYI our guidelines for code reviews are at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fineract.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fineract] github-actions[bot] closed pull request #1972: Feat: Split Interest Posting Job
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1972:
URL: https://github.com/apache/fineract/pull/1972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fineract.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fineract] fynmanoj commented on a change in pull request #1972: Feat: Split Interest Posting Job
Posted by GitBox <gi...@apache.org>.
fynmanoj commented on a change in pull request #1972:
URL: https://github.com/apache/fineract/pull/1972#discussion_r752341242
##########
File path: fineract-provider/src/main/java/org/apache/fineract/organisation/office/service/OfficeReadPlatformServiceImpl.java
##########
@@ -50,16 +53,18 @@
private final PlatformSecurityContext context;
private final CurrencyReadPlatformService currencyReadPlatformService;
private final ColumnValidator columnValidator;
+ private final OfficeRepositoryWrapper officeRepositoryWrapper;
private static final String nameDecoratedBaseOnHierarchy = "concat(substring('........................................', 1, ((LENGTH(o.hierarchy) - LENGTH(REPLACE(o.hierarchy, '.', '')) - 1) * 4)), o.name)";
@Autowired
public OfficeReadPlatformServiceImpl(final PlatformSecurityContext context,
final CurrencyReadPlatformService currencyReadPlatformService, final RoutingDataSource dataSource,
- final ColumnValidator columnValidator) {
Review comment:
Avoid repository read in read service, use JDBC query instead.
##########
File path: fineract-provider/src/main/java/org/apache/fineract/organisation/office/domain/Office.java
##########
@@ -191,6 +199,11 @@ public boolean identifiedBy(final Long id) {
return getId().equals(id);
}
+ public List<Office> getChildren() {
+ this.children.size();
Review comment:
reuse loadLazyCollections() method instead.
##########
File path: fineract-provider/src/main/java/org/apache/fineract/organisation/office/domain/OfficeRepositoryWrapper.java
##########
@@ -61,4 +62,9 @@ public Office saveAndFlush(final Office entity) {
public void delete(final Office entity) {
this.repository.delete(entity);
}
+
+ public List<Office> findAllOfficesWithParentId(final Long parentId) {
+ final Office office = findOfficeHierarchy(parentId);
+ return office.getChildren();
Review comment:
Does this fetch all child offices in all levels of hierarchy?
##########
File path: fineract-provider/src/main/java/org/apache/fineract/portfolio/savings/service/SavingsSchedularServiceImpl.java
##########
@@ -65,23 +86,61 @@ public void postInterestForAccounts() throws JobExecutionException {
Integer initialSize = 500;
Integer totalPageSize = 0;
List<Throwable> errors = new ArrayList<>();
- do {
- PageRequest pageRequest = PageRequest.of(page, initialSize);
- Page<SavingsAccount> savingsAccounts = this.savingsAccountRepository.findByStatus(ACTIVE.getValue(), pageRequest);
- for (SavingsAccount savingsAccount : savingsAccounts.getContent()) {
- try {
- this.savingAccountAssembler.assignSavingAccountHelpers(savingsAccount);
- boolean postInterestAsOn = false;
- LocalDate transactionDate = null;
- this.savingsAccountWritePlatformService.postInterest(savingsAccount, postInterestAsOn, transactionDate);
- } catch (Exception e) {
- LOG.error("Failed to post interest for Savings with id {}", savingsAccount.getId(), e);
- errors.add(e);
- }
+
+ if (this.globalConfigurationRepository.findOneByName("allow-split-interest-posting-on-savings").isEnabled()) {
Review comment:
allow-split-interest-posting-on-savings-based-on-office would be more meaningful?
##########
File path: fineract-provider/src/main/java/org/apache/fineract/portfolio/savings/service/SavingsSchedularServiceImpl.java
##########
@@ -65,23 +86,61 @@ public void postInterestForAccounts() throws JobExecutionException {
Integer initialSize = 500;
Integer totalPageSize = 0;
List<Throwable> errors = new ArrayList<>();
- do {
- PageRequest pageRequest = PageRequest.of(page, initialSize);
- Page<SavingsAccount> savingsAccounts = this.savingsAccountRepository.findByStatus(ACTIVE.getValue(), pageRequest);
- for (SavingsAccount savingsAccount : savingsAccounts.getContent()) {
- try {
- this.savingAccountAssembler.assignSavingAccountHelpers(savingsAccount);
- boolean postInterestAsOn = false;
- LocalDate transactionDate = null;
- this.savingsAccountWritePlatformService.postInterest(savingsAccount, postInterestAsOn, transactionDate);
- } catch (Exception e) {
- LOG.error("Failed to post interest for Savings with id {}", savingsAccount.getId(), e);
- errors.add(e);
- }
+
+ if (this.globalConfigurationRepository.findOneByName("allow-split-interest-posting-on-savings").isEnabled()) {
+ final ScheduledJobDetail scheduledJobDetail = this.schedularWritePlatformService.retrieveJobByNameAndNodeId(this.nodeId,
+ JobName.POST_INTEREST_FOR_SAVINGS.toString());
+ final String officeId = this.jobParameterRepository
Review comment:
you dont need to fetch jobparams directly from here. job params are suposed to be injected from jobregister service. Ref: https://github.com/apache/fineract/pull/1971/files#diff-00a8c0bc5ea386690c4ae4abc7d5a8820b2230c01bff72d24cb1c695656ccf21
##########
File path: fineract-provider/src/main/java/org/apache/fineract/portfolio/savings/service/SavingsSchedularServiceImpl.java
##########
@@ -65,23 +86,61 @@ public void postInterestForAccounts() throws JobExecutionException {
Integer initialSize = 500;
Integer totalPageSize = 0;
List<Throwable> errors = new ArrayList<>();
- do {
- PageRequest pageRequest = PageRequest.of(page, initialSize);
- Page<SavingsAccount> savingsAccounts = this.savingsAccountRepository.findByStatus(ACTIVE.getValue(), pageRequest);
- for (SavingsAccount savingsAccount : savingsAccounts.getContent()) {
- try {
- this.savingAccountAssembler.assignSavingAccountHelpers(savingsAccount);
- boolean postInterestAsOn = false;
- LocalDate transactionDate = null;
- this.savingsAccountWritePlatformService.postInterest(savingsAccount, postInterestAsOn, transactionDate);
- } catch (Exception e) {
- LOG.error("Failed to post interest for Savings with id {}", savingsAccount.getId(), e);
- errors.add(e);
- }
+
+ if (this.globalConfigurationRepository.findOneByName("allow-split-interest-posting-on-savings").isEnabled()) {
+ final ScheduledJobDetail scheduledJobDetail = this.schedularWritePlatformService.retrieveJobByNameAndNodeId(this.nodeId,
+ JobName.POST_INTEREST_FOR_SAVINGS.toString());
+ final String officeId = this.jobParameterRepository
+ .findByJobParameterByJobIdAndParameterName(scheduledJobDetail.getId(), "officeId").getParameterValue();
+
+ if (officeId == null) {
+ throw new OfficeIdParameterNotFound(scheduledJobDetail.getId());
}
- page++;
- totalPageSize = savingsAccounts.getTotalPages();
- } while (page < totalPageSize);
+
+ List<Office> offices = this.officeReadPlatformService.retrieveAllOfficesForInterestPostingOnNode(Long.getLong(officeId));
+ for (Office office : offices) {
+ // Load clients data
+ office.loadClients();
+ do {
+ PageRequest pageRequest = PageRequest.of(page, initialSize);
+ Page<SavingsAccount> savingsAccounts = this.savingsAccountRepository.findByStatus(ACTIVE.getValue(), pageRequest);
+ for (SavingsAccount savingsAccount : savingsAccounts.getContent()) {
+ if (!office.getClients().contains(savingsAccount.getClient())) {
+ continue;
+ }
+ try {
+ this.savingAccountAssembler.assignSavingAccountHelpers(savingsAccount);
+ boolean postInterestAsOn = false;
+ LocalDate transactionDate = null;
+ this.savingsAccountWritePlatformService.postInterest(savingsAccount, postInterestAsOn, transactionDate);
+ } catch (Exception e) {
+ LOG.error("Failed to post interest for Savings with id {}", savingsAccount.getId(), e);
+ errors.add(e);
+ }
+ }
+ page++;
+ totalPageSize = savingsAccounts.getTotalPages();
+ } while (page < totalPageSize);
+ }
+ } else {
+ do {
+ PageRequest pageRequest = PageRequest.of(page, initialSize);
+ Page<SavingsAccount> savingsAccounts = this.savingsAccountRepository.findByStatus(ACTIVE.getValue(), pageRequest);
Review comment:
this if..else repeats same code. instead of repeating the code ( which will be difficult to maintain) move the repeating code to a method and reuse.
##########
File path: fineract-provider/src/main/java/org/apache/fineract/portfolio/savings/service/SavingsSchedularServiceImpl.java
##########
@@ -65,23 +86,61 @@ public void postInterestForAccounts() throws JobExecutionException {
Integer initialSize = 500;
Integer totalPageSize = 0;
List<Throwable> errors = new ArrayList<>();
- do {
- PageRequest pageRequest = PageRequest.of(page, initialSize);
- Page<SavingsAccount> savingsAccounts = this.savingsAccountRepository.findByStatus(ACTIVE.getValue(), pageRequest);
- for (SavingsAccount savingsAccount : savingsAccounts.getContent()) {
- try {
- this.savingAccountAssembler.assignSavingAccountHelpers(savingsAccount);
- boolean postInterestAsOn = false;
- LocalDate transactionDate = null;
- this.savingsAccountWritePlatformService.postInterest(savingsAccount, postInterestAsOn, transactionDate);
- } catch (Exception e) {
- LOG.error("Failed to post interest for Savings with id {}", savingsAccount.getId(), e);
- errors.add(e);
- }
+
+ if (this.globalConfigurationRepository.findOneByName("allow-split-interest-posting-on-savings").isEnabled()) {
+ final ScheduledJobDetail scheduledJobDetail = this.schedularWritePlatformService.retrieveJobByNameAndNodeId(this.nodeId,
+ JobName.POST_INTEREST_FOR_SAVINGS.toString());
+ final String officeId = this.jobParameterRepository
+ .findByJobParameterByJobIdAndParameterName(scheduledJobDetail.getId(), "officeId").getParameterValue();
+
+ if (officeId == null) {
+ throw new OfficeIdParameterNotFound(scheduledJobDetail.getId());
}
- page++;
- totalPageSize = savingsAccounts.getTotalPages();
- } while (page < totalPageSize);
+
+ List<Office> offices = this.officeReadPlatformService.retrieveAllOfficesForInterestPostingOnNode(Long.getLong(officeId));
+ for (Office office : offices) {
+ // Load clients data
+ office.loadClients();
+ do {
+ PageRequest pageRequest = PageRequest.of(page, initialSize);
+ Page<SavingsAccount> savingsAccounts = this.savingsAccountRepository.findByStatus(ACTIVE.getValue(), pageRequest);
+ for (SavingsAccount savingsAccount : savingsAccounts.getContent()) {
+ if (!office.getClients().contains(savingsAccount.getClient())) {
Review comment:
This action is very costly. This loop will visit all accounts , which is not needed. add office param to the query and fetch just the relevant records
##########
File path: fineract-provider/src/main/resources/sql/migrations/core_db/V376__split_interest_posting_job.sql
##########
@@ -0,0 +1,23 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+--
+
+insert into c_configuration(name, enabled, value, description) values('allow-split-interest-posting-on-savings', 0, 0, 'Split Interest Posting job on multiple nodes');
+
+-- Need to add officeIds for all nodes
+insert into job_parameters(job_id, parameter_name, parameter_value) values(16, 'officeId', 1);
Review comment:
donot use id here. use a select query instead
insert into job_parameters(job_id, parameter_name, parameter_value) values((select id from job where name = 'interest posting .....'), 'officeId', 1);
##########
File path: fineract-provider/src/main/java/org/apache/fineract/organisation/office/service/OfficeReadPlatformServiceImpl.java
##########
@@ -187,6 +192,12 @@ public OfficeTransactionData mapRow(final ResultSet rs, @SuppressWarnings("unuse
return this.jdbcTemplate.query(sql, rm, new Object[] { hierarchySearchString });
}
+ @Override
+ public List<Office> retrieveAllOfficesForInterestPostingOnNode(final Long parentId) {
+ this.context.authenticatedUser();
+ return this.officeRepositoryWrapper.findAllOfficesWithParentId(parentId);
Review comment:
Avoid repository read in read service, use JDBC query instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fineract.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fineract] BLasan commented on pull request #1972: Feat: Split Interest Posting Job
Posted by GitBox <gi...@apache.org>.
BLasan commented on pull request #1972:
URL: https://github.com/apache/fineract/pull/1972#issuecomment-972819361
@fynmanoj Please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fineract.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fineract] github-actions[bot] commented on pull request #1972: Feat: Split Interest Posting Job
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1972:
URL: https://github.com/apache/fineract/pull/1972#issuecomment-997303384
This pull request seems to be stale. Are you still planning to work on it? We will automatically close it in 30 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fineract.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org