You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fineract.apache.org by Isaac Kamga <u2...@gmail.com> on 2019/09/01 23:52:18 UTC
Re: Testing ORM Migration to EclipseLink
Hello Graham,
That's some very good work you've done so far.
I have executed the script and everything works well on my end right
through to the fims-web-app.
EclipseLink configurations was merged into PostgreSQL library last week and
the changes to fineract-cn-test library on FINCN-114 which migrate it from
MariaDB to PostgreSQL have been merged into the develop branch. The latter
causes the Travis checks on the other repositories to succeed.
However, the Thymeleaf templating PR heading to Notifications develop
branch needs your attention.
More grease to your elbows.
Cheers,
Isaac Kamga.
On Sun, Aug 25, 2019 at 2:43 PM Ebenezer Graham <eg...@alustudent.com>
wrote:
> Hello Devs,
>
> I would like to revive the topic discussed in the previous mail.
>
> I have been running Fineract CN on EclipseLink and PostgreSQL for a
> while now and I believe it's ready to be merged into the develop branches.
> But before that, it is paramount that other experienced developers in the
> community examine what has been done so far and QA it.
>
> Simply execute this script
> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-141/scripts/Unix/eclipselink-test-setup.sh to
> get all the necessary branches.
>
> *PS: *I have experienced better performance with PostgreSQL and
> EclipseLink on my workstation.
>
> For instance, Application Startup + Provisioning took 14 mins
> Application Startup alone took 12 mins
>
> With MariaDB and Hibernate out of the way, we will be closer to making
> Fineract CN compliant. On top of that, if we are able to upgrade the spring
> versions, this should significantly improve Fineract CN's response time,
> performance and memory usage.
>
> Looking forward to hearing from you.
>
> Regards.
>
>
> On Tue, 16 Jul 2019 at 04:47, Ebenezer Graham <eg...@alustudent.com>
> wrote:
>
>> Hello Fineracters,
>>
>> I just finished integrating EclipseLink into Fineract CN. I would like to
>> request your participation in the testing of the new ORM. Your constructive
>> feedback will help me to fine-tune the integration and resolve all known
>> issues with PostgreSQL.
>>
>> *Approach Used *
>> Migrating from Hibernate to EclipseLink in Spring is actually very simple
>> and seamless.
>>
>> 1. I simply removed hibernate core and entity-manager from
>> spring-data-jpa [1]
>> 2. Then, I added the recommended EclipseLink JPA dependency [2] to the
>> MariaDB and PostgreSQL libraries.
>> 4. I extended the JpaBaseConfiguration and configured it to use
>> EclipseLink as the provider. As discussed earlier, Spring provides an
>> adapter for EclipseLink so it was quite straightforward.
>> 5. One critical decision I made, was choosing static weaving over the
>> default dynamic weaving done at runtime. Static weaving improves
>> performance and stability as it occurs during build-time. In addition, this
>> approach is recommended for large applications with lots of entities. Read
>> more at [3] and let me know what you think.
>>
>> Please conduct your tests and report any issues on this thread so that I
>> can work on them.
>>
>> *Testing EclipseLink and PostgreSQL*
>> You can test EclipseLink and PostgreSQL by using this script [4]. The
>> eclipselink-test-setup script will clone and publish the relevant artefacts.
>>
>> NB: Remember to shut down your Postgres server when doing component tests
>> so that the embedded db can start.
>>
>> NB: Use this for full integration tests
>> java -Ddemoserver.persistent=true -Dcustom.cassandra.contactPoints=
>> 127.0.0.1:9042 -Dcassandra.cluster.user=cassandra
>> -Dcassandra.cluster.pwd=password -Dcustom.postgresql.host=localhost
>> -Dcustom.postgresql.user=postgres -jar
>> ~/<fineract-cn-dir>/integration-tests/fineract-cn-demo-server/build/libs/demo-server-0.1.0-BUILD-SNAPSHOT.jar
>>
>> NB: The logs are a bit excessive as I'm currently outputting all
>> EclipseLink logs but once testing is over, I will tone it down.
>>
>> I have conducted component test for each microservice and a full
>> integration test. A few tests failed during the component test however, the
>> causes are not from the ORM but the test cases themselves. I will try and
>> fix them once I have met all of my milestones for this project or raise
>> issues for them in Jira.
>>
>> *Known Issues in Postgres (My current focus):*
>> 1. ERROR: relation "khepri_beats" does not exist. click on the link at
>> [5] to see the full log. This issue is not directly related to the ORM. I
>> actually identified this error whiles testing Postgres and Hibernate. This
>> is related to Postgres and I am currently looking into it.
>>
>> 2. Error: org.postgresql.util.PSQLException: FATAL: sorry, too many
>> clients already. click on the link [6] to see the full log.
>>
>> *Solution*, Increase max_connections found in the postgresql.conf. *260*
>> worked for me.
>> Rational, each microservice can have up to 20 unique connections to the
>> db per the BoneCP configuration.
>>
>> Also, you can test EclipseLink and MariaDB by pulling the FINCN-2 branch
>> into fineract-cn-mariadb link [7] and rebuilding all of the microservices.
>>
>> *Identified Issue when using embedded dbs in full integration test *
>> Tenant context not set, resulting in this error [8]
>>
>> Which brings me to my concern about MariaDB. Given that MariaDB is not
>> compliant with Apache, can I safely assume that it's not relevant to
>> migrate it's ORM as well?
>>
>> Looking forward to hearing from you all.
>>
>> *At your service,*
>> Graham.
>>
>> [1]
>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L76-L80
>> [2]
>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L74
>> / https://www.baeldung.com/spring-eclipselink
>> [3]
>> https://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Static_Weaving
>> https://pastebin.com/FDAzd6bx
>> [4]
>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-2/scripts/Unix/eclipselink-test-setup.sh
>> [5] https://pastebin.com/WFQ9satu
>> [6] https://pastebin.com/M8GAHsit
>> [7] https://github.com/ebenezergraham/fineract-cn-mariadb/tree/FINCN-2
>> [8] https://pastebin.com/FDAzd6bx
>>
>
Re: Testing ORM Migration to EclipseLink
Posted by Juhan Aasaru <aa...@gmail.com>.
I managed to overcome this problem myself.
I think it was caused by following value in Dockerfile:
rhythm.beatCheckRate=10
When I replaced it with
rhythm.beatCheckRate=600000
the problem disappeared.
Juhan
Kontakt Juhan Aasaru (<aa...@gmail.com>) kirjutas kuupäeval K, 18.
september 2019 kell 14:18:
> Hi!
>
> Sorry about the late reply.
>
> 1. In my opinion it is not question about provisioning as as soon as the
> rhythym application starts up, it tries to select from khepri_beats (see
> logs below).
> I'm not using demo server at all. I use Postman scripts for provisioning.
> I wonder if I should run some step before starting rhythm micro service.
> I get the error on both with latest docker image of rhythm as well as when
> starting the service locally.
> I logged bug report here: https://issues.apache.org/jira/browse/FINCN-177
>
> Here are the logs when running from localhost
>
> 13:56:54.948 [main] INFO o.s.b.c.e.j.JettyEmbeddedServletContainer -
> Jetty started on port(s) 2022 (http/1.1)
> 13:56:54.958 [main] INFO o.a.f.c.r.service.RhythmApplication - Started
> RhythmApplication in 13.207 seconds (JVM running for 14.079)
> [EL Finer]: transaction: 2019-09-18
> 13:56:54.998--UnitOfWork(641160635)--Thread(Thread[pool-2-thread-1,5,main])--begin
> unit of work flush
> [EL Finer]: transaction: 2019-09-18
> 13:56:54.999--UnitOfWork(641160635)--Thread(Thread[pool-2-thread-1,5,main])--end
> unit of work flush
> [EL Finest]: query: 2019-09-18
> 13:56:54.999--UnitOfWork(641160635)--Thread(Thread[pool-2-thread-1,5,main])--Execute
> query ReadAllQuery(referenceClass=BeatEntity )
> [EL Finest]: connection: 2019-09-18
> 13:56:55.013--ServerSession(988179589)--Connection(1724597978)--Thread(Thread[pool-2-thread-1,5,main])--Connection
> acquired from connection pool [default].
> [EL Finer]: transaction: 2019-09-18
> 13:56:55.013--ClientSession(571004194)--Connection(1724597978)--Thread(Thread[pool-2-thread-1,5,main])--begin
> transaction
> [EL Finest]: connection: 2019-09-18
> 13:56:55.013--ClientSession(571004194)--Thread(Thread[pool-2-thread-1,5,main])--reconnecting
> to external connection pool
> [EL Fine]: sql: 2019-09-18
> 13:56:55.014--ClientSession(571004194)--Connection(1203869313)--Thread(Thread[pool-2-thread-1,5,main])--SELECT
> id, alignment_hour, application_identifier, beat_identifier, next_beat,
> tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR UPDATE
> bind => [2019-09-18 13:56:54.923]
> [EL Fine]: sql: 2019-09-18
> 13:56:55.053--ClientSession(571004194)--Thread(Thread[pool-2-thread-1,5,main])--SELECT
> 1
> [EL Warning]: 2019-09-18
> 13:56:55.055--UnitOfWork(641160635)--Thread(Thread[pool-2-thread-1,5,main])--Local
> Exception Stack:
> Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.6.4.v20160829-44060b6):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: org.postgresql.util.PSQLException: ERROR: relation
> "khepri_beats" does not exist
> Position: 103
> Error Code: 0
> Call: SELECT id, alignment_hour, application_identifier, beat_identifier,
> next_beat, tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR
> UPDATE
> bind => [2019-09-18 13:56:54.923]
> Query: ReadAllQuery(referenceClass=BeatEntity sql="SELECT id,
> alignment_hour, application_identifier, beat_identifier, next_beat,
> tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR UPDATE")
> at
> org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
>
> Regards
> Juhan
>
> Kontakt Ebenezer Graham (<eg...@alustudent.com>) kirjutas kuupäeval
> R, 6. september 2019 kell 14:28:
>
>> Thanks for the update,
>>
>> 1. For the error below, I believe that is because you didn't provision
>> the microservices. Can you please confirm if you set
>> -Ddemoserver.provision=true on the first run. Also, you can use
>> https://github.com/apache/fineract-cn-demo-server/blob/develop/scripts/Unix/initial-setup.sh to
>> test the integration. I have deleted a few of the branches in the
>> eclipselink-test-script.sh
>>
>> 1. Rhythm keeps giving me this error:
>>> rhythm-ms_1 | Internal Exception:
>>> org.postgresql.util.PSQLException: ERROR: relation "khepri_beats" does not
>>> exist
>>> rhythm-ms_1 | Position: 103
>>
>>
>> 2. I will make the make the changes and submit the necessary PRs asap.
>>
>> 3. I have updated the Setup Guide to cater for PostgreSQL. However, I
>> need someone to test the MacOS instructions work correctly.
>>
>> Regards
>>
>> On Wed, 4 Sep 2019 at 15:54, Juhan Aasaru <aa...@gmail.com> wrote:
>>
>>> Hi Graham and Isaac!
>>>
>>> Great progress and excellent work regarding Postgresql and EclipseLink
>>> migration!
>>> I did some tests (using the official docker images I'm working on). This
>>> is what I found
>>>
>>> 1. Rhythm keeps giving me this error:
>>> rhythm-ms_1 | Internal Exception:
>>> org.postgresql.util.PSQLException: ERROR: relation "khepri_beats" does not
>>> exist
>>> rhythm-ms_1 | Position: 103
>>> rhythm-ms_1 | Error Code: 0
>>> rhythm-ms_1 | Call: SELECT id, alignment_hour,
>>> application_identifier, beat_identifier, next_beat, tenant_identifier FROM
>>> khepri_beats WHERE (next_beat < ?) FOR UPDATE
>>> rhythm-ms_1 | bind => [2019-09-04 11:34:43.778]
>>> rhythm-ms_1 | Query: ReadAllQuery(referenceClass=BeatEntity
>>> sql="SELECT id, alignment_hour, application_identifier, beat_identifier,
>>> next_beat, tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR
>>> UPDATE")
>>>
>>> 2. many services (provisioner, office, customer, accounting, teller,
>>> reporting, payroll) now define data source properties twice
>>> in service/src/main/resources/application.yaml
>>> The problem is that if I want to overwrite some value I have to pass it
>>> in as two separate parameters.
>>>
>>> For example:
>>> https://github.com/apache/fineract-cn-provisioner/commit/4327ce032c97cc961ce15e39c22863654cefc230
>>> has:
>>>
>>> postgresql:
>>> driverClass: org.postgresql.Driver
>>> database: seshat
>>> host: localhost
>>> port: 5432
>>> user: postgres
>>> password: postgres
>>>
>>> spring:
>>> datasource:
>>> driver-class-name: org.postgresql.Driver
>>> url: jdbc:postgresql://localhost:5432/seshat
>>> username: postgres
>>> password: postgres
>>>
>>> Please change the last block to something like this:
>>>
>>> spring:
>>> datasource:
>>> driver-class-name: org.postgresql.Driver
>>> url: jdbc:postgresql://${postgresql.host}:${postgresql.port}/seshat
>>> username: ${postgresql.user}
>>> password: ${postgresql.password}
>>>
>>> Then there should be no need to pass in the variables twice if I want to
>>> change some value.
>>>
>>> 3. Instructions should be updated as well and MariaDB/Mysql replaced
>>> with Postgresql:
>>>
>>> https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN
>>>
>>> Also Vishwas has some good instructions that need to be updated
>>> https://github.com/vishwasbabu/ProvisioningFineractCN
>>>
>>> Kind regards
>>> Juhan
>>>
>>> Kontakt Ebenezer Graham (<eg...@alustudent.com>) kirjutas kuupäeval
>>> E, 2. september 2019 kell 15:03:
>>>
>>>> Thank you for the update.
>>>>
>>>> I will resolve the conflicts on the notification service and revert to
>>>> you.
>>>>
>>>>
>>>> On Mon, 2 Sep 2019, 03:52 Isaac Kamga, <u2...@gmail.com> wrote:
>>>>
>>>>> Hello Graham,
>>>>>
>>>>> That's some very good work you've done so far.
>>>>>
>>>>> I have executed the script and everything works well on my end right
>>>>> through to the fims-web-app.
>>>>>
>>>>> EclipseLink configurations was merged into PostgreSQL library last
>>>>> week and the changes to fineract-cn-test library on FINCN-114 which migrate
>>>>> it from MariaDB to PostgreSQL have been merged into the develop branch. The
>>>>> latter causes the Travis checks on the other repositories to succeed.
>>>>>
>>>>> However, the Thymeleaf templating PR heading to Notifications develop
>>>>> branch needs your attention.
>>>>>
>>>>> More grease to your elbows.
>>>>>
>>>>> Cheers,
>>>>> Isaac Kamga.
>>>>>
>>>>> On Sun, Aug 25, 2019 at 2:43 PM Ebenezer Graham <
>>>>> egraham15@alustudent.com> wrote:
>>>>>
>>>>>> Hello Devs,
>>>>>>
>>>>>> I would like to revive the topic discussed in the previous mail.
>>>>>>
>>>>>> I have been running Fineract CN on EclipseLink and PostgreSQL for a
>>>>>> while now and I believe it's ready to be merged into the develop branches.
>>>>>> But before that, it is paramount that other experienced developers in the
>>>>>> community examine what has been done so far and QA it.
>>>>>>
>>>>>> Simply execute this script
>>>>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-141/scripts/Unix/eclipselink-test-setup.sh to
>>>>>> get all the necessary branches.
>>>>>>
>>>>>> *PS: *I have experienced better performance with PostgreSQL and
>>>>>> EclipseLink on my workstation.
>>>>>>
>>>>>> For instance, Application Startup + Provisioning took 14 mins
>>>>>> Application Startup alone took 12 mins
>>>>>>
>>>>>> With MariaDB and Hibernate out of the way, we will be closer to
>>>>>> making Fineract CN compliant. On top of that, if we are able to upgrade the
>>>>>> spring versions, this should significantly improve Fineract CN's response
>>>>>> time, performance and memory usage.
>>>>>>
>>>>>> Looking forward to hearing from you.
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>>
>>>>>> On Tue, 16 Jul 2019 at 04:47, Ebenezer Graham <
>>>>>> egraham15@alustudent.com> wrote:
>>>>>>
>>>>>>> Hello Fineracters,
>>>>>>>
>>>>>>> I just finished integrating EclipseLink into Fineract CN. I would
>>>>>>> like to request your participation in the testing of the new ORM. Your
>>>>>>> constructive feedback will help me to fine-tune the integration and resolve
>>>>>>> all known issues with PostgreSQL.
>>>>>>>
>>>>>>> *Approach Used *
>>>>>>> Migrating from Hibernate to EclipseLink in Spring is actually very
>>>>>>> simple and seamless.
>>>>>>>
>>>>>>> 1. I simply removed hibernate core and entity-manager from
>>>>>>> spring-data-jpa [1]
>>>>>>> 2. Then, I added the recommended EclipseLink JPA dependency [2] to
>>>>>>> the MariaDB and PostgreSQL libraries.
>>>>>>> 4. I extended the JpaBaseConfiguration and configured it to use
>>>>>>> EclipseLink as the provider. As discussed earlier, Spring provides an
>>>>>>> adapter for EclipseLink so it was quite straightforward.
>>>>>>> 5. One critical decision I made, was choosing static weaving over
>>>>>>> the default dynamic weaving done at runtime. Static weaving improves
>>>>>>> performance and stability as it occurs during build-time. In addition, this
>>>>>>> approach is recommended for large applications with lots of entities. Read
>>>>>>> more at [3] and let me know what you think.
>>>>>>>
>>>>>>> Please conduct your tests and report any issues on this thread so
>>>>>>> that I can work on them.
>>>>>>>
>>>>>>> *Testing EclipseLink and PostgreSQL*
>>>>>>> You can test EclipseLink and PostgreSQL by using this script [4].
>>>>>>> The eclipselink-test-setup script will clone and publish the relevant
>>>>>>> artefacts.
>>>>>>>
>>>>>>> NB: Remember to shut down your Postgres server when doing component
>>>>>>> tests so that the embedded db can start.
>>>>>>>
>>>>>>> NB: Use this for full integration tests
>>>>>>> java -Ddemoserver.persistent=true -Dcustom.cassandra.contactPoints=
>>>>>>> 127.0.0.1:9042 -Dcassandra.cluster.user=cassandra
>>>>>>> -Dcassandra.cluster.pwd=password -Dcustom.postgresql.host=localhost
>>>>>>> -Dcustom.postgresql.user=postgres -jar
>>>>>>> ~/<fineract-cn-dir>/integration-tests/fineract-cn-demo-server/build/libs/demo-server-0.1.0-BUILD-SNAPSHOT.jar
>>>>>>>
>>>>>>> NB: The logs are a bit excessive as I'm currently outputting all
>>>>>>> EclipseLink logs but once testing is over, I will tone it down.
>>>>>>>
>>>>>>> I have conducted component test for each microservice and a full
>>>>>>> integration test. A few tests failed during the component test however, the
>>>>>>> causes are not from the ORM but the test cases themselves. I will try and
>>>>>>> fix them once I have met all of my milestones for this project or raise
>>>>>>> issues for them in Jira.
>>>>>>>
>>>>>>> *Known Issues in Postgres (My current focus):*
>>>>>>> 1. ERROR: relation "khepri_beats" does not exist. click on the link
>>>>>>> at [5] to see the full log. This issue is not directly related to the ORM.
>>>>>>> I actually identified this error whiles testing Postgres and Hibernate.
>>>>>>> This is related to Postgres and I am currently looking into it.
>>>>>>>
>>>>>>> 2. Error: org.postgresql.util.PSQLException: FATAL: sorry, too many
>>>>>>> clients already. click on the link [6] to see the full log.
>>>>>>>
>>>>>>> *Solution*, Increase max_connections found in the postgresql.conf.
>>>>>>> *260* worked for me.
>>>>>>> Rational, each microservice can have up to 20 unique connections to
>>>>>>> the db per the BoneCP configuration.
>>>>>>>
>>>>>>> Also, you can test EclipseLink and MariaDB by pulling the FINCN-2
>>>>>>> branch into fineract-cn-mariadb link [7] and rebuilding all of the
>>>>>>> microservices.
>>>>>>>
>>>>>>> *Identified Issue when using embedded dbs in full integration test *
>>>>>>> Tenant context not set, resulting in this error [8]
>>>>>>>
>>>>>>> Which brings me to my concern about MariaDB. Given that MariaDB is
>>>>>>> not compliant with Apache, can I safely assume that it's not relevant to
>>>>>>> migrate it's ORM as well?
>>>>>>>
>>>>>>> Looking forward to hearing from you all.
>>>>>>>
>>>>>>> *At your service,*
>>>>>>> Graham.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L76-L80
>>>>>>> [2]
>>>>>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L74
>>>>>>> / https://www.baeldung.com/spring-eclipselink
>>>>>>> [3]
>>>>>>> https://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Static_Weaving
>>>>>>> https://pastebin.com/FDAzd6bx
>>>>>>> [4]
>>>>>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-2/scripts/Unix/eclipselink-test-setup.sh
>>>>>>> [5] https://pastebin.com/WFQ9satu
>>>>>>> [6] https://pastebin.com/M8GAHsit
>>>>>>> [7]
>>>>>>> https://github.com/ebenezergraham/fineract-cn-mariadb/tree/FINCN-2
>>>>>>> [8] https://pastebin.com/FDAzd6bx
>>>>>>>
>>>>>>
Re: Testing ORM Migration to EclipseLink
Posted by Juhan Aasaru <aa...@gmail.com>.
Hi!
Sorry about the late reply.
1. In my opinion it is not question about provisioning as as soon as the
rhythym application starts up, it tries to select from khepri_beats (see
logs below).
I'm not using demo server at all. I use Postman scripts for provisioning. I
wonder if I should run some step before starting rhythm micro service.
I get the error on both with latest docker image of rhythm as well as when
starting the service locally.
I logged bug report here: https://issues.apache.org/jira/browse/FINCN-177
Here are the logs when running from localhost
13:56:54.948 [main] INFO o.s.b.c.e.j.JettyEmbeddedServletContainer - Jetty
started on port(s) 2022 (http/1.1)
13:56:54.958 [main] INFO o.a.f.c.r.service.RhythmApplication - Started
RhythmApplication in 13.207 seconds (JVM running for 14.079)
[EL Finer]: transaction: 2019-09-18
13:56:54.998--UnitOfWork(641160635)--Thread(Thread[pool-2-thread-1,5,main])--begin
unit of work flush
[EL Finer]: transaction: 2019-09-18
13:56:54.999--UnitOfWork(641160635)--Thread(Thread[pool-2-thread-1,5,main])--end
unit of work flush
[EL Finest]: query: 2019-09-18
13:56:54.999--UnitOfWork(641160635)--Thread(Thread[pool-2-thread-1,5,main])--Execute
query ReadAllQuery(referenceClass=BeatEntity )
[EL Finest]: connection: 2019-09-18
13:56:55.013--ServerSession(988179589)--Connection(1724597978)--Thread(Thread[pool-2-thread-1,5,main])--Connection
acquired from connection pool [default].
[EL Finer]: transaction: 2019-09-18
13:56:55.013--ClientSession(571004194)--Connection(1724597978)--Thread(Thread[pool-2-thread-1,5,main])--begin
transaction
[EL Finest]: connection: 2019-09-18
13:56:55.013--ClientSession(571004194)--Thread(Thread[pool-2-thread-1,5,main])--reconnecting
to external connection pool
[EL Fine]: sql: 2019-09-18
13:56:55.014--ClientSession(571004194)--Connection(1203869313)--Thread(Thread[pool-2-thread-1,5,main])--SELECT
id, alignment_hour, application_identifier, beat_identifier, next_beat,
tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR UPDATE
bind => [2019-09-18 13:56:54.923]
[EL Fine]: sql: 2019-09-18
13:56:55.053--ClientSession(571004194)--Thread(Thread[pool-2-thread-1,5,main])--SELECT
1
[EL Warning]: 2019-09-18
13:56:55.055--UnitOfWork(641160635)--Thread(Thread[pool-2-thread-1,5,main])--Local
Exception Stack:
Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.6.4.v20160829-44060b6):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: relation
"khepri_beats" does not exist
Position: 103
Error Code: 0
Call: SELECT id, alignment_hour, application_identifier, beat_identifier,
next_beat, tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR
UPDATE
bind => [2019-09-18 13:56:54.923]
Query: ReadAllQuery(referenceClass=BeatEntity sql="SELECT id,
alignment_hour, application_identifier, beat_identifier, next_beat,
tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR UPDATE")
at
org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
Regards
Juhan
Kontakt Ebenezer Graham (<eg...@alustudent.com>) kirjutas kuupäeval R,
6. september 2019 kell 14:28:
> Thanks for the update,
>
> 1. For the error below, I believe that is because you didn't provision the
> microservices. Can you please confirm if you set
> -Ddemoserver.provision=true on the first run. Also, you can use
> https://github.com/apache/fineract-cn-demo-server/blob/develop/scripts/Unix/initial-setup.sh to
> test the integration. I have deleted a few of the branches in the
> eclipselink-test-script.sh
>
> 1. Rhythm keeps giving me this error:
>> rhythm-ms_1 | Internal Exception:
>> org.postgresql.util.PSQLException: ERROR: relation "khepri_beats" does not
>> exist
>> rhythm-ms_1 | Position: 103
>
>
> 2. I will make the make the changes and submit the necessary PRs asap.
>
> 3. I have updated the Setup Guide to cater for PostgreSQL. However, I need
> someone to test the MacOS instructions work correctly.
>
> Regards
>
> On Wed, 4 Sep 2019 at 15:54, Juhan Aasaru <aa...@gmail.com> wrote:
>
>> Hi Graham and Isaac!
>>
>> Great progress and excellent work regarding Postgresql and EclipseLink
>> migration!
>> I did some tests (using the official docker images I'm working on). This
>> is what I found
>>
>> 1. Rhythm keeps giving me this error:
>> rhythm-ms_1 | Internal Exception:
>> org.postgresql.util.PSQLException: ERROR: relation "khepri_beats" does not
>> exist
>> rhythm-ms_1 | Position: 103
>> rhythm-ms_1 | Error Code: 0
>> rhythm-ms_1 | Call: SELECT id, alignment_hour,
>> application_identifier, beat_identifier, next_beat, tenant_identifier FROM
>> khepri_beats WHERE (next_beat < ?) FOR UPDATE
>> rhythm-ms_1 | bind => [2019-09-04 11:34:43.778]
>> rhythm-ms_1 | Query: ReadAllQuery(referenceClass=BeatEntity
>> sql="SELECT id, alignment_hour, application_identifier, beat_identifier,
>> next_beat, tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR
>> UPDATE")
>>
>> 2. many services (provisioner, office, customer, accounting, teller,
>> reporting, payroll) now define data source properties twice
>> in service/src/main/resources/application.yaml
>> The problem is that if I want to overwrite some value I have to pass it
>> in as two separate parameters.
>>
>> For example:
>> https://github.com/apache/fineract-cn-provisioner/commit/4327ce032c97cc961ce15e39c22863654cefc230
>> has:
>>
>> postgresql:
>> driverClass: org.postgresql.Driver
>> database: seshat
>> host: localhost
>> port: 5432
>> user: postgres
>> password: postgres
>>
>> spring:
>> datasource:
>> driver-class-name: org.postgresql.Driver
>> url: jdbc:postgresql://localhost:5432/seshat
>> username: postgres
>> password: postgres
>>
>> Please change the last block to something like this:
>>
>> spring:
>> datasource:
>> driver-class-name: org.postgresql.Driver
>> url: jdbc:postgresql://${postgresql.host}:${postgresql.port}/seshat
>> username: ${postgresql.user}
>> password: ${postgresql.password}
>>
>> Then there should be no need to pass in the variables twice if I want to
>> change some value.
>>
>> 3. Instructions should be updated as well and MariaDB/Mysql replaced with
>> Postgresql:
>>
>> https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN
>>
>> Also Vishwas has some good instructions that need to be updated
>> https://github.com/vishwasbabu/ProvisioningFineractCN
>>
>> Kind regards
>> Juhan
>>
>> Kontakt Ebenezer Graham (<eg...@alustudent.com>) kirjutas kuupäeval
>> E, 2. september 2019 kell 15:03:
>>
>>> Thank you for the update.
>>>
>>> I will resolve the conflicts on the notification service and revert to
>>> you.
>>>
>>>
>>> On Mon, 2 Sep 2019, 03:52 Isaac Kamga, <u2...@gmail.com> wrote:
>>>
>>>> Hello Graham,
>>>>
>>>> That's some very good work you've done so far.
>>>>
>>>> I have executed the script and everything works well on my end right
>>>> through to the fims-web-app.
>>>>
>>>> EclipseLink configurations was merged into PostgreSQL library last week
>>>> and the changes to fineract-cn-test library on FINCN-114 which migrate it
>>>> from MariaDB to PostgreSQL have been merged into the develop branch. The
>>>> latter causes the Travis checks on the other repositories to succeed.
>>>>
>>>> However, the Thymeleaf templating PR heading to Notifications develop
>>>> branch needs your attention.
>>>>
>>>> More grease to your elbows.
>>>>
>>>> Cheers,
>>>> Isaac Kamga.
>>>>
>>>> On Sun, Aug 25, 2019 at 2:43 PM Ebenezer Graham <
>>>> egraham15@alustudent.com> wrote:
>>>>
>>>>> Hello Devs,
>>>>>
>>>>> I would like to revive the topic discussed in the previous mail.
>>>>>
>>>>> I have been running Fineract CN on EclipseLink and PostgreSQL for a
>>>>> while now and I believe it's ready to be merged into the develop branches.
>>>>> But before that, it is paramount that other experienced developers in the
>>>>> community examine what has been done so far and QA it.
>>>>>
>>>>> Simply execute this script
>>>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-141/scripts/Unix/eclipselink-test-setup.sh to
>>>>> get all the necessary branches.
>>>>>
>>>>> *PS: *I have experienced better performance with PostgreSQL and
>>>>> EclipseLink on my workstation.
>>>>>
>>>>> For instance, Application Startup + Provisioning took 14 mins
>>>>> Application Startup alone took 12 mins
>>>>>
>>>>> With MariaDB and Hibernate out of the way, we will be closer to making
>>>>> Fineract CN compliant. On top of that, if we are able to upgrade the spring
>>>>> versions, this should significantly improve Fineract CN's response time,
>>>>> performance and memory usage.
>>>>>
>>>>> Looking forward to hearing from you.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>> On Tue, 16 Jul 2019 at 04:47, Ebenezer Graham <
>>>>> egraham15@alustudent.com> wrote:
>>>>>
>>>>>> Hello Fineracters,
>>>>>>
>>>>>> I just finished integrating EclipseLink into Fineract CN. I would
>>>>>> like to request your participation in the testing of the new ORM. Your
>>>>>> constructive feedback will help me to fine-tune the integration and resolve
>>>>>> all known issues with PostgreSQL.
>>>>>>
>>>>>> *Approach Used *
>>>>>> Migrating from Hibernate to EclipseLink in Spring is actually very
>>>>>> simple and seamless.
>>>>>>
>>>>>> 1. I simply removed hibernate core and entity-manager from
>>>>>> spring-data-jpa [1]
>>>>>> 2. Then, I added the recommended EclipseLink JPA dependency [2] to
>>>>>> the MariaDB and PostgreSQL libraries.
>>>>>> 4. I extended the JpaBaseConfiguration and configured it to use
>>>>>> EclipseLink as the provider. As discussed earlier, Spring provides an
>>>>>> adapter for EclipseLink so it was quite straightforward.
>>>>>> 5. One critical decision I made, was choosing static weaving over the
>>>>>> default dynamic weaving done at runtime. Static weaving improves
>>>>>> performance and stability as it occurs during build-time. In addition, this
>>>>>> approach is recommended for large applications with lots of entities. Read
>>>>>> more at [3] and let me know what you think.
>>>>>>
>>>>>> Please conduct your tests and report any issues on this thread so
>>>>>> that I can work on them.
>>>>>>
>>>>>> *Testing EclipseLink and PostgreSQL*
>>>>>> You can test EclipseLink and PostgreSQL by using this script [4]. The
>>>>>> eclipselink-test-setup script will clone and publish the relevant artefacts.
>>>>>>
>>>>>> NB: Remember to shut down your Postgres server when doing component
>>>>>> tests so that the embedded db can start.
>>>>>>
>>>>>> NB: Use this for full integration tests
>>>>>> java -Ddemoserver.persistent=true -Dcustom.cassandra.contactPoints=
>>>>>> 127.0.0.1:9042 -Dcassandra.cluster.user=cassandra
>>>>>> -Dcassandra.cluster.pwd=password -Dcustom.postgresql.host=localhost
>>>>>> -Dcustom.postgresql.user=postgres -jar
>>>>>> ~/<fineract-cn-dir>/integration-tests/fineract-cn-demo-server/build/libs/demo-server-0.1.0-BUILD-SNAPSHOT.jar
>>>>>>
>>>>>> NB: The logs are a bit excessive as I'm currently outputting all
>>>>>> EclipseLink logs but once testing is over, I will tone it down.
>>>>>>
>>>>>> I have conducted component test for each microservice and a full
>>>>>> integration test. A few tests failed during the component test however, the
>>>>>> causes are not from the ORM but the test cases themselves. I will try and
>>>>>> fix them once I have met all of my milestones for this project or raise
>>>>>> issues for them in Jira.
>>>>>>
>>>>>> *Known Issues in Postgres (My current focus):*
>>>>>> 1. ERROR: relation "khepri_beats" does not exist. click on the link
>>>>>> at [5] to see the full log. This issue is not directly related to the ORM.
>>>>>> I actually identified this error whiles testing Postgres and Hibernate.
>>>>>> This is related to Postgres and I am currently looking into it.
>>>>>>
>>>>>> 2. Error: org.postgresql.util.PSQLException: FATAL: sorry, too many
>>>>>> clients already. click on the link [6] to see the full log.
>>>>>>
>>>>>> *Solution*, Increase max_connections found in the postgresql.conf.
>>>>>> *260* worked for me.
>>>>>> Rational, each microservice can have up to 20 unique connections to
>>>>>> the db per the BoneCP configuration.
>>>>>>
>>>>>> Also, you can test EclipseLink and MariaDB by pulling the FINCN-2
>>>>>> branch into fineract-cn-mariadb link [7] and rebuilding all of the
>>>>>> microservices.
>>>>>>
>>>>>> *Identified Issue when using embedded dbs in full integration test *
>>>>>> Tenant context not set, resulting in this error [8]
>>>>>>
>>>>>> Which brings me to my concern about MariaDB. Given that MariaDB is
>>>>>> not compliant with Apache, can I safely assume that it's not relevant to
>>>>>> migrate it's ORM as well?
>>>>>>
>>>>>> Looking forward to hearing from you all.
>>>>>>
>>>>>> *At your service,*
>>>>>> Graham.
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L76-L80
>>>>>> [2]
>>>>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L74
>>>>>> / https://www.baeldung.com/spring-eclipselink
>>>>>> [3]
>>>>>> https://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Static_Weaving
>>>>>> https://pastebin.com/FDAzd6bx
>>>>>> [4]
>>>>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-2/scripts/Unix/eclipselink-test-setup.sh
>>>>>> [5] https://pastebin.com/WFQ9satu
>>>>>> [6] https://pastebin.com/M8GAHsit
>>>>>> [7]
>>>>>> https://github.com/ebenezergraham/fineract-cn-mariadb/tree/FINCN-2
>>>>>> [8] https://pastebin.com/FDAzd6bx
>>>>>>
>>>>>
Re: Testing ORM Migration to EclipseLink
Posted by Ebenezer Graham <eg...@alustudent.com>.
Thanks for the update,
1. For the error below, I believe that is because you didn't provision the
microservices. Can you please confirm if you set
-Ddemoserver.provision=true on the first run. Also, you can use
https://github.com/apache/fineract-cn-demo-server/blob/develop/scripts/Unix/initial-setup.sh
to
test the integration. I have deleted a few of the branches in the
eclipselink-test-script.sh
1. Rhythm keeps giving me this error:
> rhythm-ms_1 | Internal Exception: org.postgresql.util.PSQLException:
> ERROR: relation "khepri_beats" does not exist
> rhythm-ms_1 | Position: 103
2. I will make the make the changes and submit the necessary PRs asap.
3. I have updated the Setup Guide to cater for PostgreSQL. However, I need
someone to test the MacOS instructions work correctly.
Regards
On Wed, 4 Sep 2019 at 15:54, Juhan Aasaru <aa...@gmail.com> wrote:
> Hi Graham and Isaac!
>
> Great progress and excellent work regarding Postgresql and EclipseLink
> migration!
> I did some tests (using the official docker images I'm working on). This
> is what I found
>
> 1. Rhythm keeps giving me this error:
> rhythm-ms_1 | Internal Exception: org.postgresql.util.PSQLException:
> ERROR: relation "khepri_beats" does not exist
> rhythm-ms_1 | Position: 103
> rhythm-ms_1 | Error Code: 0
> rhythm-ms_1 | Call: SELECT id, alignment_hour,
> application_identifier, beat_identifier, next_beat, tenant_identifier FROM
> khepri_beats WHERE (next_beat < ?) FOR UPDATE
> rhythm-ms_1 | bind => [2019-09-04 11:34:43.778]
> rhythm-ms_1 | Query: ReadAllQuery(referenceClass=BeatEntity
> sql="SELECT id, alignment_hour, application_identifier, beat_identifier,
> next_beat, tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR
> UPDATE")
>
> 2. many services (provisioner, office, customer, accounting, teller,
> reporting, payroll) now define data source properties twice
> in service/src/main/resources/application.yaml
> The problem is that if I want to overwrite some value I have to pass it in
> as two separate parameters.
>
> For example:
> https://github.com/apache/fineract-cn-provisioner/commit/4327ce032c97cc961ce15e39c22863654cefc230
> has:
>
> postgresql:
> driverClass: org.postgresql.Driver
> database: seshat
> host: localhost
> port: 5432
> user: postgres
> password: postgres
>
> spring:
> datasource:
> driver-class-name: org.postgresql.Driver
> url: jdbc:postgresql://localhost:5432/seshat
> username: postgres
> password: postgres
>
> Please change the last block to something like this:
>
> spring:
> datasource:
> driver-class-name: org.postgresql.Driver
> url: jdbc:postgresql://${postgresql.host}:${postgresql.port}/seshat
> username: ${postgresql.user}
> password: ${postgresql.password}
>
> Then there should be no need to pass in the variables twice if I want to
> change some value.
>
> 3. Instructions should be updated as well and MariaDB/Mysql replaced with
> Postgresql:
>
> https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN
>
> Also Vishwas has some good instructions that need to be updated
> https://github.com/vishwasbabu/ProvisioningFineractCN
>
> Kind regards
> Juhan
>
> Kontakt Ebenezer Graham (<eg...@alustudent.com>) kirjutas kuupäeval
> E, 2. september 2019 kell 15:03:
>
>> Thank you for the update.
>>
>> I will resolve the conflicts on the notification service and revert to
>> you.
>>
>>
>> On Mon, 2 Sep 2019, 03:52 Isaac Kamga, <u2...@gmail.com> wrote:
>>
>>> Hello Graham,
>>>
>>> That's some very good work you've done so far.
>>>
>>> I have executed the script and everything works well on my end right
>>> through to the fims-web-app.
>>>
>>> EclipseLink configurations was merged into PostgreSQL library last week
>>> and the changes to fineract-cn-test library on FINCN-114 which migrate it
>>> from MariaDB to PostgreSQL have been merged into the develop branch. The
>>> latter causes the Travis checks on the other repositories to succeed.
>>>
>>> However, the Thymeleaf templating PR heading to Notifications develop
>>> branch needs your attention.
>>>
>>> More grease to your elbows.
>>>
>>> Cheers,
>>> Isaac Kamga.
>>>
>>> On Sun, Aug 25, 2019 at 2:43 PM Ebenezer Graham <
>>> egraham15@alustudent.com> wrote:
>>>
>>>> Hello Devs,
>>>>
>>>> I would like to revive the topic discussed in the previous mail.
>>>>
>>>> I have been running Fineract CN on EclipseLink and PostgreSQL for a
>>>> while now and I believe it's ready to be merged into the develop branches.
>>>> But before that, it is paramount that other experienced developers in the
>>>> community examine what has been done so far and QA it.
>>>>
>>>> Simply execute this script
>>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-141/scripts/Unix/eclipselink-test-setup.sh to
>>>> get all the necessary branches.
>>>>
>>>> *PS: *I have experienced better performance with PostgreSQL and
>>>> EclipseLink on my workstation.
>>>>
>>>> For instance, Application Startup + Provisioning took 14 mins
>>>> Application Startup alone took 12 mins
>>>>
>>>> With MariaDB and Hibernate out of the way, we will be closer to making
>>>> Fineract CN compliant. On top of that, if we are able to upgrade the spring
>>>> versions, this should significantly improve Fineract CN's response time,
>>>> performance and memory usage.
>>>>
>>>> Looking forward to hearing from you.
>>>>
>>>> Regards.
>>>>
>>>>
>>>> On Tue, 16 Jul 2019 at 04:47, Ebenezer Graham <eg...@alustudent.com>
>>>> wrote:
>>>>
>>>>> Hello Fineracters,
>>>>>
>>>>> I just finished integrating EclipseLink into Fineract CN. I would like
>>>>> to request your participation in the testing of the new ORM. Your
>>>>> constructive feedback will help me to fine-tune the integration and resolve
>>>>> all known issues with PostgreSQL.
>>>>>
>>>>> *Approach Used *
>>>>> Migrating from Hibernate to EclipseLink in Spring is actually very
>>>>> simple and seamless.
>>>>>
>>>>> 1. I simply removed hibernate core and entity-manager from
>>>>> spring-data-jpa [1]
>>>>> 2. Then, I added the recommended EclipseLink JPA dependency [2] to the
>>>>> MariaDB and PostgreSQL libraries.
>>>>> 4. I extended the JpaBaseConfiguration and configured it to use
>>>>> EclipseLink as the provider. As discussed earlier, Spring provides an
>>>>> adapter for EclipseLink so it was quite straightforward.
>>>>> 5. One critical decision I made, was choosing static weaving over the
>>>>> default dynamic weaving done at runtime. Static weaving improves
>>>>> performance and stability as it occurs during build-time. In addition, this
>>>>> approach is recommended for large applications with lots of entities. Read
>>>>> more at [3] and let me know what you think.
>>>>>
>>>>> Please conduct your tests and report any issues on this thread so that
>>>>> I can work on them.
>>>>>
>>>>> *Testing EclipseLink and PostgreSQL*
>>>>> You can test EclipseLink and PostgreSQL by using this script [4]. The
>>>>> eclipselink-test-setup script will clone and publish the relevant artefacts.
>>>>>
>>>>> NB: Remember to shut down your Postgres server when doing component
>>>>> tests so that the embedded db can start.
>>>>>
>>>>> NB: Use this for full integration tests
>>>>> java -Ddemoserver.persistent=true -Dcustom.cassandra.contactPoints=
>>>>> 127.0.0.1:9042 -Dcassandra.cluster.user=cassandra
>>>>> -Dcassandra.cluster.pwd=password -Dcustom.postgresql.host=localhost
>>>>> -Dcustom.postgresql.user=postgres -jar
>>>>> ~/<fineract-cn-dir>/integration-tests/fineract-cn-demo-server/build/libs/demo-server-0.1.0-BUILD-SNAPSHOT.jar
>>>>>
>>>>> NB: The logs are a bit excessive as I'm currently outputting all
>>>>> EclipseLink logs but once testing is over, I will tone it down.
>>>>>
>>>>> I have conducted component test for each microservice and a full
>>>>> integration test. A few tests failed during the component test however, the
>>>>> causes are not from the ORM but the test cases themselves. I will try and
>>>>> fix them once I have met all of my milestones for this project or raise
>>>>> issues for them in Jira.
>>>>>
>>>>> *Known Issues in Postgres (My current focus):*
>>>>> 1. ERROR: relation "khepri_beats" does not exist. click on the link at
>>>>> [5] to see the full log. This issue is not directly related to the ORM. I
>>>>> actually identified this error whiles testing Postgres and Hibernate. This
>>>>> is related to Postgres and I am currently looking into it.
>>>>>
>>>>> 2. Error: org.postgresql.util.PSQLException: FATAL: sorry, too many
>>>>> clients already. click on the link [6] to see the full log.
>>>>>
>>>>> *Solution*, Increase max_connections found in the postgresql.conf.
>>>>> *260* worked for me.
>>>>> Rational, each microservice can have up to 20 unique connections to
>>>>> the db per the BoneCP configuration.
>>>>>
>>>>> Also, you can test EclipseLink and MariaDB by pulling the FINCN-2
>>>>> branch into fineract-cn-mariadb link [7] and rebuilding all of the
>>>>> microservices.
>>>>>
>>>>> *Identified Issue when using embedded dbs in full integration test *
>>>>> Tenant context not set, resulting in this error [8]
>>>>>
>>>>> Which brings me to my concern about MariaDB. Given that MariaDB is not
>>>>> compliant with Apache, can I safely assume that it's not relevant to
>>>>> migrate it's ORM as well?
>>>>>
>>>>> Looking forward to hearing from you all.
>>>>>
>>>>> *At your service,*
>>>>> Graham.
>>>>>
>>>>> [1]
>>>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L76-L80
>>>>> [2]
>>>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L74
>>>>> / https://www.baeldung.com/spring-eclipselink
>>>>> [3]
>>>>> https://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Static_Weaving
>>>>> https://pastebin.com/FDAzd6bx
>>>>> [4]
>>>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-2/scripts/Unix/eclipselink-test-setup.sh
>>>>> [5] https://pastebin.com/WFQ9satu
>>>>> [6] https://pastebin.com/M8GAHsit
>>>>> [7] https://github.com/ebenezergraham/fineract-cn-mariadb/tree/FINCN-2
>>>>> [8] https://pastebin.com/FDAzd6bx
>>>>>
>>>>
Re: Testing ORM Migration to EclipseLink
Posted by Juhan Aasaru <aa...@gmail.com>.
Hi Graham and Isaac!
Great progress and excellent work regarding Postgresql and EclipseLink
migration!
I did some tests (using the official docker images I'm working on). This is
what I found
1. Rhythm keeps giving me this error:
rhythm-ms_1 | Internal Exception: org.postgresql.util.PSQLException:
ERROR: relation "khepri_beats" does not exist
rhythm-ms_1 | Position: 103
rhythm-ms_1 | Error Code: 0
rhythm-ms_1 | Call: SELECT id, alignment_hour,
application_identifier, beat_identifier, next_beat, tenant_identifier FROM
khepri_beats WHERE (next_beat < ?) FOR UPDATE
rhythm-ms_1 | bind => [2019-09-04 11:34:43.778]
rhythm-ms_1 | Query: ReadAllQuery(referenceClass=BeatEntity
sql="SELECT id, alignment_hour, application_identifier, beat_identifier,
next_beat, tenant_identifier FROM khepri_beats WHERE (next_beat < ?) FOR
UPDATE")
2. many services (provisioner, office, customer, accounting, teller,
reporting, payroll) now define data source properties twice
in service/src/main/resources/application.yaml
The problem is that if I want to overwrite some value I have to pass it in
as two separate parameters.
For example:
https://github.com/apache/fineract-cn-provisioner/commit/4327ce032c97cc961ce15e39c22863654cefc230
has:
postgresql:
driverClass: org.postgresql.Driver
database: seshat
host: localhost
port: 5432
user: postgres
password: postgres
spring:
datasource:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://localhost:5432/seshat
username: postgres
password: postgres
Please change the last block to something like this:
spring:
datasource:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://${postgresql.host}:${postgresql.port}/seshat
username: ${postgresql.user}
password: ${postgresql.password}
Then there should be no need to pass in the variables twice if I want to
change some value.
3. Instructions should be updated as well and MariaDB/Mysql replaced with
Postgresql:
https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN
Also Vishwas has some good instructions that need to be updated
https://github.com/vishwasbabu/ProvisioningFineractCN
Kind regards
Juhan
Kontakt Ebenezer Graham (<eg...@alustudent.com>) kirjutas kuupäeval E,
2. september 2019 kell 15:03:
> Thank you for the update.
>
> I will resolve the conflicts on the notification service and revert to
> you.
>
>
> On Mon, 2 Sep 2019, 03:52 Isaac Kamga, <u2...@gmail.com> wrote:
>
>> Hello Graham,
>>
>> That's some very good work you've done so far.
>>
>> I have executed the script and everything works well on my end right
>> through to the fims-web-app.
>>
>> EclipseLink configurations was merged into PostgreSQL library last week
>> and the changes to fineract-cn-test library on FINCN-114 which migrate it
>> from MariaDB to PostgreSQL have been merged into the develop branch. The
>> latter causes the Travis checks on the other repositories to succeed.
>>
>> However, the Thymeleaf templating PR heading to Notifications develop
>> branch needs your attention.
>>
>> More grease to your elbows.
>>
>> Cheers,
>> Isaac Kamga.
>>
>> On Sun, Aug 25, 2019 at 2:43 PM Ebenezer Graham <eg...@alustudent.com>
>> wrote:
>>
>>> Hello Devs,
>>>
>>> I would like to revive the topic discussed in the previous mail.
>>>
>>> I have been running Fineract CN on EclipseLink and PostgreSQL for a
>>> while now and I believe it's ready to be merged into the develop branches.
>>> But before that, it is paramount that other experienced developers in the
>>> community examine what has been done so far and QA it.
>>>
>>> Simply execute this script
>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-141/scripts/Unix/eclipselink-test-setup.sh to
>>> get all the necessary branches.
>>>
>>> *PS: *I have experienced better performance with PostgreSQL and
>>> EclipseLink on my workstation.
>>>
>>> For instance, Application Startup + Provisioning took 14 mins
>>> Application Startup alone took 12 mins
>>>
>>> With MariaDB and Hibernate out of the way, we will be closer to making
>>> Fineract CN compliant. On top of that, if we are able to upgrade the spring
>>> versions, this should significantly improve Fineract CN's response time,
>>> performance and memory usage.
>>>
>>> Looking forward to hearing from you.
>>>
>>> Regards.
>>>
>>>
>>> On Tue, 16 Jul 2019 at 04:47, Ebenezer Graham <eg...@alustudent.com>
>>> wrote:
>>>
>>>> Hello Fineracters,
>>>>
>>>> I just finished integrating EclipseLink into Fineract CN. I would like
>>>> to request your participation in the testing of the new ORM. Your
>>>> constructive feedback will help me to fine-tune the integration and resolve
>>>> all known issues with PostgreSQL.
>>>>
>>>> *Approach Used *
>>>> Migrating from Hibernate to EclipseLink in Spring is actually very
>>>> simple and seamless.
>>>>
>>>> 1. I simply removed hibernate core and entity-manager from
>>>> spring-data-jpa [1]
>>>> 2. Then, I added the recommended EclipseLink JPA dependency [2] to the
>>>> MariaDB and PostgreSQL libraries.
>>>> 4. I extended the JpaBaseConfiguration and configured it to use
>>>> EclipseLink as the provider. As discussed earlier, Spring provides an
>>>> adapter for EclipseLink so it was quite straightforward.
>>>> 5. One critical decision I made, was choosing static weaving over the
>>>> default dynamic weaving done at runtime. Static weaving improves
>>>> performance and stability as it occurs during build-time. In addition, this
>>>> approach is recommended for large applications with lots of entities. Read
>>>> more at [3] and let me know what you think.
>>>>
>>>> Please conduct your tests and report any issues on this thread so that
>>>> I can work on them.
>>>>
>>>> *Testing EclipseLink and PostgreSQL*
>>>> You can test EclipseLink and PostgreSQL by using this script [4]. The
>>>> eclipselink-test-setup script will clone and publish the relevant artefacts.
>>>>
>>>> NB: Remember to shut down your Postgres server when doing component
>>>> tests so that the embedded db can start.
>>>>
>>>> NB: Use this for full integration tests
>>>> java -Ddemoserver.persistent=true -Dcustom.cassandra.contactPoints=
>>>> 127.0.0.1:9042 -Dcassandra.cluster.user=cassandra
>>>> -Dcassandra.cluster.pwd=password -Dcustom.postgresql.host=localhost
>>>> -Dcustom.postgresql.user=postgres -jar
>>>> ~/<fineract-cn-dir>/integration-tests/fineract-cn-demo-server/build/libs/demo-server-0.1.0-BUILD-SNAPSHOT.jar
>>>>
>>>> NB: The logs are a bit excessive as I'm currently outputting all
>>>> EclipseLink logs but once testing is over, I will tone it down.
>>>>
>>>> I have conducted component test for each microservice and a full
>>>> integration test. A few tests failed during the component test however, the
>>>> causes are not from the ORM but the test cases themselves. I will try and
>>>> fix them once I have met all of my milestones for this project or raise
>>>> issues for them in Jira.
>>>>
>>>> *Known Issues in Postgres (My current focus):*
>>>> 1. ERROR: relation "khepri_beats" does not exist. click on the link at
>>>> [5] to see the full log. This issue is not directly related to the ORM. I
>>>> actually identified this error whiles testing Postgres and Hibernate. This
>>>> is related to Postgres and I am currently looking into it.
>>>>
>>>> 2. Error: org.postgresql.util.PSQLException: FATAL: sorry, too many
>>>> clients already. click on the link [6] to see the full log.
>>>>
>>>> *Solution*, Increase max_connections found in the postgresql.conf.
>>>> *260* worked for me.
>>>> Rational, each microservice can have up to 20 unique connections to the
>>>> db per the BoneCP configuration.
>>>>
>>>> Also, you can test EclipseLink and MariaDB by pulling the FINCN-2
>>>> branch into fineract-cn-mariadb link [7] and rebuilding all of the
>>>> microservices.
>>>>
>>>> *Identified Issue when using embedded dbs in full integration test *
>>>> Tenant context not set, resulting in this error [8]
>>>>
>>>> Which brings me to my concern about MariaDB. Given that MariaDB is not
>>>> compliant with Apache, can I safely assume that it's not relevant to
>>>> migrate it's ORM as well?
>>>>
>>>> Looking forward to hearing from you all.
>>>>
>>>> *At your service,*
>>>> Graham.
>>>>
>>>> [1]
>>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L76-L80
>>>> [2]
>>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L74
>>>> / https://www.baeldung.com/spring-eclipselink
>>>> [3]
>>>> https://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Static_Weaving
>>>> https://pastebin.com/FDAzd6bx
>>>> [4]
>>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-2/scripts/Unix/eclipselink-test-setup.sh
>>>> [5] https://pastebin.com/WFQ9satu
>>>> [6] https://pastebin.com/M8GAHsit
>>>> [7] https://github.com/ebenezergraham/fineract-cn-mariadb/tree/FINCN-2
>>>> [8] https://pastebin.com/FDAzd6bx
>>>>
>>>
Re: Testing ORM Migration to EclipseLink
Posted by Ebenezer Graham <eg...@alustudent.com>.
Thank you for the update.
I will resolve the conflicts on the notification service and revert to you.
On Mon, 2 Sep 2019, 03:52 Isaac Kamga, <u2...@gmail.com> wrote:
> Hello Graham,
>
> That's some very good work you've done so far.
>
> I have executed the script and everything works well on my end right
> through to the fims-web-app.
>
> EclipseLink configurations was merged into PostgreSQL library last week
> and the changes to fineract-cn-test library on FINCN-114 which migrate it
> from MariaDB to PostgreSQL have been merged into the develop branch. The
> latter causes the Travis checks on the other repositories to succeed.
>
> However, the Thymeleaf templating PR heading to Notifications develop
> branch needs your attention.
>
> More grease to your elbows.
>
> Cheers,
> Isaac Kamga.
>
> On Sun, Aug 25, 2019 at 2:43 PM Ebenezer Graham <eg...@alustudent.com>
> wrote:
>
>> Hello Devs,
>>
>> I would like to revive the topic discussed in the previous mail.
>>
>> I have been running Fineract CN on EclipseLink and PostgreSQL for a
>> while now and I believe it's ready to be merged into the develop branches.
>> But before that, it is paramount that other experienced developers in the
>> community examine what has been done so far and QA it.
>>
>> Simply execute this script
>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-141/scripts/Unix/eclipselink-test-setup.sh to
>> get all the necessary branches.
>>
>> *PS: *I have experienced better performance with PostgreSQL and
>> EclipseLink on my workstation.
>>
>> For instance, Application Startup + Provisioning took 14 mins
>> Application Startup alone took 12 mins
>>
>> With MariaDB and Hibernate out of the way, we will be closer to making
>> Fineract CN compliant. On top of that, if we are able to upgrade the spring
>> versions, this should significantly improve Fineract CN's response time,
>> performance and memory usage.
>>
>> Looking forward to hearing from you.
>>
>> Regards.
>>
>>
>> On Tue, 16 Jul 2019 at 04:47, Ebenezer Graham <eg...@alustudent.com>
>> wrote:
>>
>>> Hello Fineracters,
>>>
>>> I just finished integrating EclipseLink into Fineract CN. I would like
>>> to request your participation in the testing of the new ORM. Your
>>> constructive feedback will help me to fine-tune the integration and resolve
>>> all known issues with PostgreSQL.
>>>
>>> *Approach Used *
>>> Migrating from Hibernate to EclipseLink in Spring is actually very
>>> simple and seamless.
>>>
>>> 1. I simply removed hibernate core and entity-manager from
>>> spring-data-jpa [1]
>>> 2. Then, I added the recommended EclipseLink JPA dependency [2] to the
>>> MariaDB and PostgreSQL libraries.
>>> 4. I extended the JpaBaseConfiguration and configured it to use
>>> EclipseLink as the provider. As discussed earlier, Spring provides an
>>> adapter for EclipseLink so it was quite straightforward.
>>> 5. One critical decision I made, was choosing static weaving over the
>>> default dynamic weaving done at runtime. Static weaving improves
>>> performance and stability as it occurs during build-time. In addition, this
>>> approach is recommended for large applications with lots of entities. Read
>>> more at [3] and let me know what you think.
>>>
>>> Please conduct your tests and report any issues on this thread so that I
>>> can work on them.
>>>
>>> *Testing EclipseLink and PostgreSQL*
>>> You can test EclipseLink and PostgreSQL by using this script [4]. The
>>> eclipselink-test-setup script will clone and publish the relevant artefacts.
>>>
>>> NB: Remember to shut down your Postgres server when doing component
>>> tests so that the embedded db can start.
>>>
>>> NB: Use this for full integration tests
>>> java -Ddemoserver.persistent=true -Dcustom.cassandra.contactPoints=
>>> 127.0.0.1:9042 -Dcassandra.cluster.user=cassandra
>>> -Dcassandra.cluster.pwd=password -Dcustom.postgresql.host=localhost
>>> -Dcustom.postgresql.user=postgres -jar
>>> ~/<fineract-cn-dir>/integration-tests/fineract-cn-demo-server/build/libs/demo-server-0.1.0-BUILD-SNAPSHOT.jar
>>>
>>> NB: The logs are a bit excessive as I'm currently outputting all
>>> EclipseLink logs but once testing is over, I will tone it down.
>>>
>>> I have conducted component test for each microservice and a full
>>> integration test. A few tests failed during the component test however, the
>>> causes are not from the ORM but the test cases themselves. I will try and
>>> fix them once I have met all of my milestones for this project or raise
>>> issues for them in Jira.
>>>
>>> *Known Issues in Postgres (My current focus):*
>>> 1. ERROR: relation "khepri_beats" does not exist. click on the link at
>>> [5] to see the full log. This issue is not directly related to the ORM. I
>>> actually identified this error whiles testing Postgres and Hibernate. This
>>> is related to Postgres and I am currently looking into it.
>>>
>>> 2. Error: org.postgresql.util.PSQLException: FATAL: sorry, too many
>>> clients already. click on the link [6] to see the full log.
>>>
>>> *Solution*, Increase max_connections found in the postgresql.conf. *260*
>>> worked for me.
>>> Rational, each microservice can have up to 20 unique connections to the
>>> db per the BoneCP configuration.
>>>
>>> Also, you can test EclipseLink and MariaDB by pulling the FINCN-2 branch
>>> into fineract-cn-mariadb link [7] and rebuilding all of the microservices.
>>>
>>> *Identified Issue when using embedded dbs in full integration test *
>>> Tenant context not set, resulting in this error [8]
>>>
>>> Which brings me to my concern about MariaDB. Given that MariaDB is not
>>> compliant with Apache, can I safely assume that it's not relevant to
>>> migrate it's ORM as well?
>>>
>>> Looking forward to hearing from you all.
>>>
>>> *At your service,*
>>> Graham.
>>>
>>> [1]
>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L76-L80
>>> [2]
>>> https://github.com/ebenezergraham/fineract-cn-postgresql/blob/FINCN-2/build.gradle#L74
>>> / https://www.baeldung.com/spring-eclipselink
>>> [3]
>>> https://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Static_Weaving
>>> https://pastebin.com/FDAzd6bx
>>> [4]
>>> https://raw.githubusercontent.com/ebenezergraham/fineract-cn-demo-server/FINCN-2/scripts/Unix/eclipselink-test-setup.sh
>>> [5] https://pastebin.com/WFQ9satu
>>> [6] https://pastebin.com/M8GAHsit
>>> [7] https://github.com/ebenezergraham/fineract-cn-mariadb/tree/FINCN-2
>>> [8] https://pastebin.com/FDAzd6bx
>>>
>>