You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fineract.apache.org by Ed Cable <ed...@mifos.org> on 2018/09/18 15:26:46 UTC
Re: Status of Demo Server on Apache VM Infrastructure
Hi all,
Just to bring this thread back up to top of everyone's inbox. Aleks
estimates he has about a day left to get the demo server up and running is
hoping to tackle it by tomorrow. Here are the two approaches he suggested
now that we have the second VM allocated:
- continue with the JUnit based startup sequence of the demo module; the
one thing that I would need to change is to split up the startup sequence
and recompile
- create a set of Docker Compose files to do the startup on the 2 VM
instances (maybe I could borrow some stuff from Courage)
I would prefer the latter, because you have more fine grained control and
it's easier move service instances around... and you don't have to
recompile anything.
Ed
On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <my...@apache.org> wrote:
> Hey Aleks,
>
> I don’t see anything in that log that would explain the behavior you’re
> observing. Markus did most of the work on deposits. Markus? Do you have any
> ideas?
>
> Best Regards,
> Myrle
>
>
>
> On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
> cheetah@monkeysintown.com>
> wrote:
>
> > @Myrle: with your proposed changes things look a bit better. I've
> disabled
> > Rhythm as you suggested and get past all microservice startups with a lot
> > less exceptions (one is coming repeatedly from Netty native transport,
> > because epoll is not available).
> >
> > The one exception (actually it happened on 2 occasions) that is still
> > there:
> >
> > [snip]
> >
> > 13:33:45.080 [main] INFO
> o.a.f.c.d.s.DepositAccountManagementApplication -
> > Started DepositAccountManagementApplication in 55.187 seconds (JVM
> running
> > for 62.772)
> > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
> > configuration
> > Resolving artifact
> > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > Resolving metadata
> >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > from /root/.m2/repository (enhanced)
> > Resolved metadata
> >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > from /root/.m2/repository (enhanced)
> > Resolved artifact
> > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT from
> > /root/.m2/repository (enhanced)
> > 13:34:56.800 [DiscoveryClient-0] ERROR c.n.discovery.TimedSupervisorTask
> -
> > task supervisor timed out
> > java.util.concurrent.TimeoutException: null
> > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > at
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 13:34:56.801 [DiscoveryClient-1] ERROR c.n.discovery.TimedSupervisorTask
> -
> > task supervisor timed out
> > java.util.concurrent.TimeoutException: null
> > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > at
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 13:34:56.916 [DiscoveryClient-2] ERROR c.n.discovery.TimedSupervisorTask
> -
> > task supervisor timed out
> > java.util.concurrent.TimeoutException: null
> > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > at
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 13:35:08.586 [main] INFO o.s.c.a.AnnotationConfigApplicationContext -
> > Refreshing
> >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > :
> > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context hierarchy
> > 13:35:10.182 [main] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
> -
> > JSR-330 'javax.inject.Inject' annotation found and supported for
> autowiring
> > 13:35:10.371 [main] INFO
> > o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean
> > 'configurationPropertiesRebinderAutoConfiguration' of type [class
> >
> >
> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
> > is not eligible for getting processed by all BeanPostProcessors (for
> > example: not eligible for auto-proxying)
> > 13:35:10.435 [background-preinit] INFO
> o.h.validator.internal.util.Version
> > - HV000001: Hibernate Validator 5.3.0.Final
> >
> > [/snip]
> >
> > It looks like the whole startup procedure goes through and I can see some
> > kind of "summary" that lists all REST endpoints. But it stays in that
> state
> > for only a minute or so and then a cascade of shutdowns are triggered
> > again. The only thing that stays alive is Eureka which complains about
> the
> > inaccessible services.
> >
> > Any idea what else we could do other than just ordering an additional
> > machine?
> >
> > Cheers,
> >
> > Aleks
> >
> > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <my...@apache.org> wrote:
> >
> > > Hey Aleks,
> > >
> > > It looks like that exception is coming from Rhythm. I suspect it's not
> > the
> > > cause of your difficulties, since you say your other services are going
> > > down too. You can run most of the Fineract server without Rhythm. You
> > > could edit the demo-server script to leave it out for the purposes of
> > > testing. The consequence will be a lack of interest calculations for
> > > disbursed loans, but for finding out what the real problem is, it'd be
> > good
> > > to get this one out of the way.
> > >
> > > Best Regards,
> > > Myrle
> > >
> > > (FYI: I've removed the non-list subscribers from the to. Most of them
> > are
> > > already subscribed, and those who aren't, don't want to be.)
> > >
> > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
> > > cheetah@monkeysintown.com> wrote:
> > >
> > > > I managed to further improve the startup procedure (aka less
> exceptions
> > > > than before)... there are still some timeouts, but not as many as
> > before.
> > > >
> > > > The demo server still dies after this exception:
> > > >
> > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
> > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
> > > > configuration
> > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
> > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
> > > > configuration
> > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
> > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
> > > > configuration
> > > > 23:28:45.990 [pool-9-thread-1] ERROR
> > > o.s.s.s.TaskUtils$LoggingErrorHandler
> > > > - Unexpected error occurred in scheduled task.
> > > > org.springframework.transaction.CannotCreateTransactionException:
> Could
> > > not
> > > > open JPA EntityManager for transaction; nested exception is
> > > > javax.persistence.PersistenceException:
> > > > java.sql.SQLNonTransientConnectionException: Could not read
> resultset:
> > > > unexpected end of stream, read 0 bytes from 4
> > > > Query is : set autocommit=0
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > at
> > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > Caused by: javax.persistence.PersistenceException:
> > > > java.sql.SQLNonTransientConnectionException: Could not read
> resultset:
> > > > unexpected end of stream, read 0 bytes from 4
> > > > Query is : set autocommit=0
> > > > at
> > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
> > > > at
> > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
> > > > at
> > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
> > > > at
> > > >
> > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
> > > > ... 20 common frames omitted
> > > > Caused by: java.sql.SQLNonTransientConnectionException: Could not
> read
> > > > resultset: unexpected end of stream, read 0 bytes from 4
> > > > Query is : set autocommit=0
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
> > > > at
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
> > > > at
> > > >
> > > >
> > >
> >
> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
> > > > at
> > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
> > > > at
> > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
> > > > at
> > > >
> > > >
> > >
> >
> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
> > > > at
> > > >
> > > >
> > >
> >
> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
> > > > at
> > > >
> > >
> >
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
> > > > at
> > > >
> > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
> > > > ... 22 common frames omitted
> > > > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
> not
> > > > read resultset: unexpected end of stream, read 0 bytes from 4
> > > > Query is : set autocommit=0
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
> > > > ... 31 common frames omitted
> > > > Caused by: java.io.EOFException: unexpected end of stream, read 0
> bytes
> > > > from 4
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
> > > > at
> > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
> > > > ... 33 common frames omitted
> > > > 23:29:05.500 [qtp6422064-20] INFO
> o.a.f.c.l.c.ServiceExceptionFilter -
> > > > Responding with a service error ServiceError{code=409,
> > > message='Application
> > > > identity-v1 already exists!'}
> > > > 23:29:21.282 [Thread-15] INFO
> > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Closing
> > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > :
> > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
> > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > 23:29:23.717 [Thread-15] INFO o.s.c.s.DefaultLifecycleProcessor -
> > > Stopping
> > > > beans in phase 2147483647
> > > > 23:29:24.267 [Thread-15] INFO o.s.c.s.DefaultLifecycleProcessor -
> > > Stopping
> > > > beans in phase 0
> > > > 23:29:24.557 [Thread-15] INFO o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > Unregistering JMX-exposed beans on shutdown
> > > > 23:29:24.579 [Thread-15] INFO o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > Unregistering JMX-exposed beans
> > > >
> > > > The only thing I've left: start the databases (aka Cassandra) on a
> > > separate
> > > > machine (e. g. on Digital Ocean) and see how this is working then.
> > > >
> > > > ... FYI.
> > > >
> > > > Cheers,
> > > >
> > > > Aleks
> > > >
> > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
> > > > cheetah@monkeysintown.com> wrote:
> > > >
> > > > > @Rajan: it's not running, because the demo server dies... See logs
> > from
> > > > > previous message...
> > > > >
> > > > > We just have to figure that one out and we should be good to go.
> > > > >
> > > > >
> > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
> rajanmaurya154@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Thanks, Alex,
> > > > >>
> > > > >> I can see web app is running but I can't log in with the
> credentials
> > > > that
> > > > >> you shared earlier
> > > > >>
> > > > >> Tenant : playground
> > > > >> Username : operator
> > > > >> Password : init1@l
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
> > > > >> cheetah@monkeysintown.com> wrote:
> > > > >>
> > > > >> > Hi all,
> > > > >> >
> > > > >> > ... sweeeet! I got a considerable step further.
> > > > >> >
> > > > >> > @Markus: thanks for the conversation... gave me an idea how to
> fix
> > > > (most
> > > > >> > of) it!
> > > > >> >
> > > > >> > The problem was - as suspected - the memory (or lack of). The VM
> > > > >> instance
> > > > >> > at Apache has net 30G available. 16G are needed for Cassandra
> and
> > in
> > > > my
> > > > >> > previous tests I tried to run the demo server with up to 16G.
> The
> > > > >> problem
> > > > >> > is that the demo server needs considerably more memory than 16G.
> > > > >> According
> > > > >> > to my observation it's around 23-24G. Even if you limit the demo
> > > > server
> > > > >> > memory allocation with "-Xmx" to 12G (for example) it would just
> > > > >> continue
> > > > >> > to consume more memory.
> > > > >> >
> > > > >> > My solution: add a swap file. Not ideal in terms of performance,
> > but
> > > > it
> > > > >> > gets the whole thing going... at least mostly (please read on).
> > > > >> >
> > > > >> > Current status:
> > > > >> >
> > > > >> > - Cassandra (and all other Docker containers) are still
> > running;
> > > > >> > Cassandra died in my previous attempts at the latest when the
> > > > Teller
> > > > >> > application started, but with the swap file it's still
> running
> > > > >> > - all Spring Boot apps are starting now
> > > > >> > - I get once in a while some spurious timeouts (I think when
> > > either
> > > > >> the
> > > > >> > Eureka server is contacted or the config server)
> > > > >> > - there are also sometimes exceptions concerning MySQL
> > > connections,
> > > > >> but
> > > > >> > doesn't seem to matter overall
> > > > >> > - I could issue an authentication request with Postman to
> > > > >> > http://fineract-vm.apache.org:2021/identity/v1; just to test
> > if
> > > > >> > anything
> > > > >> > is responding
> > > > >> > - PROBLEM: but the demo server (the Spring Boot apps)
> > eventually
> > > > dies
> > > > >> > after a couple of minutes; I don't think that the memory is
> the
> > > > >> problem
> > > > >> > at
> > > > >> > this point; I think it's more of a timeout problem concerning
> > the
> > > > >> > communication with Eureka and/or the config server (maybe
> > because
> > > > the
> > > > >> > system is a bit slow)
> > > > >> >
> > > > >> > ... and here the memory footprint ("free -h") when everything is
> > > > >> running,
> > > > >> > just FYI:
> > > > >> >
> > > > >> > total used free shared
> > buff/cache
> > > > >> > available
> > > > >> > Mem: 31G 31G 226M 8.4M
> > 169M
> > > > >> > 47M
> > > > >> > Swap: 15G 6.8G 9.2G
> > > > >> >
> > > > >> > ... after the Payroll application is started (approx.) I see
> these
> > > > >> > exceptions appearing in the logs (a whole series of them):
> > > > >> >
> > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
> > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > >> > configuration
> > > > >> > Resolving artifact
> > > > >> >
> org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > >> > Resolving metadata
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > >> > from /root/.m2/repository (enhanced)
> > > > >> > Resolved metadata
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > >> > from /root/.m2/repository (enhanced)
> > > > >> > Resolved artifact
> > > > >> >
> org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > from
> > > > >> > /root/.m2/repository (enhanced)
> > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
> > > > >> c.n.discovery.TimedSupervisorTask -
> > > > >> > task supervisor timed out
> > > > >> > java.util.concurrent.TimeoutException: null
> > > > >> > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > >> > at
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > >> > at
> > > > >>
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > >> > at
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > >> > at
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > >> > at
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > >> > at
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > >> > at java.lang.Thread.run(Thread.java:748)
> > > > >> >
> > > > >> > ... but it still seems to have no problem starting the next app
> > > > (Group)
> > > > >> and
> > > > >> > things look good for a while...
> > > > >> >
> > > > >> > ... until suddenly it seems that app instances are beginning to
> > > > >> shutdown...
> > > > >> > and I'm not sure why this happens:
> > > > >> >
> > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
> > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > >> > configuration
> > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
> > > > >> c.datastax.driver.core.ClockFactory -
> > > > >> > Using native clock to generate timestamps.
> > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
> > > > >> c.datastax.driver.core.CodecRegistry
> > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
> > > > >> java.time.LocalDateTime]
> > > > >> > because it collides with previously registered codec
> > > > LocalDateTimeCodec
> > > > >> > [timestamp <-> java.time.LocalDateTime]
> > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
> > > > c.d.d.c.p.DCAwareRoundRobinPolicy
> > > > >> -
> > > > >> > Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy
> > (if
> > > > >> this
> > > > >> > is incorrect, please provide the correct datacenter name with
> > > > >> > DCAwareRoundRobinPolicy constructor)
> > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
> > > > com.datastax.driver.core.Cluster -
> > > > >> > New Cassandra host /127.0.0.1:9042 added
> > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
> > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > >> > configuration
> > > > >> > 18:27:51.110 [Thread-15] INFO
> > > > >> > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> Closing
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > >> > :
> > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
> > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > >> > configuration
> > > > >> > 18:27:53.373 [Thread-15] INFO
> o.s.c.s.DefaultLifecycleProcessor -
> > > > >> Stopping
> > > > >> > beans in phase 2147483647
> > > > >> > 18:27:54.331 [Thread-15] INFO
> o.s.c.s.DefaultLifecycleProcessor -
> > > > >> Stopping
> > > > >> > beans in phase 0
> > > > >> > 18:27:54.736 [Thread-15] INFO
> > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > >> > 18:27:54.740 [Thread-15] INFO
> > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > >> > Unregistering JMX-exposed beans
> > > > >> > 18:27:54.798 [Thread-15] INFO
> o.s.j.e.a.AnnotationMBeanExporter -
> > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > >> > 18:27:54.800 [Thread-15] INFO
> o.s.j.e.a.AnnotationMBeanExporter -
> > > > >> > Unregistering JMX-exposed beans
> > > > >> > 18:27:54.937 [Thread-15] INFO
> > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean - Closing JPA
> > > > >> > EntityManagerFactory for persistence unit 'metaPU'
> > > > >> > 18:27:58.376 [async-processor-1] INFO
> > > > >> o.f.c.internal.util.VersionPrinter -
> > > > >> > Flyway 3.2.1 by Boxfuse
> > > > >> > 18:27:58.677 [async-processor-1] INFO
> > > > >> o.f.c.i.dbsupport.DbSupportFactory -
> > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL 10.3)
> > > > >> > 18:28:00.620 [async-processor-1] INFO
> > > > >> o.f.core.internal.command.DbValidate
> > > > >> > - Validated 2 migrations (execution time 00:01.169s)
> > > > >> > 18:28:00.901 [Thread-15] INFO
> o.e.jetty.server.AbstractConnector
> > -
> > > > >> Stopped
> > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{0.0.0.0:2020}
> > > > >> > 18:28:01.268 [Thread-15] INFO
> > o.e.j.server.handler.ContextHandler -
> > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
> > > > >> >
> > > > >> > ... the list is actually a lot longer... in the rest of the log
> > you
> > > > can
> > > > >> see
> > > > >> > that one by one all the apps are shutting down.
> > > > >> >
> > > > >> > The last (error) entries I see are these:
> > > > >> >
> > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR
> > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not refresh JMS
> > > > >> Connection
> > > > >> > for destination 'cheques-v1' - retrying using
> > > > >> FixedBackOff{interval=5000,
> > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error while
> > > > >> attempting to
> > > > >> > retrieve a connection from the pool; nested exception is
> > > > >> > javax.jms.JMSException: Could not connect to broker URL:
> > > > >> > tcp://localhost:61616. Reason: java.net.ConnectException:
> > Connection
> > > > >> > refused (Connection refused)
> > > > >> >
> > > > >> > ... I guess an embedded ActiveMQ was also shutdown in the
> process
> > of
> > > > the
> > > > >> > previous errors.
> > > > >> >
> > > > >> > So far from here... any input on the above highly appreciated.
> > > > Overall I
> > > > >> > think we are close.
> > > > >> >
> > > > >> > Cheers,
> > > > >> >
> > > > >> > Aleks
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <ma...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> > > One thing we maybe can do, given we are part of the Apache
> > Family,
> > > > is
> > > > >> to
> > > > >> > > ask the Cassandra community for some suggestions.
> > > > >> > >
> > > > >> > > Cheers
> > > > >> > >
> > > > >> > > Markus
> > > > >> > >
> > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <mage@apache.org
> >
> > > > wrote:
> > > > >> > >
> > > > >> > > > Hey all,
> > > > >> > > >
> > > > >> > > > we are not running Cassandra in a container, we have
> dedicated
> > > VMs
> > > > >> to
> > > > >> > run
> > > > >> > > > our cluster.
> > > > >> > > >
> > > > >> > > > Given this we are not the best persons to help here, sorry.
> (;
> > > > >> > > >
> > > > >> > > > Cheers
> > > > >> > > >
> > > > >> > > > Markus
> > > > >> > > >
> > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
> edcable@mifos.org>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > >> Adding Patric directly to this thread too so he can give
> his
> > > > input
> > > > >> > > >> regarding to the challenges with Cassandra.
> > > > >> > > >>
> > > > >> > > >> Ed
> > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar Vidakovic <
> > > > >> > > >> cheetah@monkeysintown.com> wrote:
> > > > >> > > >>
> > > > >> > > >>> Hi Victor,
> > > > >> > > >>>
> > > > >> > > >>> .... this is the relevant part of the docker-compose.yml
> > file:
> > > > >> > > >>>
> > > > >> > > >>> cassandra0:
> > > > >> > > >>> image: cassandra:3.11.1
> > > > >> > > >>> container_name: cassandra0
> > > > >> > > >>> ports:
> > > > >> > > >>> - 9042:9042
> > > > >> > > >>> - 9160:9160
> > > > >> > > >>> - 7199:7199
> > > > >> > > >>> - 8778:8778
> > > > >> > > >>> volumes:
> > > > >> > > >>> - ./cassandra:/etc/cassandra
> > > > >> > > >>> environment:
> > > > >> > > >>> - CASSANDRA_START_RPC=true
> > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0
> > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster
> > > > >> > > >>> ulimits:
> > > > >> > > >>> memlock: -1
> > > > >> > > >>> nproc: 32768
> > > > >> > > >>> nofile: 100000
> > > > >> > > >>>
> > > > >> > > >>> ... and just to be complete... here's the Docker service
> > > > >> > configuration
> > > > >> > > >>> ("/lib/systemd/system/docker.service"):
> > > > >> > > >>>
> > > > >> > > >>> [Unit]
> > > > >> > > >>> Description=Docker Application Container Engine
> > > > >> > > >>> Documentation=https://docs.docker.com
> > > > >> > > >>> After=network-online.target docker.socket
> firewalld.service
> > > > >> > > >>> Wants=network-online.target
> > > > >> > > >>> Requires=docker.socket
> > > > >> > > >>>
> > > > >> > > >>> [Service]
> > > > >> > > >>> Type=notify
> > > > >> > > >>> # the default is not to use systemd for cgroups because
> the
> > > > >> delegate
> > > > >> > > >>> issues
> > > > >> > > >>> still
> > > > >> > > >>> # exists and systemd currently does not support the cgroup
> > > > feature
> > > > >> > set
> > > > >> > > >>> required
> > > > >> > > >>> # for containers run by docker
> > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
> > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
> > > > >> > > >>> LimitNOFILE=1048576
> > > > >> > > >>> # Having non-zero Limit*s causes performance problems due
> to
> > > > >> > accounting
> > > > >> > > >>> overhead
> > > > >> > > >>> # in the kernel. We recommend using cgroups to do
> > > > container-local
> > > > >> > > >>> accounting.
> > > > >> > > >>> LimitNPROC=infinity
> > > > >> > > >>> LimitCORE=infinity
> > > > >> > > >>> LimitMEMLOCK=infinity
> > > > >> > > >>> # Uncomment TasksMax if your systemd version supports it.
> > > > >> > > >>> # Only systemd 226 and above support this version.
> > > > >> > > >>> TasksMax=infinity
> > > > >> > > >>> TimeoutStartSec=0
> > > > >> > > >>> # set delegate yes so that systemd does not reset the
> > cgroups
> > > of
> > > > >> > docker
> > > > >> > > >>> containers
> > > > >> > > >>> Delegate=yes
> > > > >> > > >>> # kill only the docker process, not all processes in the
> > > cgroup
> > > > >> > > >>> KillMode=process
> > > > >> > > >>> # restart the docker process if it exits prematurely
> > > > >> > > >>> Restart=on-failure
> > > > >> > > >>> StartLimitBurst=3
> > > > >> > > >>> StartLimitInterval=60s
> > > > >> > > >>>
> > > > >> > > >>> [Install]
> > > > >> > > >>> WantedBy=multi-user.target
> > > > >> > > >>>
> > > > >> > > >>> ... the one tweak I did there is to set LimitMEMLOCK to
> > > > >> infinity...
> > > > >> > do
> > > > >> > > >>> you
> > > > >> > > >>> think the problems are file handle related? Should I set
> > > > >> LimitNOFILE
> > > > >> > > also
> > > > >> > > >>> to infinity?
> > > > >> > > >>>
> > > > >> > > >>> Appreciate the help.
> > > > >> > > >>>
> > > > >> > > >>> Cheers,
> > > > >> > > >>>
> > > > >> > > >>> Aleks
> > > > >> > > >>>
> > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
> > > > >> > > >>> victor.romero@fintecheando.mx>
> > > > >> > > >>> wrote:
> > > > >> > > >>>
> > > > >> > > >>> > Hi Aleks,
> > > > >> > > >>> >
> > > > >> > > >>> > Can you share the ulimits flags that the cassandra's
> > > container
> > > > >> is
> > > > >> > > using
> > > > >> > > >>> > while it is running?
> > > > >> > > >>> >
> > > > >> > > >>> > The values are being set in the composer file? Or in the
> > > > >> upstart or
> > > > >> > > >>> > systemd docker's deamon config files?
> > > > >> > > >>> >
> > > > >> > > >>> >
> > > > >> > > >>> >
> > > > >> > > >>> > Enviado desde TypeApp
> > > > >> > > >>> >
> > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM, Aleksandar
> > > Vidakovic <
> > > > >> > > >>> > cheetah@monkeysintown.com> escribió:
> > > > >> > > >>> > >... and I should have attached the Cassandra log
> dump...
> > so
> > > > >> here
> > > > >> > it
> > > > >> > > >>> > >is...
> > > > >> > > >>> > >
> > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar Vidakovic <
> > > > >> > > >>> > >cheetah@monkeysintown.com> wrote:
> > > > >> > > >>> > >
> > > > >> > > >>> > >> Hi all,
> > > > >> > > >>> > >>
> > > > >> > > >>> > >> ... as you might have noticed I did a couple of
> > restarts
> > > > >> > today...
> > > > >> > > >>> the
> > > > >> > > >>> > >> problem I am facing now: I can't get Cassandra to run
> > in
> > > a
> > > > >> > stable
> > > > >> > > >>> > >way.
> > > > >> > > >>> > >>
> > > > >> > > >>> > >> Things look quite OK for a while when running the
> demo
> > > > >> server,
> > > > >> > but
> > > > >> > > >>> > >then
> > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts
> > misbehaving
> > > > with
> > > > >> > the
> > > > >> > > >>> > >deposit
> > > > >> > > >>> > >> microservice startup, sometimes with portfolio or
> > > teller).
> > > > I
> > > > >> > tried
> > > > >> > > >>> to
> > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set some
> > Docker
> > > > >> limits
> > > > >> > to
> > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
> > > > >> > > >>> > >>
> > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe someone
> can
> > > > help
> > > > >> out
> > > > >> > > >>> > >here? Is
> > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB of
> memory?
> > > > >> > > >>> > >>
> > > > >> > > >>> > >> Other than that the setup would be ready to go... we
> > just
> > > > >> need
> > > > >> > to
> > > > >> > > >>> get
> > > > >> > > >>> > >the
> > > > >> > > >>> > >> database running more reliably.
> > > > >> > > >>> > >>
> > > > >> > > >>> > >> Cheers,
> > > > >> > > >>> > >>
> > > > >> > > >>> > >> Aleks
> > > > >> > > >>> > >>
> > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar Vidakovic
> <
> > > > >> > > >>> > >> cheetah@monkeysintown.com> wrote:
> > > > >> > > >>> > >>
> > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed to add
> some
> > > > >> > additional
> > > > >> > > >>> > >reverse
> > > > >> > > >>> > >>> proxy configuration for the web UI to work)... and
> > there
> > > > >> were
> > > > >> > > more
> > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
> > > > >> > > >>> > >>>
> > > > >> > > >>> > >>> Just FYI
> > > > >> > > >>> > >>>
> > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
> Vidakovic <
> > > > >> > > >>> > >>> cheetah@monkeysintown.com> wrote:
> > > > >> > > >>> > >>>
> > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can get back
> > to
> > > > >> testing
> > > > >> > > >>> only
> > > > >> > > >>> > >>>> later tonight.
> > > > >> > > >>> > >>>>
> > > > >> > > >>> > >>>> Cheers
> > > > >> > > >>> > >>>>
> > > > >> > > >>> > >>>>
> > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya
> > > > >> > > >>> > ><ra...@gmail.com>
> > > > >> > > >>> > >>>> wrote:
> > > > >> > > >>> > >>>>
> > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test after 30 min.
> > > > >> > > >>> > >>>>>
> > > > >> > > >>> > >>>>> Big thanks for this 🙂
> > > > >> > > >>> > >>>>>
> > > > >> > > >>> > >>>>>
> > > > >> > > >>> > >>>>>
> > > > >> > > >>> > >>>>>
> > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM Aleksandar
> > Vidakovic <
> > > > >> > > >>> > >>>>> cheetah@monkeysintown.com> wrote:
> > > > >> > > >>> > >>>>>
> > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted.
> > > > >> > > >>> > >>>>> >
> > > > >> > > >>> > >>>>> > ... and as I said: the services are still
> > > starting...
> > > > >> and
> > > > >> > the
> > > > >> > > >>> > >best
> > > > >> > > >>> > >>>>> estimate
> > > > >> > > >>> > >>>>> > I have right now (as already mentioned): 30min
> > > > >> > > >>> > >>>>> >
> > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan Maurya <
> > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > >> > > >>> > >>>>> > wrote:
> > > > >> > > >>> > >>>>> >
> > > > >> > > >>> > >>>>> > > [image: image.png]
> > > > >> > > >>> > >>>>> > >
> > > > >> > > >>> > >>>>> > >
> > > > >> > > >>> > >>>>> > >
> > > > >> > > >>> > >>>>> > >
> > > > >> > > >>> > >>>>> > >
> > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan Maurya <
> > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > >> > > >>> > >>>>> > > wrote:
> > > > >> > > >>> > >>>>> > >
> > > > >> > > >>> > >>>>> > >> [image: image.png]
> > > > >> > > >>> > >>>>> > >> I am getting this.
> > > > >> > > >>> > >>>>> > >>
> > > > >> > > >>> > >>>>> > >>
> > > > >> > > >>> > >>>>> > >>
> > > > >> > > >>> > >>>>> > >>
> > > > >> > > >>> > >>>>> > >>
> > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM Aleksandar
> > > > Vidakovic
> > > > >> <
> > > > >> > > >>> > >>>>> > >> cheetah@monkeysintown.com> wrote:
> > > > >> > > >>> > >>>>> > >>
> > > > >> > > >>> > >>>>> > >>> Hi all,
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost) ready...
> it
> > > took
> > > > >> me a
> > > > >> > > >>> > >moment
> > > > >> > > >>> > >>>>> and a
> > > > >> > > >>> > >>>>> > >>> couple
> > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot
> > failures...
> > > > the
> > > > >> > > >>> services
> > > > >> > > >>> > >are
> > > > >> > > >>> > >>>>> quite
> > > > >> > > >>> > >>>>> > >>> resource hungry and default settings won't
> do
> > > it.
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> The services are still starting and this
> will
> > > > take a
> > > > >> > > while
> > > > >> > > >>> > >(my
> > > > >> > > >>> > >>>>> best
> > > > >> > > >>> > >>>>> > guess
> > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
> > > > >> > > >>> > >http://fineract-vm.apache.org/login
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> Credentials:
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> Tenant : playground
> > > > >> > > >>> > >>>>> > >>> Username : operator
> > > > >> > > >>> > >>>>> > >>> Password : init1@l
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to compile
> this
> > > app
> > > > >> is
> > > > >> > > with
> > > > >> > > >>> > >NodeJS
> > > > >> > > >>> > >>>>> > 8.11.1
> > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't work).
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> Identity Service:
> > > > >> > > >>> > >http://fineract-vm.apache.org:2021/identity/v1
> > > > >> > > >>> > >>>>> > >>> Office Service:
> > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
> > > > >> > > >>> > >>>>> > >>> Customer Service:
> > > > >> > > >>> > >http://fineract-vm.apache.org:2024/customer/v1
> > > > >> > > >>> > >>>>> > >>> Accounting Service:
> > > > >> > > >>> > >>>>> http://fineract-vm.apache.org:2025/accounting/v1
> > > > >> > > >>> > >>>>> > >>> Portfolio Service:
> > > > >> > > >>> > >>>>> http://fineract-vm.apache.org:2026/portfolio/v1
> > > > >> > > >>> > >>>>> > >>> Deposit Service:
> > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1
> > > > >> > > >>> > >>>>> > >>> Teller Service:
> > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
> > > > >> > > >>> > >>>>> > >>> Reporting Service:
> > > > >> > > >>> > >>>>> http://fineract-vm.apache.org:2029/reporting/v1
> > > > >> > > >>> > >>>>> > >>> Cheque Service:
> > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1
> > > > >> > > >>> > >>>>> > >>> Payroll Service:
> > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes quite a
> > > > >> while... if
> > > > >> > > you
> > > > >> > > >>> > >>>>> encounter
> > > > >> > > >>> > >>>>> > >>> connection problems retry a couple of
> minutes
> > > > later
> > > > >> to
> > > > >> > > >>> > >ensure I
> > > > >> > > >>> > >>>>> am not
> > > > >> > > >>> > >>>>> > >>> currently working on something.
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the backend
> > services
> > > > >> once
> > > > >> > per
> > > > >> > > >>> > >day or
> > > > >> > > >>> > >>>>> so to
> > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to figure
> out
> > > how
> > > > we
> > > > >> > want
> > > > >> > > >>> to
> > > > >> > > >>> > >>>>> handle
> > > > >> > > >>> > >>>>> > this
> > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> If you have any suggestions where to put
> this
> > > demo
> > > > >> > server
> > > > >> > > >>> > >>>>> configuration
> > > > >> > > >>> > >>>>> > >>> then let me know here.
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any problems (I
> > > have
> > > > >> not
> > > > >> > > >>> > >extensively
> > > > >> > > >>> > >>>>> > tested
> > > > >> > > >>> > >>>>> > >>> it
> > > > >> > > >>> > >>>>> > >>> yet).
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> Cheers,
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> Aleks
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM Aleksandar
> > > > >> Vidakovic <
> > > > >> > > >>> > >>>>> > >>> cheetah@monkeysintown.com> wrote:
> > > > >> > > >>> > >>>>> > >>>
> > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem to be
> > starting
> > > > fine
> > > > >> > > now...
> > > > >> > > >>> > >just
> > > > >> > > >>> > >>>>> have
> > > > >> > > >>> > >>>>> > to
> > > > >> > > >>> > >>>>> > >>> > work on a couple of improvements on the
> > config
> > > > >> files.
> > > > >> > > >>> > >>>>> > >>> >
> > > > >> > > >>> > >>>>> > >>> > Currently having another stab to get the
> > > webapp
> > > > >> > > running.
> > > > >> > > >>> > >>>>> > >>> >
> > > > >> > > >>> > >>>>> > >>> > BTW: it should be fairly easy to adapt the
> > > > >> simplistic
> > > > >> > > >>> > >Docker
> > > > >> > > >>> > >>>>> Compose
> > > > >> > > >>> > >>>>> > >>> file
> > > > >> > > >>> > >>>>> > >>> > for a CI pipeline.
> > > > >> > > >>> > >>>>> > >>> >
> > > > >> > > >>> > >>>>> > >>> > Will keep you posted.
> > > > >> > > >>> > >>>>> > >>> >
> > > > >> > > >>> > >>>>> > >>> > Cheers,
> > > > >> > > >>> > >>>>> > >>> >
> > > > >> > > >>> > >>>>> > >>> > Aleks
> > > > >> > > >>> > >>>>> > >>> >
> > > > >> > > >>> > >>>>> > >>> > On Mon, Jun 11, 2018 at 9:41 AM Aleksandar
> > > > >> Vidakovic
> > > > >> > <
> > > > >> > > >>> > >>>>> > >>> > cheetah@monkeysintown.com> wrote:
> > > > >> > > >>> > >>>>> > >>> >
> > > > >> > > >>> > >>>>> > >>> >> I am still working on it... Will ping you
> > > here
> > > > >> when
> > > > >> > > it's
> > > > >> > > >>> > >ready
> > > > >> > > >>> > >>>>> (hope
> > > > >> > > >>> > >>>>> > >>> >> soon).
> > > > >> > > >>> > >>>>> > >>> >>
> > > > >> > > >>> > >>>>> > >>> >> On Mon, Jun 11, 2018, 9:40 AM Rajan
> Maurya
> > <
> > > > >> > > >>> > >>>>> > rajanmaurya154@gmail.com>
> > > > >> > > >>> > >>>>> > >>> >> wrote:
> > > > >> > > >>> > >>>>> > >>> >>
> > > > >> > > >>> > >>>>> > >>> >>> Hi Alex,
> > > > >> > > >>> > >>>>> > >>> >>>
> > > > >> > > >>> > >>>>> > >>> >>> Is the demo server up, If yes then
> what's
> > > the
> > > > >> base
> > > > >> > > url
> > > > >> > > >>> > >so I
> > > > >> > > >>> > >>>>> can
> > > > >> > > >>> > >>>>> > >>> access.
> > > > >> > > >>> > >>>>> > >>> >>>
> > > > >> > > >>> > >>>>> > >>> >>>
> > > > >> > > >>> > >>>>> > >>> >>>
> > > > >> > > >>> > >>>>> > >>> >>>
> > > > >> > > >>> > >>>>> > >>> >>> On Fri, Jun 8, 2018 at 2:03 PM Rajan
> > Maurya
> > > <
> > > > >> > > >>> > >>>>> > >>> rajanmaurya154@gmail.com>
> > > > >> > > >>> > >>>>> > >>> >>> wrote:
> > > > >> > > >>> > >>>>> > >>> >>>
> > > > >> > > >>> > >>>>> > >>> >>> > Hi Alex,
> > > > >> > > >>> > >>>>> > >>> >>> >
> > > > >> > > >>> > >>>>> > >>> >>> > This is really great news, I will wait
> > > until
> > > > >> > then.
> > > > >> > > >>> > >>>>> > >>> >>> >
> > > > >> > > >>> > >>>>> > >>> >>> > Thanks
> > > > >> > > >>> > >>>>> > >>> >>> > Rajan Maurya
> > > > >> > > >>> > >>>>> > >>> >>> >
> > > > >> > > >>> > >>>>> > >>> >>> >
> > > > >> > > >>> > >>>>> > >>> >>> > On Fri, Jun 8, 2018 at 11:36 AM
> > Aleksandar
> > > > >> > > Vidakovic
> > > > >> > > >>> <
> > > > >> > > >>> > >>>>> > >>> >>> > cheetah@monkeysintown.com> wrote:
> > > > >> > > >>> > >>>>> > >>> >>> >
> > > > >> > > >>> > >>>>> > >>> >>> >> Hi Rajan,
> > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > >> > > >>> > >>>>> > >>> >>> >> My plan is to get this up and running
> > > until
> > > > >> > > Monday.
> > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > >> > > >>> > >>>>> > >>> >>> >> Cheers,
> > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > >> > > >>> > >>>>> > >>> >>> >> Aleks
> > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > >> > > >>> > >>>>> > >>> >>> >> On Fri, Jun 8, 2018, 7:23 AM Rajan
> > > Maurya <
> > > > >> > > >>> > >>>>> > >>> rajanmaurya154@gmail.com>
> > > > >> > > >>> > >>>>> > >>> >>> >> wrote:
> > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > >> > > >>> > >>>>> > >>> >>> >> > Hi Markus, Mark, Myrle, Alex, Ed
> > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > >> > > >>> > >>>>> > >>> >>> >> > I am mentoring fineract-cn-mobile
> > GSoC
> > > > >> > project.
> > > > >> > > I
> > > > >> > > >>> > >am
> > > > >> > > >>> > >>>>> waiting
> > > > >> > > >>> > >>>>> > for
> > > > >> > > >>> > >>>>> > >>> >>> this
> > > > >> > > >>> > >>>>> > >>> >>> >> demo
> > > > >> > > >>> > >>>>> > >>> >>> >> > server for a long time to see
> what's
> > > most
> > > > >> > > >>> important
> > > > >> > > >>> > >to
> > > > >> > > >>> > >>>>> > implement
> > > > >> > > >>> > >>>>> > >>> >>> in-app
> > > > >> > > >>> > >>>>> > >>> >>> >> > first.
> > > > >> > > >>> > >>>>> > >>> >>> >> > My GSoC students almost finished
> the
> > > task
> > > > >> that
> > > > >> > > was
> > > > >> > > >>> > >>>>> necessary.
> > > > >> > > >>> > >>>>> > We
> > > > >> > > >>> > >>>>> > >>> >>> have
> > > > >> > > >>> > >>>>> > >>> >>> >> kinda
> > > > >> > > >>> > >>>>> > >>> >>> >> > of stuck and We really need this
> demo
> > > > >> server
> > > > >> > and
> > > > >> > > >>> > >web app
> > > > >> > > >>> > >>>>> to
> > > > >> > > >>> > >>>>> > >>> assign
> > > > >> > > >>> > >>>>> > >>> >>> them
> > > > >> > > >>> > >>>>> > >>> >>> >> > tasks.
> > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > >> > > >>> > >>>>> > >>> >>> >> > It would be nice we can get the
> demo
> > > > >> server up
> > > > >> > > >>> > >ASAP.
> > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > >> > > >>> > >>>>> > >>> >>> >> > Thanks
> > > > >> > > >>> > >>>>> > >>> >>> >> > Rajan Maurya
> > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > >> > > >>> > >>>>> > >>> >>> >> > On Fri, Jun 1, 2018 at 6:41 PM Mark
> > van
> > > > >> Veen <
> > > > >> > > >>> > >>>>> > >>> >>> mark.vanveen@kuelap.io>
> > > > >> > > >>> > >>>>> > >>> >>> >> > wrote:
> > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > Hi Aleks,
> > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > Re your second issue:
> > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > It looks like that node-sass has
> > not
> > > > been
> > > > >> > > >>> > >installed
> > > > >> > > >>> > >>>>> > correctly.
> > > > >> > > >>> > >>>>> > >>> >>> Can you
> > > > >> > > >>> > >>>>> > >>> >>> >> > try
> > > > >> > > >>> > >>>>> > >>> >>> >> > > to remove the node_modules folder
> > and
> > > > run
> > > > >> > npm
> > > > >> > > >>> > >install
> > > > >> > > >>> > >>>>> again?
> > > > >> > > >>> > >>>>> > >>> This
> > > > >> > > >>> > >>>>> > >>> >>> >> should
> > > > >> > > >>> > >>>>> > >>> >>> >> > > run without errors.
> > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > Mark
> > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > On Thu, May 31, 2018 at 9:05 PM
> > > Courage
> > > > >> > Angeh
> > > > >> > > <
> > > > >> > > >>> > >>>>> > >>> >>> courageangeh@gmail.com
> > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > wrote:
> > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > Well, i can it with
> containerized
> > > > >> MariaDB
> > > > >> > > and
> > > > >> > > >>> > >it
> > > > >> > > >>> > >>>>> work till
> > > > >> > > >>> > >>>>> > >>> my
> > > > >> > > >>> > >>>>> > >>> >>> >> computer
> > > > >> > > >>> > >>>>> > >>> >>> >> > > ran
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > out of space (in the middle of
> > > > >> > provisioning)
> > > > >> > > >>> > >>>>> > >>> >>> >> > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > On Thu, May 31, 2018, 6:29 PM
> > > Victor
> > > > >> > Romero
> > > > >> > > <
> > > > >> > > >>> > >>>>> > >>> >>> >> > > victor.romero@fintecheando.mx
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > wrote:
> > > > >> > > >>> > >>>>> > >>> >>> >> > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > Hi Aleks,
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > Could you please share the
> > > composer
> > > > >> > file?
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > Courage,
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > Have you tested Mifos while
> > using
> > > > >> > MariaDb?
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > I have faced issues while
> > running
> > > > >> Mifos
> > > > >> > > with
> > > > >> > > >>> > >>>>> MariaDb.
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > Regards,
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > Victor
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > Enviado desde TypeApp
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > En may. 31, 2018 6:10 AM, en
> > 6:10
> > > > AM,
> > > > >> > > >>> Courage
> > > > >> > > >>> > >>>>> Angeh <
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > courageangeh@gmail.com>
> > > escribió:
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >You are welcome :-)
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >No, I am not.
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >On Thu, May 31, 2018 at
> 11:57
> > > AM,
> > > > >> > > >>> Aleksandar
> > > > >> > > >>> > >>>>> Vidakovic
> > > > >> > > >>> > >>>>> > <
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >cheetah@monkeysintown.com>
> > > wrote:
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >> Hi Courage,
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >>
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >> ... ok... I see one
> > difference
> > > > >> that I
> > > > >> > > >>> > >have: I
> > > > >> > > >>> > >>>>> used
> > > > >> > > >>> > >>>>> > >>> MySQL
> > > > >> > > >>> > >>>>> > >>> >>> (as
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >described on
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >> the top of the Wiki
> page)...
> > > > I'll
> > > > >> try
> > > > >> > > >>> > >MariaDB
> > > > >> > > >>> > >>>>> and
> > > > >> > > >>> > >>>>> > >>> should
> > > > >> > > >>> > >>>>> > >>> >>> get
> > > > >> > > >>> > >>>>> > >>> >>> >> the
> > > > >> > > >>> > >>>>> > >>> >>> >> > > same
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >> results as you.
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >>
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >> Thanks for providing your
> > > > command
> > > > >> > > line, I
> > > > >> > > >>> > >used
> > > > >> > > >>> > >>>>> the
> > > > >> > > >>> > >>>>> > >>> same.
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >>
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >> Are you working on the
> same
> > > > >> server?
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >>
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >> Cheers,
> > > > >> > > >>> > >>>>> > >>> >>> >> > > > > >>
> > > > >> > > >>> > >>>>> >
--
*Ed Cable*
President/CEO, Mifos Initiative
edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
*Collectively Creating a World of 3 Billion Maries | *http://mifos.org
<http://facebook.com/mifos> <http://www.twitter.com/mifos>
Fineract CN Demo Server Next Steps
Posted by Ed Cable <ed...@mifos.org>.
Hi all,
Starting a new thread here so it's a bit more manageable to open up and
reply to.
@Aleksandar Vidakovic <ch...@monkeysintown.com> can you give us an update
and more guidance on next steps in case someone is able to assist.
We're getting close!
Ed
On Fri, Dec 14, 2018 at 7:33 AM Ed Cable <ed...@mifos.org> wrote:
> Hi gentlemen,
>
> I just wanted to follow up on these two items once again.
>
> Thanks,
>
> Ed
>
> On Tue, Dec 4, 2018 at 2:53 PM Ed Cable <ed...@mifos.org> wrote:
>
>> @Avik Ganguly <av...@gmail.com> were you able to take up any of
>> the suggested tasks by Myrle?
>>
>> @Aleksandar Vidakovic <ch...@monkeysintown.com> Can you provide an
>> update on the status of deploying Fineract CN on the Apache VMs?
>>
>> Thanks,
>>
>> Ed
>>
>> On Thu, Oct 25, 2018 at 3:39 AM Myrle Krantz <my...@apache.org> wrote:
>>
>>> Hi Avik,
>>>
>>> You've asked several questions here, which is why it took me a while to
>>> get
>>> to answering.
>>>
>>> Before I go on, allow me to clarify my use of certain terms in the
>>> context
>>> of Fineract CN:
>>> * unit test -- a test for just one or two classes. Doesn't require
>>> spinning up the service. Doesn't require resources like a DB or user
>>> interaction. Is fast enough to be made part of the build. These tests
>>> are
>>> included in the same module and package as the class which they are
>>> intended to test. (This is a definition which is generally agreed upon in
>>> the industry.)
>>> * component test -- a test for a service. Requires spinning up just that
>>> service. Requires the use of an embedded DB. Individual tests might be
>>> fast, but the testsuite is rather expensive to run because startup times
>>> are impacted by running two DBs and spinning up the service. These tests
>>> are included in the same repository as the service they are intended to
>>> test. External services are mocked. (The industry has no widely agreed
>>> upon definition for component tests; this is my definition.)
>>> * integration test -- a test for multiple services. These tests have
>>> their
>>> own repositories.
>>>
>>> demo-server serves three purposes:
>>> * It illustrates the steps need to start and provision the services.
>>> * It serves as an integration test for one possible the provisioning
>>> workflow.
>>> * It makes it possible to start up the services locally so that a UI
>>> developer can program against the latest changes.
>>>
>>> demo-server is not intended to be a used in production for multiple
>>> reasons:
>>> * It's slow. It starts everything, even stuff you may not need, and it
>>> provisions everything. It then waits for everything to register itself
>>> with Eureka. It lowers the times for the eureka server to speed up
>>> startup, but this results in to many cycles being used for Eureka
>>> updates.
>>> * It's not secure. For example: identity.token.refresh.secureCookie is
>>> set
>>> to false to make http transport possible.
>>>
>>> If you're creating a new service, I don't expect that you'll need to run
>>> demo-server very often. Your emphasis should be on your component tests.
>>> These can run in just a minute or two, and you don't have to worry (much)
>>> about JWT tokens, headers, and the like inside of these tests.
>>>
>>> If demo-server doesn't serve your purposes I see a few possibilities:
>>> * create a private fork of the demo-server, then create the setup you
>>> prefer.
>>> * submit a PR with another "test" in demo-server which starts the
>>> services
>>> you need. I'd be happy to help you with this.
>>> * submit a PR which makes the starting of various services in the
>>> demo-service optional. I'd be happy to help you with this as well.
>>>
>>> One idea for speeding up local deployments (which is intended only to
>>> enable developers and not to be a production deployment method) is to
>>> start
>>> all the services in one process. This would reduce service startup times
>>> by folding them all into one. It would also replace inter-process
>>> communication during provisioning with in-process communication. I
>>> expect
>>> that we could get startup under 5 minutes. It would also make it
>>> possible
>>> to do component-testing on services which are dependent on other services
>>> without extensive, error-prone mocking. Writing tests for portfolio
>>> probably cost me more time than writing the code, and debugging mocks
>>> isn't
>>> fun. But those benefits would only apply for development environments.
>>> That's what FINCN-25 is about.
>>>
>>> Rhythm isn't resource-hungry. Not sure where that came from...
>>>
>>> I don't believe Cassandra is the problem here, but I haven't examined
>>> that
>>> closely. Please prove me wrong. I'd love to see a fix like that, if it
>>> really is that easy. : o)
>>>
>>> Best Regards,
>>> Myrle
>>>
>>>
>>>
>>> On Wed, Oct 10, 2018 at 6:24 PM Avik Ganguly <av...@gmail.com>
>>> wrote:
>>>
>>> > Hi guys,
>>> >
>>> > I am assuming that the suggested form of development for Fineract CN
>>> is to
>>> > write unit tests to test the code which you just wrote locally and then
>>> > push the code to a remote cluster or a massive VM and wait 40+ minutes
>>> to
>>> > run the API / integration tests there. A lot of the confusion
>>> regarding the
>>> > stacktraces in identity, rhythm, timeouts and auto-shutdown might or
>>> might
>>> > not have anything to do with the resource crunch in my local. (i7, 8GB,
>>> > SSD)
>>> >
>>> > Maybe a FAQ section in "How to build Fineract CN" to help devs with
>>> local
>>> > setup would go a long way to bring in contributions to the core
>>> modules.
>>> >
>>> > 1. Change demo server orchestration logic from running Identity,
>>> > Organization, Customer, Accounting, Portfolio, Deposit, Teller,
>>> Reporting,
>>> > Cheque, Payroll, Group and Notification services to running only
>>> Identity,
>>> > Organization, Customer and working set of your choice (Portfolio and
>>> > Accounting OR Deposit and Accounting OR Group and Portfolio)
>>> >
>>> > 2. Leave out Rhythm from orchestration logic for now. A JIRA ticket
>>> > tracking ability of Rhythm to run with low resource footprint will
>>> probably
>>> > help.
>>> >
>>> >
>>> > @Myrle, can you elaborate on FINCN-25? Is the motivation of this
>>> ticket to
>>> > reduce resource footprint only or also to reduce the startup time from
>>> 40
>>> > minutes to a reasonable couple of minutes? Do you have any debugging
>>> tips
>>> > with respect to the slow startup caused by services is due to
>>> registering
>>> > with eureka or provisioning or sequential startup?
>>> >
>>> > Maybe we should also follow up in Cassandra mailing list regarding
>>> > configuring Cassandra to run within 8GB of RAM and include the same
>>> config
>>> > as part of the FAQ.
>>> >
>>> >
>>> > Regards,
>>> > Avik.
>>> > ᐧ
>>> >
>>> > On Wed, Sep 19, 2018 at 2:23 AM Courage Angeh <co...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks for the update. Ed.
>>> > >
>>> > > If you need an extra hand please let me know Aleks.
>>> > >
>>> > > On Tue, Sep 18, 2018, 4:27 PM Ed Cable <ed...@mifos.org> wrote:
>>> > >
>>> > > > Hi all,
>>> > > >
>>> > > > Just to bring this thread back up to top of everyone's inbox. Aleks
>>> > > > estimates he has about a day left to get the demo server up and
>>> running
>>> > > is
>>> > > > hoping to tackle it by tomorrow. Here are the two approaches he
>>> > suggested
>>> > > > now that we have the second VM allocated:
>>> > > >
>>> > > > - continue with the JUnit based startup sequence of the demo
>>> module;
>>> > the
>>> > > > one thing that I would need to change is to split up the startup
>>> > sequence
>>> > > > and recompile
>>> > > > - create a set of Docker Compose files to do the startup on the 2
>>> VM
>>> > > > instances (maybe I could borrow some stuff from Courage)
>>> > > >
>>> > > > I would prefer the latter, because you have more fine grained
>>> control
>>> > and
>>> > > > it's easier move service instances around... and you don't have to
>>> > > > recompile anything.
>>> > > >
>>> > > > Ed
>>> > > >
>>> > > > On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <my...@apache.org>
>>> wrote:
>>> > > >
>>> > > > > Hey Aleks,
>>> > > > >
>>> > > > > I don’t see anything in that log that would explain the behavior
>>> > you’re
>>> > > > > observing. Markus did most of the work on deposits. Markus? Do
>>> you
>>> > have
>>> > > > any
>>> > > > > ideas?
>>> > > > >
>>> > > > > Best Regards,
>>> > > > > Myrle
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
>>> > > > > cheetah@monkeysintown.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > @Myrle: with your proposed changes things look a bit better.
>>> I've
>>> > > > > disabled
>>> > > > > > Rhythm as you suggested and get past all microservice startups
>>> > with a
>>> > > > lot
>>> > > > > > less exceptions (one is coming repeatedly from Netty native
>>> > > transport,
>>> > > > > > because epoll is not available).
>>> > > > > >
>>> > > > > > The one exception (actually it happened on 2 occasions) that is
>>> > still
>>> > > > > > there:
>>> > > > > >
>>> > > > > > [snip]
>>> > > > > >
>>> > > > > > 13:33:45.080 [main] INFO
>>> > > > > o.a.f.c.d.s.DepositAccountManagementApplication -
>>> > > > > > Started DepositAccountManagementApplication in 55.187 seconds
>>> (JVM
>>> > > > > running
>>> > > > > > for 62.772)
>>> > > > > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
>>> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>>> endpoints
>>> > via
>>> > > > > > configuration
>>> > > > > > Resolving artifact
>>> > > > > >
>>> org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
>>> > > > > > Resolving metadata
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
>>> > > > > > from /root/.m2/repository (enhanced)
>>> > > > > > Resolved metadata
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
>>> > > > > > from /root/.m2/repository (enhanced)
>>> > > > > > Resolved artifact
>>> > > > > >
>>> org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
>>> > > > from
>>> > > > > > /root/.m2/repository (enhanced)
>>> > > > > > 13:34:56.800 [DiscoveryClient-0] ERROR
>>> > > > c.n.discovery.TimedSupervisorTask
>>> > > > > -
>>> > > > > > task supervisor timed out
>>> > > > > > java.util.concurrent.TimeoutException: null
>>> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
>>> > > > > > at
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
>>> > > > > > at
>>> > > > >
>>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> > > > > > at java.lang.Thread.run(Thread.java:748)
>>> > > > > > 13:34:56.801 [DiscoveryClient-1] ERROR
>>> > > > c.n.discovery.TimedSupervisorTask
>>> > > > > -
>>> > > > > > task supervisor timed out
>>> > > > > > java.util.concurrent.TimeoutException: null
>>> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
>>> > > > > > at
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
>>> > > > > > at
>>> > > > >
>>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> > > > > > at java.lang.Thread.run(Thread.java:748)
>>> > > > > > 13:34:56.916 [DiscoveryClient-2] ERROR
>>> > > > c.n.discovery.TimedSupervisorTask
>>> > > > > -
>>> > > > > > task supervisor timed out
>>> > > > > > java.util.concurrent.TimeoutException: null
>>> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
>>> > > > > > at
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
>>> > > > > > at
>>> > > > >
>>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> > > > > > at
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> > > > > > at java.lang.Thread.run(Thread.java:748)
>>> > > > > > 13:35:08.586 [main] INFO
>>> > o.s.c.a.AnnotationConfigApplicationContext
>>> > > -
>>> > > > > > Refreshing
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
>>> > > > > > :
>>> > > > > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context
>>> > > hierarchy
>>> > > > > > 13:35:10.182 [main] INFO
>>> > > > o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
>>> > > > > -
>>> > > > > > JSR-330 'javax.inject.Inject' annotation found and supported
>>> for
>>> > > > > autowiring
>>> > > > > > 13:35:10.371 [main] INFO
>>> > > > > >
>>> o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker
>>> > -
>>> > > > Bean
>>> > > > > > 'configurationPropertiesRebinderAutoConfiguration' of type
>>> [class
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
>>> > > > > > is not eligible for getting processed by all BeanPostProcessors
>>> > (for
>>> > > > > > example: not eligible for auto-proxying)
>>> > > > > > 13:35:10.435 [background-preinit] INFO
>>> > > > > o.h.validator.internal.util.Version
>>> > > > > > - HV000001: Hibernate Validator 5.3.0.Final
>>> > > > > >
>>> > > > > > [/snip]
>>> > > > > >
>>> > > > > > It looks like the whole startup procedure goes through and I
>>> can
>>> > see
>>> > > > some
>>> > > > > > kind of "summary" that lists all REST endpoints. But it stays
>>> in
>>> > that
>>> > > > > state
>>> > > > > > for only a minute or so and then a cascade of shutdowns are
>>> > triggered
>>> > > > > > again. The only thing that stays alive is Eureka which
>>> complains
>>> > > about
>>> > > > > the
>>> > > > > > inaccessible services.
>>> > > > > >
>>> > > > > > Any idea what else we could do other than just ordering an
>>> > additional
>>> > > > > > machine?
>>> > > > > >
>>> > > > > > Cheers,
>>> > > > > >
>>> > > > > > Aleks
>>> > > > > >
>>> > > > > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <myrle@apache.org
>>> >
>>> > > wrote:
>>> > > > > >
>>> > > > > > > Hey Aleks,
>>> > > > > > >
>>> > > > > > > It looks like that exception is coming from Rhythm. I
>>> suspect
>>> > it's
>>> > > > not
>>> > > > > > the
>>> > > > > > > cause of your difficulties, since you say your other
>>> services are
>>> > > > going
>>> > > > > > > down too. You can run most of the Fineract server without
>>> > Rhythm.
>>> > > > You
>>> > > > > > > could edit the demo-server script to leave it out for the
>>> > purposes
>>> > > of
>>> > > > > > > testing. The consequence will be a lack of interest
>>> calculations
>>> > > for
>>> > > > > > > disbursed loans, but for finding out what the real problem
>>> is,
>>> > it'd
>>> > > > be
>>> > > > > > good
>>> > > > > > > to get this one out of the way.
>>> > > > > > >
>>> > > > > > > Best Regards,
>>> > > > > > > Myrle
>>> > > > > > >
>>> > > > > > > (FYI: I've removed the non-list subscribers from the to.
>>> Most of
>>> > > > them
>>> > > > > > are
>>> > > > > > > already subscribed, and those who aren't, don't want to be.)
>>> > > > > > >
>>> > > > > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
>>> > > > > > > cheetah@monkeysintown.com> wrote:
>>> > > > > > >
>>> > > > > > > > I managed to further improve the startup procedure (aka
>>> less
>>> > > > > exceptions
>>> > > > > > > > than before)... there are still some timeouts, but not as
>>> many
>>> > as
>>> > > > > > before.
>>> > > > > > > >
>>> > > > > > > > The demo server still dies after this exception:
>>> > > > > > > >
>>> > > > > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
>>> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>>> > endpoints
>>> > > > via
>>> > > > > > > > configuration
>>> > > > > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
>>> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>>> > endpoints
>>> > > > via
>>> > > > > > > > configuration
>>> > > > > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
>>> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>>> > endpoints
>>> > > > via
>>> > > > > > > > configuration
>>> > > > > > > > 23:28:45.990 [pool-9-thread-1] ERROR
>>> > > > > > > o.s.s.s.TaskUtils$LoggingErrorHandler
>>> > > > > > > > - Unexpected error occurred in scheduled task.
>>> > > > > > > >
>>> > org.springframework.transaction.CannotCreateTransactionException:
>>> > > > > Could
>>> > > > > > > not
>>> > > > > > > > open JPA EntityManager for transaction; nested exception is
>>> > > > > > > > javax.persistence.PersistenceException:
>>> > > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
>>> > > > > resultset:
>>> > > > > > > > unexpected end of stream, read 0 bytes from 4
>>> > > > > > > > Query is : set autocommit=0
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
>>> > > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
>>> > > > > > > > at
>>> > > > > > >
>>> > > >
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> > > > > > > > at
>>> > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> > > > > > > > at java.lang.Thread.run(Thread.java:748)
>>> > > > > > > > Caused by: javax.persistence.PersistenceException:
>>> > > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
>>> > > > > resultset:
>>> > > > > > > > unexpected end of stream, read 0 bytes from 4
>>> > > > > > > > Query is : set autocommit=0
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > >
>>> > > >
>>> >
>>> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
>>> > > > > > > > ... 20 common frames omitted
>>> > > > > > > > Caused by: java.sql.SQLNonTransientConnectionException:
>>> Could
>>> > not
>>> > > > > read
>>> > > > > > > > resultset: unexpected end of stream, read 0 bytes from 4
>>> > > > > > > > Query is : set autocommit=0
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > >
>>> > > >
>>> >
>>> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
>>> > > > > > > > ... 22 common frames omitted
>>> > > > > > > > Caused by:
>>> org.mariadb.jdbc.internal.util.dao.QueryException:
>>> > > Could
>>> > > > > not
>>> > > > > > > > read resultset: unexpected end of stream, read 0 bytes
>>> from 4
>>> > > > > > > > Query is : set autocommit=0
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
>>> > > > > > > > ... 31 common frames omitted
>>> > > > > > > > Caused by: java.io.EOFException: unexpected end of stream,
>>> > read 0
>>> > > > > bytes
>>> > > > > > > > from 4
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
>>> > > > > > > > at
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
>>> > > > > > > > ... 33 common frames omitted
>>> > > > > > > > 23:29:05.500 [qtp6422064-20] INFO
>>> > > > > o.a.f.c.l.c.ServiceExceptionFilter -
>>> > > > > > > > Responding with a service error ServiceError{code=409,
>>> > > > > > > message='Application
>>> > > > > > > > identity-v1 already exists!'}
>>> > > > > > > > 23:29:21.282 [Thread-15] INFO
>>> > > > > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
>>> > Closing
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
>>> > > > > > > > :
>>> > > > > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
>>> > > > > > > > 23:29:23.717 [Thread-15] INFO
>>> > o.s.c.s.DefaultLifecycleProcessor
>>> > > -
>>> > > > > > > Stopping
>>> > > > > > > > beans in phase 2147483647
>>> > > > > > > > 23:29:24.267 [Thread-15] INFO
>>> > o.s.c.s.DefaultLifecycleProcessor
>>> > > -
>>> > > > > > > Stopping
>>> > > > > > > > beans in phase 0
>>> > > > > > > > 23:29:24.557 [Thread-15] INFO
>>> > > o.s.b.a.e.jmx.EndpointMBeanExporter
>>> > > > -
>>> > > > > > > > Unregistering JMX-exposed beans on shutdown
>>> > > > > > > > 23:29:24.579 [Thread-15] INFO
>>> > > o.s.b.a.e.jmx.EndpointMBeanExporter
>>> > > > -
>>> > > > > > > > Unregistering JMX-exposed beans
>>> > > > > > > >
>>> > > > > > > > The only thing I've left: start the databases (aka
>>> Cassandra)
>>> > on
>>> > > a
>>> > > > > > > separate
>>> > > > > > > > machine (e. g. on Digital Ocean) and see how this is
>>> working
>>> > > then.
>>> > > > > > > >
>>> > > > > > > > ... FYI.
>>> > > > > > > >
>>> > > > > > > > Cheers,
>>> > > > > > > >
>>> > > > > > > > Aleks
>>> > > > > > > >
>>> > > > > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
>>> > > > > > > > cheetah@monkeysintown.com> wrote:
>>> > > > > > > >
>>> > > > > > > > > @Rajan: it's not running, because the demo server
>>> dies... See
>>> > > > logs
>>> > > > > > from
>>> > > > > > > > > previous message...
>>> > > > > > > > >
>>> > > > > > > > > We just have to figure that one out and we should be
>>> good to
>>> > > go.
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
>>> > > > > rajanmaurya154@gmail.com
>>> > > > > > >
>>> > > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > >> Thanks, Alex,
>>> > > > > > > > >>
>>> > > > > > > > >> I can see web app is running but I can't log in with the
>>> > > > > credentials
>>> > > > > > > > that
>>> > > > > > > > >> you shared earlier
>>> > > > > > > > >>
>>> > > > > > > > >> Tenant : playground
>>> > > > > > > > >> Username : operator
>>> > > > > > > > >> Password : init1@l
>>> > > > > > > > >>
>>> > > > > > > > >> Thanks
>>> > > > > > > > >>
>>> > > > > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
>>> > > > > > > > >> cheetah@monkeysintown.com> wrote:
>>> > > > > > > > >>
>>> > > > > > > > >> > Hi all,
>>> > > > > > > > >> >
>>> > > > > > > > >> > ... sweeeet! I got a considerable step further.
>>> > > > > > > > >> >
>>> > > > > > > > >> > @Markus: thanks for the conversation... gave me an
>>> idea
>>> > how
>>> > > to
>>> > > > > fix
>>> > > > > > > > (most
>>> > > > > > > > >> > of) it!
>>> > > > > > > > >> >
>>> > > > > > > > >> > The problem was - as suspected - the memory (or lack
>>> of).
>>> > > The
>>> > > > VM
>>> > > > > > > > >> instance
>>> > > > > > > > >> > at Apache has net 30G available. 16G are needed for
>>> > > Cassandra
>>> > > > > and
>>> > > > > > in
>>> > > > > > > > my
>>> > > > > > > > >> > previous tests I tried to run the demo server with up
>>> to
>>> > > 16G.
>>> > > > > The
>>> > > > > > > > >> problem
>>> > > > > > > > >> > is that the demo server needs considerably more memory
>>> > than
>>> > > > 16G.
>>> > > > > > > > >> According
>>> > > > > > > > >> > to my observation it's around 23-24G. Even if you
>>> limit
>>> > the
>>> > > > demo
>>> > > > > > > > server
>>> > > > > > > > >> > memory allocation with "-Xmx" to 12G (for example) it
>>> > would
>>> > > > just
>>> > > > > > > > >> continue
>>> > > > > > > > >> > to consume more memory.
>>> > > > > > > > >> >
>>> > > > > > > > >> > My solution: add a swap file. Not ideal in terms of
>>> > > > performance,
>>> > > > > > but
>>> > > > > > > > it
>>> > > > > > > > >> > gets the whole thing going... at least mostly (please
>>> read
>>> > > > on).
>>> > > > > > > > >> >
>>> > > > > > > > >> > Current status:
>>> > > > > > > > >> >
>>> > > > > > > > >> > - Cassandra (and all other Docker containers) are
>>> still
>>> > > > > > running;
>>> > > > > > > > >> > Cassandra died in my previous attempts at the
>>> latest
>>> > when
>>> > > > the
>>> > > > > > > > Teller
>>> > > > > > > > >> > application started, but with the swap file it's
>>> still
>>> > > > > running
>>> > > > > > > > >> > - all Spring Boot apps are starting now
>>> > > > > > > > >> > - I get once in a while some spurious timeouts (I
>>> think
>>> > > > when
>>> > > > > > > either
>>> > > > > > > > >> the
>>> > > > > > > > >> > Eureka server is contacted or the config server)
>>> > > > > > > > >> > - there are also sometimes exceptions concerning
>>> MySQL
>>> > > > > > > connections,
>>> > > > > > > > >> but
>>> > > > > > > > >> > doesn't seem to matter overall
>>> > > > > > > > >> > - I could issue an authentication request with
>>> Postman
>>> > to
>>> > > > > > > > >> > http://fineract-vm.apache.org:2021/identity/v1;
>>> just
>>> > to
>>> > > > test
>>> > > > > > if
>>> > > > > > > > >> > anything
>>> > > > > > > > >> > is responding
>>> > > > > > > > >> > - PROBLEM: but the demo server (the Spring Boot
>>> apps)
>>> > > > > > eventually
>>> > > > > > > > dies
>>> > > > > > > > >> > after a couple of minutes; I don't think that the
>>> > memory
>>> > > is
>>> > > > > the
>>> > > > > > > > >> problem
>>> > > > > > > > >> > at
>>> > > > > > > > >> > this point; I think it's more of a timeout problem
>>> > > > concerning
>>> > > > > > the
>>> > > > > > > > >> > communication with Eureka and/or the config server
>>> > (maybe
>>> > > > > > because
>>> > > > > > > > the
>>> > > > > > > > >> > system is a bit slow)
>>> > > > > > > > >> >
>>> > > > > > > > >> > ... and here the memory footprint ("free -h") when
>>> > > everything
>>> > > > is
>>> > > > > > > > >> running,
>>> > > > > > > > >> > just FYI:
>>> > > > > > > > >> >
>>> > > > > > > > >> > total used free
>>> shared
>>> > > > > > buff/cache
>>> > > > > > > > >> > available
>>> > > > > > > > >> > Mem: 31G 31G 226M
>>> 8.4M
>>> > > > > > 169M
>>> > > > > > > > >> > 47M
>>> > > > > > > > >> > Swap: 15G 6.8G 9.2G
>>> > > > > > > > >> >
>>> > > > > > > > >> > ... after the Payroll application is started
>>> (approx.) I
>>> > see
>>> > > > > these
>>> > > > > > > > >> > exceptions appearing in the logs (a whole series of
>>> them):
>>> > > > > > > > >> >
>>> > > > > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
>>> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>>> > > > endpoints
>>> > > > > > via
>>> > > > > > > > >> > configuration
>>> > > > > > > > >> > Resolving artifact
>>> > > > > > > > >> >
>>> > > > >
>>> org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
>>> > > > > > > > >> > Resolving metadata
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
>>> > > > > > > > >> > from /root/.m2/repository (enhanced)
>>> > > > > > > > >> > Resolved metadata
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
>>> > > > > > > > >> > from /root/.m2/repository (enhanced)
>>> > > > > > > > >> > Resolved artifact
>>> > > > > > > > >> >
>>> > > > >
>>> org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
>>> > > > > > > > from
>>> > > > > > > > >> > /root/.m2/repository (enhanced)
>>> > > > > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
>>> > > > > > > > >> c.n.discovery.TimedSupervisorTask -
>>> > > > > > > > >> > task supervisor timed out
>>> > > > > > > > >> > java.util.concurrent.TimeoutException: null
>>> > > > > > > > >> > at
>>> > java.util.concurrent.FutureTask.get(FutureTask.java:205)
>>> > > > > > > > >> > at
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
>>> > > > > > > > >> > at
>>> > > > > > > > >>
>>> > > > > > >
>>> > > >
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> > > > > > > > >> > at
>>> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> > > > > > > > >> > at
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> > > > > > > > >> > at
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> > > > > > > > >> > at
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> > > > > > > > >> > at
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> > > > > > > > >> > at java.lang.Thread.run(Thread.java:748)
>>> > > > > > > > >> >
>>> > > > > > > > >> > ... but it still seems to have no problem starting the
>>> > next
>>> > > > app
>>> > > > > > > > (Group)
>>> > > > > > > > >> and
>>> > > > > > > > >> > things look good for a while...
>>> > > > > > > > >> >
>>> > > > > > > > >> > ... until suddenly it seems that app instances are
>>> > beginning
>>> > > > to
>>> > > > > > > > >> shutdown...
>>> > > > > > > > >> > and I'm not sure why this happens:
>>> > > > > > > > >> >
>>> > > > > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
>>> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>>> > > > endpoints
>>> > > > > > via
>>> > > > > > > > >> > configuration
>>> > > > > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
>>> > > > > > > > >> c.datastax.driver.core.ClockFactory -
>>> > > > > > > > >> > Using native clock to generate timestamps.
>>> > > > > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
>>> > > > > > > > >> c.datastax.driver.core.CodecRegistry
>>> > > > > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
>>> > > > > > > > >> java.time.LocalDateTime]
>>> > > > > > > > >> > because it collides with previously registered codec
>>> > > > > > > > LocalDateTimeCodec
>>> > > > > > > > >> > [timestamp <-> java.time.LocalDateTime]
>>> > > > > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
>>> > > > > > > > c.d.d.c.p.DCAwareRoundRobinPolicy
>>> > > > > > > > >> -
>>> > > > > > > > >> > Using data-center name 'datacenter1' for
>>> > > > DCAwareRoundRobinPolicy
>>> > > > > > (if
>>> > > > > > > > >> this
>>> > > > > > > > >> > is incorrect, please provide the correct datacenter
>>> name
>>> > > with
>>> > > > > > > > >> > DCAwareRoundRobinPolicy constructor)
>>> > > > > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
>>> > > > > > > > com.datastax.driver.core.Cluster -
>>> > > > > > > > >> > New Cassandra host /127.0.0.1:9042 added
>>> > > > > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
>>> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>>> > > > endpoints
>>> > > > > > via
>>> > > > > > > > >> > configuration
>>> > > > > > > > >> > 18:27:51.110 [Thread-15] INFO
>>> > > > > > > > >> >
>>> o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
>>> > > > > Closing
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
>>> > > > > > > > >> > :
>>> > > > > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
>>> > > > > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
>>> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>>> > > > endpoints
>>> > > > > > via
>>> > > > > > > > >> > configuration
>>> > > > > > > > >> > 18:27:53.373 [Thread-15] INFO
>>> > > > > o.s.c.s.DefaultLifecycleProcessor -
>>> > > > > > > > >> Stopping
>>> > > > > > > > >> > beans in phase 2147483647
>>> > > > > > > > >> > 18:27:54.331 [Thread-15] INFO
>>> > > > > o.s.c.s.DefaultLifecycleProcessor -
>>> > > > > > > > >> Stopping
>>> > > > > > > > >> > beans in phase 0
>>> > > > > > > > >> > 18:27:54.736 [Thread-15] INFO
>>> > > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
>>> > > > > > > > >> > Unregistering JMX-exposed beans on shutdown
>>> > > > > > > > >> > 18:27:54.740 [Thread-15] INFO
>>> > > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
>>> > > > > > > > >> > Unregistering JMX-exposed beans
>>> > > > > > > > >> > 18:27:54.798 [Thread-15] INFO
>>> > > > > o.s.j.e.a.AnnotationMBeanExporter -
>>> > > > > > > > >> > Unregistering JMX-exposed beans on shutdown
>>> > > > > > > > >> > 18:27:54.800 [Thread-15] INFO
>>> > > > > o.s.j.e.a.AnnotationMBeanExporter -
>>> > > > > > > > >> > Unregistering JMX-exposed beans
>>> > > > > > > > >> > 18:27:54.937 [Thread-15] INFO
>>> > > > > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean -
>>> Closing
>>> > JPA
>>> > > > > > > > >> > EntityManagerFactory for persistence unit 'metaPU'
>>> > > > > > > > >> > 18:27:58.376 [async-processor-1] INFO
>>> > > > > > > > >> o.f.c.internal.util.VersionPrinter -
>>> > > > > > > > >> > Flyway 3.2.1 by Boxfuse
>>> > > > > > > > >> > 18:27:58.677 [async-processor-1] INFO
>>> > > > > > > > >> o.f.c.i.dbsupport.DbSupportFactory -
>>> > > > > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL
>>> 10.3)
>>> > > > > > > > >> > 18:28:00.620 [async-processor-1] INFO
>>> > > > > > > > >> o.f.core.internal.command.DbValidate
>>> > > > > > > > >> > - Validated 2 migrations (execution time 00:01.169s)
>>> > > > > > > > >> > 18:28:00.901 [Thread-15] INFO
>>> > > > > o.e.jetty.server.AbstractConnector
>>> > > > > > -
>>> > > > > > > > >> Stopped
>>> > > > > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{
>>> 0.0.0.0:2020
>>> > }
>>> > > > > > > > >> > 18:28:01.268 [Thread-15] INFO
>>> > > > > > o.e.j.server.handler.ContextHandler -
>>> > > > > > > > >> > Stopped
>>> o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
>>> > > > > > > > >> >
>>> > > > > > > > >> > ... the list is actually a lot longer... in the rest
>>> of
>>> > the
>>> > > > log
>>> > > > > > you
>>> > > > > > > > can
>>> > > > > > > > >> see
>>> > > > > > > > >> > that one by one all the apps are shutting down.
>>> > > > > > > > >> >
>>> > > > > > > > >> > The last (error) entries I see are these:
>>> > > > > > > > >> >
>>> > > > > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3]
>>> ERROR
>>> > > > > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not
>>> > refresh
>>> > > > JMS
>>> > > > > > > > >> Connection
>>> > > > > > > > >> > for destination 'cheques-v1' - retrying using
>>> > > > > > > > >> FixedBackOff{interval=5000,
>>> > > > > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause:
>>> Error
>>> > > while
>>> > > > > > > > >> attempting to
>>> > > > > > > > >> > retrieve a connection from the pool; nested exception
>>> is
>>> > > > > > > > >> > javax.jms.JMSException: Could not connect to broker
>>> URL:
>>> > > > > > > > >> > tcp://localhost:61616. Reason:
>>> java.net.ConnectException:
>>> > > > > > Connection
>>> > > > > > > > >> > refused (Connection refused)
>>> > > > > > > > >> >
>>> > > > > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in
>>> the
>>> > > > > process
>>> > > > > > of
>>> > > > > > > > the
>>> > > > > > > > >> > previous errors.
>>> > > > > > > > >> >
>>> > > > > > > > >> > So far from here... any input on the above highly
>>> > > appreciated.
>>> > > > > > > > Overall I
>>> > > > > > > > >> > think we are close.
>>> > > > > > > > >> >
>>> > > > > > > > >> > Cheers,
>>> > > > > > > > >> >
>>> > > > > > > > >> > Aleks
>>> > > > > > > > >> >
>>> > > > > > > > >> >
>>> > > > > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <
>>> > > mage@apache.org
>>> > > > >
>>> > > > > > > wrote:
>>> > > > > > > > >> >
>>> > > > > > > > >> > > One thing we maybe can do, given we are part of the
>>> > Apache
>>> > > > > > Family,
>>> > > > > > > > is
>>> > > > > > > > >> to
>>> > > > > > > > >> > > ask the Cassandra community for some suggestions.
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > Cheers
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > Markus
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <
>>> > > > mage@apache.org
>>> > > > > >
>>> > > > > > > > wrote:
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > > Hey all,
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > we are not running Cassandra in a container, we
>>> have
>>> > > > > dedicated
>>> > > > > > > VMs
>>> > > > > > > > >> to
>>> > > > > > > > >> > run
>>> > > > > > > > >> > > > our cluster.
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > Given this we are not the best persons to help
>>> here,
>>> > > > sorry.
>>> > > > > (;
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > Cheers
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > Markus
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
>>> > > > > edcable@mifos.org>
>>> > > > > > > > >> wrote:
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > >> Adding Patric directly to this thread too so he
>>> can
>>> > > give
>>> > > > > his
>>> > > > > > > > input
>>> > > > > > > > >> > > >> regarding to the challenges with Cassandra.
>>> > > > > > > > >> > > >>
>>> > > > > > > > >> > > >> Ed
>>> > > > > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar
>>> > Vidakovic <
>>> > > > > > > > >> > > >> cheetah@monkeysintown.com> wrote:
>>> > > > > > > > >> > > >>
>>> > > > > > > > >> > > >>> Hi Victor,
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> .... this is the relevant part of the
>>> > > docker-compose.yml
>>> > > > > > file:
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> cassandra0:
>>> > > > > > > > >> > > >>> image: cassandra:3.11.1
>>> > > > > > > > >> > > >>> container_name: cassandra0
>>> > > > > > > > >> > > >>> ports:
>>> > > > > > > > >> > > >>> - 9042:9042
>>> > > > > > > > >> > > >>> - 9160:9160
>>> > > > > > > > >> > > >>> - 7199:7199
>>> > > > > > > > >> > > >>> - 8778:8778
>>> > > > > > > > >> > > >>> volumes:
>>> > > > > > > > >> > > >>> - ./cassandra:/etc/cassandra
>>> > > > > > > > >> > > >>> environment:
>>> > > > > > > > >> > > >>> - CASSANDRA_START_RPC=true
>>> > > > > > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0
>>> > > > > > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster
>>> > > > > > > > >> > > >>> ulimits:
>>> > > > > > > > >> > > >>> memlock: -1
>>> > > > > > > > >> > > >>> nproc: 32768
>>> > > > > > > > >> > > >>> nofile: 100000
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> ... and just to be complete... here's the Docker
>>> > > service
>>> > > > > > > > >> > configuration
>>> > > > > > > > >> > > >>> ("/lib/systemd/system/docker.service"):
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> [Unit]
>>> > > > > > > > >> > > >>> Description=Docker Application Container Engine
>>> > > > > > > > >> > > >>> Documentation=https://docs.docker.com
>>> > > > > > > > >> > > >>> After=network-online.target docker.socket
>>> > > > > firewalld.service
>>> > > > > > > > >> > > >>> Wants=network-online.target
>>> > > > > > > > >> > > >>> Requires=docker.socket
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> [Service]
>>> > > > > > > > >> > > >>> Type=notify
>>> > > > > > > > >> > > >>> # the default is not to use systemd for cgroups
>>> > > because
>>> > > > > the
>>> > > > > > > > >> delegate
>>> > > > > > > > >> > > >>> issues
>>> > > > > > > > >> > > >>> still
>>> > > > > > > > >> > > >>> # exists and systemd currently does not support
>>> the
>>> > > > cgroup
>>> > > > > > > > feature
>>> > > > > > > > >> > set
>>> > > > > > > > >> > > >>> required
>>> > > > > > > > >> > > >>> # for containers run by docker
>>> > > > > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
>>> > > > > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
>>> > > > > > > > >> > > >>> LimitNOFILE=1048576
>>> > > > > > > > >> > > >>> # Having non-zero Limit*s causes performance
>>> > problems
>>> > > > due
>>> > > > > to
>>> > > > > > > > >> > accounting
>>> > > > > > > > >> > > >>> overhead
>>> > > > > > > > >> > > >>> # in the kernel. We recommend using cgroups to
>>> do
>>> > > > > > > > container-local
>>> > > > > > > > >> > > >>> accounting.
>>> > > > > > > > >> > > >>> LimitNPROC=infinity
>>> > > > > > > > >> > > >>> LimitCORE=infinity
>>> > > > > > > > >> > > >>> LimitMEMLOCK=infinity
>>> > > > > > > > >> > > >>> # Uncomment TasksMax if your systemd version
>>> > supports
>>> > > > it.
>>> > > > > > > > >> > > >>> # Only systemd 226 and above support this
>>> version.
>>> > > > > > > > >> > > >>> TasksMax=infinity
>>> > > > > > > > >> > > >>> TimeoutStartSec=0
>>> > > > > > > > >> > > >>> # set delegate yes so that systemd does not
>>> reset
>>> > the
>>> > > > > > cgroups
>>> > > > > > > of
>>> > > > > > > > >> > docker
>>> > > > > > > > >> > > >>> containers
>>> > > > > > > > >> > > >>> Delegate=yes
>>> > > > > > > > >> > > >>> # kill only the docker process, not all
>>> processes in
>>> > > the
>>> > > > > > > cgroup
>>> > > > > > > > >> > > >>> KillMode=process
>>> > > > > > > > >> > > >>> # restart the docker process if it exits
>>> prematurely
>>> > > > > > > > >> > > >>> Restart=on-failure
>>> > > > > > > > >> > > >>> StartLimitBurst=3
>>> > > > > > > > >> > > >>> StartLimitInterval=60s
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> [Install]
>>> > > > > > > > >> > > >>> WantedBy=multi-user.target
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> ... the one tweak I did there is to set
>>> LimitMEMLOCK
>>> > > to
>>> > > > > > > > >> infinity...
>>> > > > > > > > >> > do
>>> > > > > > > > >> > > >>> you
>>> > > > > > > > >> > > >>> think the problems are file handle related?
>>> Should I
>>> > > set
>>> > > > > > > > >> LimitNOFILE
>>> > > > > > > > >> > > also
>>> > > > > > > > >> > > >>> to infinity?
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> Appreciate the help.
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> Cheers,
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> Aleks
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
>>> > > > > > > > >> > > >>> victor.romero@fintecheando.mx>
>>> > > > > > > > >> > > >>> wrote:
>>> > > > > > > > >> > > >>>
>>> > > > > > > > >> > > >>> > Hi Aleks,
>>> > > > > > > > >> > > >>> >
>>> > > > > > > > >> > > >>> > Can you share the ulimits flags that the
>>> > cassandra's
>>> > > > > > > container
>>> > > > > > > > >> is
>>> > > > > > > > >> > > using
>>> > > > > > > > >> > > >>> > while it is running?
>>> > > > > > > > >> > > >>> >
>>> > > > > > > > >> > > >>> > The values are being set in the composer
>>> file? Or
>>> > in
>>> > > > the
>>> > > > > > > > >> upstart or
>>> > > > > > > > >> > > >>> > systemd docker's deamon config files?
>>> > > > > > > > >> > > >>> >
>>> > > > > > > > >> > > >>> >
>>> > > > > > > > >> > > >>> >
>>> > > > > > > > >> > > >>> > Enviado desde TypeApp
>>> > > > > > > > >> > > >>> >
>>> > > > > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM,
>>> Aleksandar
>>> > > > > > > Vidakovic <
>>> > > > > > > > >> > > >>> > cheetah@monkeysintown.com> escribió:
>>> > > > > > > > >> > > >>> > >... and I should have attached the Cassandra
>>> log
>>> > > > > dump...
>>> > > > > > so
>>> > > > > > > > >> here
>>> > > > > > > > >> > it
>>> > > > > > > > >> > > >>> > >is...
>>> > > > > > > > >> > > >>> > >
>>> > > > > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar
>>> > > Vidakovic
>>> > > > <
>>> > > > > > > > >> > > >>> > >cheetah@monkeysintown.com> wrote:
>>> > > > > > > > >> > > >>> > >
>>> > > > > > > > >> > > >>> > >> Hi all,
>>> > > > > > > > >> > > >>> > >>
>>> > > > > > > > >> > > >>> > >> ... as you might have noticed I did a
>>> couple of
>>> > > > > > restarts
>>> > > > > > > > >> > today...
>>> > > > > > > > >> > > >>> the
>>> > > > > > > > >> > > >>> > >> problem I am facing now: I can't get
>>> Cassandra
>>> > to
>>> > > > run
>>> > > > > > in
>>> > > > > > > a
>>> > > > > > > > >> > stable
>>> > > > > > > > >> > > >>> > >way.
>>> > > > > > > > >> > > >>> > >>
>>> > > > > > > > >> > > >>> > >> Things look quite OK for a while when
>>> running
>>> > the
>>> > > > > demo
>>> > > > > > > > >> server,
>>> > > > > > > > >> > but
>>> > > > > > > > >> > > >>> > >then
>>> > > > > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it
>>> starts
>>> > > > > > misbehaving
>>> > > > > > > > with
>>> > > > > > > > >> > the
>>> > > > > > > > >> > > >>> > >deposit
>>> > > > > > > > >> > > >>> > >> microservice startup, sometimes with
>>> portfolio
>>> > or
>>> > > > > > > teller).
>>> > > > > > > > I
>>> > > > > > > > >> > tried
>>> > > > > > > > >> > > >>> to
>>> > > > > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and
>>> set
>>> > some
>>> > > > > > Docker
>>> > > > > > > > >> limits
>>> > > > > > > > >> > to
>>> > > > > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
>>> > > > > > > > >> > > >>> > >>
>>> > > > > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe
>>> > > someone
>>> > > > > can
>>> > > > > > > > help
>>> > > > > > > > >> out
>>> > > > > > > > >> > > >>> > >here? Is
>>> > > > > > > > >> > > >>> > >> it even possible to run Fineract CN on
>>> 32GB of
>>> > > > > memory?
>>> > > > > > > > >> > > >>> > >>
>>> > > > > > > > >> > > >>> > >> Other than that the setup would be ready to
>>> > go...
>>> > > > we
>>> > > > > > just
>>> > > > > > > > >> need
>>> > > > > > > > >> > to
>>> > > > > > > > >> > > >>> get
>>> > > > > > > > >> > > >>> > >the
>>> > > > > > > > >> > > >>> > >> database running more reliably.
>>> > > > > > > > >> > > >>> > >>
>>> > > > > > > > >> > > >>> > >> Cheers,
>>> > > > > > > > >> > > >>> > >>
>>> > > > > > > > >> > > >>> > >> Aleks
>>> > > > > > > > >> > > >>> > >>
>>> > > > > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar
>>> > > > Vidakovic
>>> > > > > <
>>> > > > > > > > >> > > >>> > >> cheetah@monkeysintown.com> wrote:
>>> > > > > > > > >> > > >>> > >>
>>> > > > > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed
>>> to
>>> > add
>>> > > > > some
>>> > > > > > > > >> > additional
>>> > > > > > > > >> > > >>> > >reverse
>>> > > > > > > > >> > > >>> > >>> proxy configuration for the web UI to
>>> work)...
>>> > > and
>>> > > > > > there
>>> > > > > > > > >> were
>>> > > > > > > > >> > > more
>>> > > > > > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
>>> > > > > > > > >> > > >>> > >>>
>>> > > > > > > > >> > > >>> > >>> Just FYI
>>> > > > > > > > >> > > >>> > >>>
>>> > > > > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
>>> > > > > Vidakovic <
>>> > > > > > > > >> > > >>> > >>> cheetah@monkeysintown.com> wrote:
>>> > > > > > > > >> > > >>> > >>>
>>> > > > > > > > >> > > >>> > >>>> No prob.... Let me know how it goes...
>>> Can
>>> > get
>>> > > > back
>>> > > > > > to
>>> > > > > > > > >> testing
>>> > > > > > > > >> > > >>> only
>>> > > > > > > > >> > > >>> > >>>> later tonight.
>>> > > > > > > > >> > > >>> > >>>>
>>> > > > > > > > >> > > >>> > >>>> Cheers
>>> > > > > > > > >> > > >>> > >>>>
>>> > > > > > > > >> > > >>> > >>>>
>>> > > > > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan
>>> Maurya
>>> > > > > > > > >> > > >>> > ><ra...@gmail.com>
>>> > > > > > > > >> > > >>> > >>>> wrote:
>>> > > > > > > > >> > > >>> > >>>>
>>> > > > > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test
>>> after
>>> > 30
>>> > > > min.
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > > > > > > >> > > >>> > >>>>> Big thanks for this 🙂
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > > > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM
>>> Aleksandar
>>> > > > > > Vidakovic <
>>> > > > > > > > >> > > >>> > >>>>> cheetah@monkeysintown.com> wrote:
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > > > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you
>>> posted.
>>> > > > > > > > >> > > >>> > >>>>> >
>>> > > > > > > > >> > > >>> > >>>>> > ... and as I said: the services are
>>> still
>>> > > > > > > starting...
>>> > > > > > > > >> and
>>> > > > > > > > >> > the
>>> > > > > > > > >> > > >>> > >best
>>> > > > > > > > >> > > >>> > >>>>> estimate
>>> > > > > > > > >> > > >>> > >>>>> > I have right now (as already
>>> mentioned):
>>> > > 30min
>>> > > > > > > > >> > > >>> > >>>>> >
>>> > > > > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan
>>> > > Maurya <
>>> > > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
>>> > > > > > > > >> > > >>> > >>>>> > wrote:
>>> > > > > > > > >> > > >>> > >>>>> >
>>> > > > > > > > >> > > >>> > >>>>> > > [image: image.png]
>>> > > > > > > > >> > > >>> > >>>>> > >
>>> > > > > > > > >> > > >>> > >>>>> > >
>>> > > > > > > > >> > > >>> > >>>>> > >
>>> > > > > > > > >> > > >>> > >>>>> > >
>>> > > > > > > > >> > > >>> > >>>>> > >
>>> > > > > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM
>>> Rajan
>>> > > > Maurya <
>>> > > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
>>> > > > > > > > >> > > >>> > >>>>> > > wrote:
>>> > > > > > > > >> > > >>> > >>>>> > >
>>> > > > > > > > >> > > >>> > >>>>> > >> [image: image.png]
>>> > > > > > > > >> > > >>> > >>>>> > >> I am getting this.
>>> > > > > > > > >> > > >>> > >>>>> > >>
>>> > > > > > > > >> > > >>> > >>>>> > >>
>>> > > > > > > > >> > > >>> > >>>>> > >>
>>> > > > > > > > >> > > >>> > >>>>> > >>
>>> > > > > > > > >> > > >>> > >>>>> > >>
>>> > > > > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM
>>> > > Aleksandar
>>> > > > > > > > Vidakovic
>>> > > > > > > > >> <
>>> > > > > > > > >> > > >>> > >>>>> > >> cheetah@monkeysintown.com> wrote:
>>> > > > > > > > >> > > >>> > >>>>> > >>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Hi all,
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost)
>>> > > ready...
>>> > > > > it
>>> > > > > > > took
>>> > > > > > > > >> me a
>>> > > > > > > > >> > > >>> > >moment
>>> > > > > > > > >> > > >>> > >>>>> and a
>>> > > > > > > > >> > > >>> > >>>>> > >>> couple
>>> > > > > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some
>>> boot
>>> > > > > > failures...
>>> > > > > > > > the
>>> > > > > > > > >> > > >>> services
>>> > > > > > > > >> > > >>> > >are
>>> > > > > > > > >> > > >>> > >>>>> quite
>>> > > > > > > > >> > > >>> > >>>>> > >>> resource hungry and default
>>> settings
>>> > > won't
>>> > > > > do
>>> > > > > > > it.
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> The services are still starting
>>> and
>>> > this
>>> > > > > will
>>> > > > > > > > take a
>>> > > > > > > > >> > > while
>>> > > > > > > > >> > > >>> > >(my
>>> > > > > > > > >> > > >>> > >>>>> best
>>> > > > > > > > >> > > >>> > >>>>> > guess
>>> > > > > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
>>> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org/login
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Credentials:
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Tenant : playground
>>> > > > > > > > >> > > >>> > >>>>> > >>> Username : operator
>>> > > > > > > > >> > > >>> > >>>>> > >>> Password : init1@l
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to
>>> > > compile
>>> > > > > this
>>> > > > > > > app
>>> > > > > > > > >> is
>>> > > > > > > > >> > > with
>>> > > > > > > > >> > > >>> > >NodeJS
>>> > > > > > > > >> > > >>> > >>>>> > 8.11.1
>>> > > > > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't
>>> > > work).
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Identity Service:
>>> > > > > > > > >> > > >>> > >
>>> http://fineract-vm.apache.org:2021/identity/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Office Service:
>>> > > > > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Customer Service:
>>> > > > > > > > >> > > >>> > >
>>> http://fineract-vm.apache.org:2024/customer/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Accounting Service:
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > > http://fineract-vm.apache.org:2025/accounting/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Portfolio Service:
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > http://fineract-vm.apache.org:2026/portfolio/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Deposit Service:
>>> > > > > > > > >> > > >>> > >
>>> http://fineract-vm.apache.org:2027/deposit/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Teller Service:
>>> > > > > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Reporting Service:
>>> > > > > > > > >> > > >>> > >>>>>
>>> > > http://fineract-vm.apache.org:2029/reporting/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Cheque Service:
>>> > > > > > > > >> > > >>> > >
>>> http://fineract-vm.apache.org:2030/cheques/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>> Payroll Service:
>>> > > > > > > > >> > > >>> > >
>>> http://fineract-vm.apache.org:2031/payroll/v1
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Note: restarting the services
>>> takes
>>> > > quite
>>> > > > a
>>> > > > > > > > >> while... if
>>> > > > > > > > >> > > you
>>> > > > > > > > >> > > >>> > >>>>> encounter
>>> > > > > > > > >> > > >>> > >>>>> > >>> connection problems retry a
>>> couple of
>>> > > > > minutes
>>> > > > > > > > later
>>> > > > > > > > >> to
>>> > > > > > > > >> > > >>> > >ensure I
>>> > > > > > > > >> > > >>> > >>>>> am not
>>> > > > > > > > >> > > >>> > >>>>> > >>> currently working on something.
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the
>>> > backend
>>> > > > > > services
>>> > > > > > > > >> once
>>> > > > > > > > >> > per
>>> > > > > > > > >> > > >>> > >day or
>>> > > > > > > > >> > > >>> > >>>>> so to
>>> > > > > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to
>>> > > figure
>>> > > > > out
>>> > > > > > > how
>>> > > > > > > > we
>>> > > > > > > > >> > want
>>> > > > > > > > >> > > >>> to
>>> > > > > > > > >> > > >>> > >>>>> handle
>>> > > > > > > > >> > > >>> > >>>>> > this
>>> > > > > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where
>>> to
>>> > put
>>> > > > > this
>>> > > > > > > demo
>>> > > > > > > > >> > server
>>> > > > > > > > >> > > >>> > >>>>> configuration
>>> > > > > > > > >> > > >>> > >>>>> > >>> then let me know here.
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any
>>> > > problems
>>> > > > (I
>>> > > > > > > have
>>> > > > > > > > >> not
>>> > > > > > > > >> > > >>> > >extensively
>>> > > > > > > > >> > > >>> > >>>>> > tested
>>> > > > > > > > >> > > >>> > >>>>> > >>> it
>>> > > > > > > > >> > > >>> > >>>>> > >>> yet).
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Cheers,
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> Aleks
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM
>>> > > > Aleksandar
>>> > > > > > > > >> Vidakovic <
>>> > > > > > > > >> > > >>> > >>>>> > >>> cheetah@monkeysintown.com> wrote:
>>> > > > > > > > >> > > >>> > >>>>> > >>>
>>> > > > > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem
>>> to be
>>> > > > > > starting
>>> > > > > > > > fine
>>> > > > > > > > >> > > now...
>>> > > > > > > > >> > > >>> > >just
>>> > > > > > > > >> > > >>> > >>>>> have
>>> > > > > > > > >> > >
>>>
>>
>>
>> --
>> *Ed Cable*
>> President/CEO, Mifos Initiative
>> edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
>>
>> *Collectively Creating a World of 3 Billion Maries | *http://mifos.org
>> <http://facebook.com/mifos> <http://www.twitter.com/mifos>
>>
>>
>
> --
> *Ed Cable*
> President/CEO, Mifos Initiative
> edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
>
> *Collectively Creating a World of 3 Billion Maries | *http://mifos.org
> <http://facebook.com/mifos> <http://www.twitter.com/mifos>
>
>
--
*Ed Cable*
President/CEO, Mifos Initiative
edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
*Collectively Creating a World of 3 Billion Maries | *http://mifos.org
<http://facebook.com/mifos> <http://www.twitter.com/mifos>
Re: Status of Demo Server on Apache VM Infrastructure
Posted by Ed Cable <ed...@mifos.org>.
Hi gentlemen,
I just wanted to follow up on these two items once again.
Thanks,
Ed
On Tue, Dec 4, 2018 at 2:53 PM Ed Cable <ed...@mifos.org> wrote:
> @Avik Ganguly <av...@gmail.com> were you able to take up any of
> the suggested tasks by Myrle?
>
> @Aleksandar Vidakovic <ch...@monkeysintown.com> Can you provide an
> update on the status of deploying Fineract CN on the Apache VMs?
>
> Thanks,
>
> Ed
>
> On Thu, Oct 25, 2018 at 3:39 AM Myrle Krantz <my...@apache.org> wrote:
>
>> Hi Avik,
>>
>> You've asked several questions here, which is why it took me a while to
>> get
>> to answering.
>>
>> Before I go on, allow me to clarify my use of certain terms in the context
>> of Fineract CN:
>> * unit test -- a test for just one or two classes. Doesn't require
>> spinning up the service. Doesn't require resources like a DB or user
>> interaction. Is fast enough to be made part of the build. These tests are
>> included in the same module and package as the class which they are
>> intended to test. (This is a definition which is generally agreed upon in
>> the industry.)
>> * component test -- a test for a service. Requires spinning up just that
>> service. Requires the use of an embedded DB. Individual tests might be
>> fast, but the testsuite is rather expensive to run because startup times
>> are impacted by running two DBs and spinning up the service. These tests
>> are included in the same repository as the service they are intended to
>> test. External services are mocked. (The industry has no widely agreed
>> upon definition for component tests; this is my definition.)
>> * integration test -- a test for multiple services. These tests have
>> their
>> own repositories.
>>
>> demo-server serves three purposes:
>> * It illustrates the steps need to start and provision the services.
>> * It serves as an integration test for one possible the provisioning
>> workflow.
>> * It makes it possible to start up the services locally so that a UI
>> developer can program against the latest changes.
>>
>> demo-server is not intended to be a used in production for multiple
>> reasons:
>> * It's slow. It starts everything, even stuff you may not need, and it
>> provisions everything. It then waits for everything to register itself
>> with Eureka. It lowers the times for the eureka server to speed up
>> startup, but this results in to many cycles being used for Eureka updates.
>> * It's not secure. For example: identity.token.refresh.secureCookie is
>> set
>> to false to make http transport possible.
>>
>> If you're creating a new service, I don't expect that you'll need to run
>> demo-server very often. Your emphasis should be on your component tests.
>> These can run in just a minute or two, and you don't have to worry (much)
>> about JWT tokens, headers, and the like inside of these tests.
>>
>> If demo-server doesn't serve your purposes I see a few possibilities:
>> * create a private fork of the demo-server, then create the setup you
>> prefer.
>> * submit a PR with another "test" in demo-server which starts the services
>> you need. I'd be happy to help you with this.
>> * submit a PR which makes the starting of various services in the
>> demo-service optional. I'd be happy to help you with this as well.
>>
>> One idea for speeding up local deployments (which is intended only to
>> enable developers and not to be a production deployment method) is to
>> start
>> all the services in one process. This would reduce service startup times
>> by folding them all into one. It would also replace inter-process
>> communication during provisioning with in-process communication. I expect
>> that we could get startup under 5 minutes. It would also make it possible
>> to do component-testing on services which are dependent on other services
>> without extensive, error-prone mocking. Writing tests for portfolio
>> probably cost me more time than writing the code, and debugging mocks
>> isn't
>> fun. But those benefits would only apply for development environments.
>> That's what FINCN-25 is about.
>>
>> Rhythm isn't resource-hungry. Not sure where that came from...
>>
>> I don't believe Cassandra is the problem here, but I haven't examined that
>> closely. Please prove me wrong. I'd love to see a fix like that, if it
>> really is that easy. : o)
>>
>> Best Regards,
>> Myrle
>>
>>
>>
>> On Wed, Oct 10, 2018 at 6:24 PM Avik Ganguly <av...@gmail.com>
>> wrote:
>>
>> > Hi guys,
>> >
>> > I am assuming that the suggested form of development for Fineract CN is
>> to
>> > write unit tests to test the code which you just wrote locally and then
>> > push the code to a remote cluster or a massive VM and wait 40+ minutes
>> to
>> > run the API / integration tests there. A lot of the confusion regarding
>> the
>> > stacktraces in identity, rhythm, timeouts and auto-shutdown might or
>> might
>> > not have anything to do with the resource crunch in my local. (i7, 8GB,
>> > SSD)
>> >
>> > Maybe a FAQ section in "How to build Fineract CN" to help devs with
>> local
>> > setup would go a long way to bring in contributions to the core modules.
>> >
>> > 1. Change demo server orchestration logic from running Identity,
>> > Organization, Customer, Accounting, Portfolio, Deposit, Teller,
>> Reporting,
>> > Cheque, Payroll, Group and Notification services to running only
>> Identity,
>> > Organization, Customer and working set of your choice (Portfolio and
>> > Accounting OR Deposit and Accounting OR Group and Portfolio)
>> >
>> > 2. Leave out Rhythm from orchestration logic for now. A JIRA ticket
>> > tracking ability of Rhythm to run with low resource footprint will
>> probably
>> > help.
>> >
>> >
>> > @Myrle, can you elaborate on FINCN-25? Is the motivation of this
>> ticket to
>> > reduce resource footprint only or also to reduce the startup time from
>> 40
>> > minutes to a reasonable couple of minutes? Do you have any debugging
>> tips
>> > with respect to the slow startup caused by services is due to
>> registering
>> > with eureka or provisioning or sequential startup?
>> >
>> > Maybe we should also follow up in Cassandra mailing list regarding
>> > configuring Cassandra to run within 8GB of RAM and include the same
>> config
>> > as part of the FAQ.
>> >
>> >
>> > Regards,
>> > Avik.
>> > ᐧ
>> >
>> > On Wed, Sep 19, 2018 at 2:23 AM Courage Angeh <co...@gmail.com>
>> > wrote:
>> >
>> > > Thanks for the update. Ed.
>> > >
>> > > If you need an extra hand please let me know Aleks.
>> > >
>> > > On Tue, Sep 18, 2018, 4:27 PM Ed Cable <ed...@mifos.org> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > Just to bring this thread back up to top of everyone's inbox. Aleks
>> > > > estimates he has about a day left to get the demo server up and
>> running
>> > > is
>> > > > hoping to tackle it by tomorrow. Here are the two approaches he
>> > suggested
>> > > > now that we have the second VM allocated:
>> > > >
>> > > > - continue with the JUnit based startup sequence of the demo module;
>> > the
>> > > > one thing that I would need to change is to split up the startup
>> > sequence
>> > > > and recompile
>> > > > - create a set of Docker Compose files to do the startup on the 2 VM
>> > > > instances (maybe I could borrow some stuff from Courage)
>> > > >
>> > > > I would prefer the latter, because you have more fine grained
>> control
>> > and
>> > > > it's easier move service instances around... and you don't have to
>> > > > recompile anything.
>> > > >
>> > > > Ed
>> > > >
>> > > > On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <my...@apache.org>
>> wrote:
>> > > >
>> > > > > Hey Aleks,
>> > > > >
>> > > > > I don’t see anything in that log that would explain the behavior
>> > you’re
>> > > > > observing. Markus did most of the work on deposits. Markus? Do you
>> > have
>> > > > any
>> > > > > ideas?
>> > > > >
>> > > > > Best Regards,
>> > > > > Myrle
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
>> > > > > cheetah@monkeysintown.com>
>> > > > > wrote:
>> > > > >
>> > > > > > @Myrle: with your proposed changes things look a bit better.
>> I've
>> > > > > disabled
>> > > > > > Rhythm as you suggested and get past all microservice startups
>> > with a
>> > > > lot
>> > > > > > less exceptions (one is coming repeatedly from Netty native
>> > > transport,
>> > > > > > because epoll is not available).
>> > > > > >
>> > > > > > The one exception (actually it happened on 2 occasions) that is
>> > still
>> > > > > > there:
>> > > > > >
>> > > > > > [snip]
>> > > > > >
>> > > > > > 13:33:45.080 [main] INFO
>> > > > > o.a.f.c.d.s.DepositAccountManagementApplication -
>> > > > > > Started DepositAccountManagementApplication in 55.187 seconds
>> (JVM
>> > > > > running
>> > > > > > for 62.772)
>> > > > > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
>> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
>> > via
>> > > > > > configuration
>> > > > > > Resolving artifact
>> > > > > >
>> org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
>> > > > > > Resolving metadata
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
>> > > > > > from /root/.m2/repository (enhanced)
>> > > > > > Resolved metadata
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
>> > > > > > from /root/.m2/repository (enhanced)
>> > > > > > Resolved artifact
>> > > > > >
>> org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
>> > > > from
>> > > > > > /root/.m2/repository (enhanced)
>> > > > > > 13:34:56.800 [DiscoveryClient-0] ERROR
>> > > > c.n.discovery.TimedSupervisorTask
>> > > > > -
>> > > > > > task supervisor timed out
>> > > > > > java.util.concurrent.TimeoutException: null
>> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
>> > > > > > at
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
>> > > > > > at
>> > > > >
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > > > > > at java.lang.Thread.run(Thread.java:748)
>> > > > > > 13:34:56.801 [DiscoveryClient-1] ERROR
>> > > > c.n.discovery.TimedSupervisorTask
>> > > > > -
>> > > > > > task supervisor timed out
>> > > > > > java.util.concurrent.TimeoutException: null
>> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
>> > > > > > at
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
>> > > > > > at
>> > > > >
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > > > > > at java.lang.Thread.run(Thread.java:748)
>> > > > > > 13:34:56.916 [DiscoveryClient-2] ERROR
>> > > > c.n.discovery.TimedSupervisorTask
>> > > > > -
>> > > > > > task supervisor timed out
>> > > > > > java.util.concurrent.TimeoutException: null
>> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
>> > > > > > at
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
>> > > > > > at
>> > > > >
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > > > > > at java.lang.Thread.run(Thread.java:748)
>> > > > > > 13:35:08.586 [main] INFO
>> > o.s.c.a.AnnotationConfigApplicationContext
>> > > -
>> > > > > > Refreshing
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
>> > > > > > :
>> > > > > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context
>> > > hierarchy
>> > > > > > 13:35:10.182 [main] INFO
>> > > > o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
>> > > > > -
>> > > > > > JSR-330 'javax.inject.Inject' annotation found and supported for
>> > > > > autowiring
>> > > > > > 13:35:10.371 [main] INFO
>> > > > > >
>> o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker
>> > -
>> > > > Bean
>> > > > > > 'configurationPropertiesRebinderAutoConfiguration' of type
>> [class
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
>> > > > > > is not eligible for getting processed by all BeanPostProcessors
>> > (for
>> > > > > > example: not eligible for auto-proxying)
>> > > > > > 13:35:10.435 [background-preinit] INFO
>> > > > > o.h.validator.internal.util.Version
>> > > > > > - HV000001: Hibernate Validator 5.3.0.Final
>> > > > > >
>> > > > > > [/snip]
>> > > > > >
>> > > > > > It looks like the whole startup procedure goes through and I can
>> > see
>> > > > some
>> > > > > > kind of "summary" that lists all REST endpoints. But it stays in
>> > that
>> > > > > state
>> > > > > > for only a minute or so and then a cascade of shutdowns are
>> > triggered
>> > > > > > again. The only thing that stays alive is Eureka which complains
>> > > about
>> > > > > the
>> > > > > > inaccessible services.
>> > > > > >
>> > > > > > Any idea what else we could do other than just ordering an
>> > additional
>> > > > > > machine?
>> > > > > >
>> > > > > > Cheers,
>> > > > > >
>> > > > > > Aleks
>> > > > > >
>> > > > > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <my...@apache.org>
>> > > wrote:
>> > > > > >
>> > > > > > > Hey Aleks,
>> > > > > > >
>> > > > > > > It looks like that exception is coming from Rhythm. I suspect
>> > it's
>> > > > not
>> > > > > > the
>> > > > > > > cause of your difficulties, since you say your other services
>> are
>> > > > going
>> > > > > > > down too. You can run most of the Fineract server without
>> > Rhythm.
>> > > > You
>> > > > > > > could edit the demo-server script to leave it out for the
>> > purposes
>> > > of
>> > > > > > > testing. The consequence will be a lack of interest
>> calculations
>> > > for
>> > > > > > > disbursed loans, but for finding out what the real problem is,
>> > it'd
>> > > > be
>> > > > > > good
>> > > > > > > to get this one out of the way.
>> > > > > > >
>> > > > > > > Best Regards,
>> > > > > > > Myrle
>> > > > > > >
>> > > > > > > (FYI: I've removed the non-list subscribers from the to.
>> Most of
>> > > > them
>> > > > > > are
>> > > > > > > already subscribed, and those who aren't, don't want to be.)
>> > > > > > >
>> > > > > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
>> > > > > > > cheetah@monkeysintown.com> wrote:
>> > > > > > >
>> > > > > > > > I managed to further improve the startup procedure (aka less
>> > > > > exceptions
>> > > > > > > > than before)... there are still some timeouts, but not as
>> many
>> > as
>> > > > > > before.
>> > > > > > > >
>> > > > > > > > The demo server still dies after this exception:
>> > > > > > > >
>> > > > > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
>> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>> > endpoints
>> > > > via
>> > > > > > > > configuration
>> > > > > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
>> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>> > endpoints
>> > > > via
>> > > > > > > > configuration
>> > > > > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
>> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>> > endpoints
>> > > > via
>> > > > > > > > configuration
>> > > > > > > > 23:28:45.990 [pool-9-thread-1] ERROR
>> > > > > > > o.s.s.s.TaskUtils$LoggingErrorHandler
>> > > > > > > > - Unexpected error occurred in scheduled task.
>> > > > > > > >
>> > org.springframework.transaction.CannotCreateTransactionException:
>> > > > > Could
>> > > > > > > not
>> > > > > > > > open JPA EntityManager for transaction; nested exception is
>> > > > > > > > javax.persistence.PersistenceException:
>> > > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
>> > > > > resultset:
>> > > > > > > > unexpected end of stream, read 0 bytes from 4
>> > > > > > > > Query is : set autocommit=0
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
>> > > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
>> > > > > > > > at
>> > > > > > >
>> > > >
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > > > > > > > at
>> > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > > > > > > > at java.lang.Thread.run(Thread.java:748)
>> > > > > > > > Caused by: javax.persistence.PersistenceException:
>> > > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
>> > > > > resultset:
>> > > > > > > > unexpected end of stream, read 0 bytes from 4
>> > > > > > > > Query is : set autocommit=0
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
>> > > > > > > > at
>> > > > > > > >
>> > > > > >
>> > > >
>> >
>> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
>> > > > > > > > ... 20 common frames omitted
>> > > > > > > > Caused by: java.sql.SQLNonTransientConnectionException:
>> Could
>> > not
>> > > > > read
>> > > > > > > > resultset: unexpected end of stream, read 0 bytes from 4
>> > > > > > > > Query is : set autocommit=0
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
>> > > > > > > > at
>> > > > > > > >
>> > > > > >
>> > > >
>> >
>> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
>> > > > > > > > ... 22 common frames omitted
>> > > > > > > > Caused by:
>> org.mariadb.jdbc.internal.util.dao.QueryException:
>> > > Could
>> > > > > not
>> > > > > > > > read resultset: unexpected end of stream, read 0 bytes from
>> 4
>> > > > > > > > Query is : set autocommit=0
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
>> > > > > > > > ... 31 common frames omitted
>> > > > > > > > Caused by: java.io.EOFException: unexpected end of stream,
>> > read 0
>> > > > > bytes
>> > > > > > > > from 4
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
>> > > > > > > > at
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
>> > > > > > > > ... 33 common frames omitted
>> > > > > > > > 23:29:05.500 [qtp6422064-20] INFO
>> > > > > o.a.f.c.l.c.ServiceExceptionFilter -
>> > > > > > > > Responding with a service error ServiceError{code=409,
>> > > > > > > message='Application
>> > > > > > > > identity-v1 already exists!'}
>> > > > > > > > 23:29:21.282 [Thread-15] INFO
>> > > > > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
>> > Closing
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
>> > > > > > > > :
>> > > > > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
>> > > > > > > > 23:29:23.717 [Thread-15] INFO
>> > o.s.c.s.DefaultLifecycleProcessor
>> > > -
>> > > > > > > Stopping
>> > > > > > > > beans in phase 2147483647
>> > > > > > > > 23:29:24.267 [Thread-15] INFO
>> > o.s.c.s.DefaultLifecycleProcessor
>> > > -
>> > > > > > > Stopping
>> > > > > > > > beans in phase 0
>> > > > > > > > 23:29:24.557 [Thread-15] INFO
>> > > o.s.b.a.e.jmx.EndpointMBeanExporter
>> > > > -
>> > > > > > > > Unregistering JMX-exposed beans on shutdown
>> > > > > > > > 23:29:24.579 [Thread-15] INFO
>> > > o.s.b.a.e.jmx.EndpointMBeanExporter
>> > > > -
>> > > > > > > > Unregistering JMX-exposed beans
>> > > > > > > >
>> > > > > > > > The only thing I've left: start the databases (aka
>> Cassandra)
>> > on
>> > > a
>> > > > > > > separate
>> > > > > > > > machine (e. g. on Digital Ocean) and see how this is working
>> > > then.
>> > > > > > > >
>> > > > > > > > ... FYI.
>> > > > > > > >
>> > > > > > > > Cheers,
>> > > > > > > >
>> > > > > > > > Aleks
>> > > > > > > >
>> > > > > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
>> > > > > > > > cheetah@monkeysintown.com> wrote:
>> > > > > > > >
>> > > > > > > > > @Rajan: it's not running, because the demo server dies...
>> See
>> > > > logs
>> > > > > > from
>> > > > > > > > > previous message...
>> > > > > > > > >
>> > > > > > > > > We just have to figure that one out and we should be good
>> to
>> > > go.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
>> > > > > rajanmaurya154@gmail.com
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Thanks, Alex,
>> > > > > > > > >>
>> > > > > > > > >> I can see web app is running but I can't log in with the
>> > > > > credentials
>> > > > > > > > that
>> > > > > > > > >> you shared earlier
>> > > > > > > > >>
>> > > > > > > > >> Tenant : playground
>> > > > > > > > >> Username : operator
>> > > > > > > > >> Password : init1@l
>> > > > > > > > >>
>> > > > > > > > >> Thanks
>> > > > > > > > >>
>> > > > > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
>> > > > > > > > >> cheetah@monkeysintown.com> wrote:
>> > > > > > > > >>
>> > > > > > > > >> > Hi all,
>> > > > > > > > >> >
>> > > > > > > > >> > ... sweeeet! I got a considerable step further.
>> > > > > > > > >> >
>> > > > > > > > >> > @Markus: thanks for the conversation... gave me an idea
>> > how
>> > > to
>> > > > > fix
>> > > > > > > > (most
>> > > > > > > > >> > of) it!
>> > > > > > > > >> >
>> > > > > > > > >> > The problem was - as suspected - the memory (or lack
>> of).
>> > > The
>> > > > VM
>> > > > > > > > >> instance
>> > > > > > > > >> > at Apache has net 30G available. 16G are needed for
>> > > Cassandra
>> > > > > and
>> > > > > > in
>> > > > > > > > my
>> > > > > > > > >> > previous tests I tried to run the demo server with up
>> to
>> > > 16G.
>> > > > > The
>> > > > > > > > >> problem
>> > > > > > > > >> > is that the demo server needs considerably more memory
>> > than
>> > > > 16G.
>> > > > > > > > >> According
>> > > > > > > > >> > to my observation it's around 23-24G. Even if you limit
>> > the
>> > > > demo
>> > > > > > > > server
>> > > > > > > > >> > memory allocation with "-Xmx" to 12G (for example) it
>> > would
>> > > > just
>> > > > > > > > >> continue
>> > > > > > > > >> > to consume more memory.
>> > > > > > > > >> >
>> > > > > > > > >> > My solution: add a swap file. Not ideal in terms of
>> > > > performance,
>> > > > > > but
>> > > > > > > > it
>> > > > > > > > >> > gets the whole thing going... at least mostly (please
>> read
>> > > > on).
>> > > > > > > > >> >
>> > > > > > > > >> > Current status:
>> > > > > > > > >> >
>> > > > > > > > >> > - Cassandra (and all other Docker containers) are
>> still
>> > > > > > running;
>> > > > > > > > >> > Cassandra died in my previous attempts at the latest
>> > when
>> > > > the
>> > > > > > > > Teller
>> > > > > > > > >> > application started, but with the swap file it's
>> still
>> > > > > running
>> > > > > > > > >> > - all Spring Boot apps are starting now
>> > > > > > > > >> > - I get once in a while some spurious timeouts (I
>> think
>> > > > when
>> > > > > > > either
>> > > > > > > > >> the
>> > > > > > > > >> > Eureka server is contacted or the config server)
>> > > > > > > > >> > - there are also sometimes exceptions concerning
>> MySQL
>> > > > > > > connections,
>> > > > > > > > >> but
>> > > > > > > > >> > doesn't seem to matter overall
>> > > > > > > > >> > - I could issue an authentication request with
>> Postman
>> > to
>> > > > > > > > >> > http://fineract-vm.apache.org:2021/identity/v1;
>> just
>> > to
>> > > > test
>> > > > > > if
>> > > > > > > > >> > anything
>> > > > > > > > >> > is responding
>> > > > > > > > >> > - PROBLEM: but the demo server (the Spring Boot
>> apps)
>> > > > > > eventually
>> > > > > > > > dies
>> > > > > > > > >> > after a couple of minutes; I don't think that the
>> > memory
>> > > is
>> > > > > the
>> > > > > > > > >> problem
>> > > > > > > > >> > at
>> > > > > > > > >> > this point; I think it's more of a timeout problem
>> > > > concerning
>> > > > > > the
>> > > > > > > > >> > communication with Eureka and/or the config server
>> > (maybe
>> > > > > > because
>> > > > > > > > the
>> > > > > > > > >> > system is a bit slow)
>> > > > > > > > >> >
>> > > > > > > > >> > ... and here the memory footprint ("free -h") when
>> > > everything
>> > > > is
>> > > > > > > > >> running,
>> > > > > > > > >> > just FYI:
>> > > > > > > > >> >
>> > > > > > > > >> > total used free shared
>> > > > > > buff/cache
>> > > > > > > > >> > available
>> > > > > > > > >> > Mem: 31G 31G 226M 8.4M
>> > > > > > 169M
>> > > > > > > > >> > 47M
>> > > > > > > > >> > Swap: 15G 6.8G 9.2G
>> > > > > > > > >> >
>> > > > > > > > >> > ... after the Payroll application is started (approx.)
>> I
>> > see
>> > > > > these
>> > > > > > > > >> > exceptions appearing in the logs (a whole series of
>> them):
>> > > > > > > > >> >
>> > > > > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
>> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>> > > > endpoints
>> > > > > > via
>> > > > > > > > >> > configuration
>> > > > > > > > >> > Resolving artifact
>> > > > > > > > >> >
>> > > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
>> > > > > > > > >> > Resolving metadata
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
>> > > > > > > > >> > from /root/.m2/repository (enhanced)
>> > > > > > > > >> > Resolved metadata
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
>> > > > > > > > >> > from /root/.m2/repository (enhanced)
>> > > > > > > > >> > Resolved artifact
>> > > > > > > > >> >
>> > > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
>> > > > > > > > from
>> > > > > > > > >> > /root/.m2/repository (enhanced)
>> > > > > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
>> > > > > > > > >> c.n.discovery.TimedSupervisorTask -
>> > > > > > > > >> > task supervisor timed out
>> > > > > > > > >> > java.util.concurrent.TimeoutException: null
>> > > > > > > > >> > at
>> > java.util.concurrent.FutureTask.get(FutureTask.java:205)
>> > > > > > > > >> > at
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
>> > > > > > > > >> > at
>> > > > > > > > >>
>> > > > > > >
>> > > >
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > > > > > > > >> > at
>> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > > > > > > > >> > at
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> > > > > > > > >> > at
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> > > > > > > > >> > at
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > > > > > > > >> > at
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > > > > > > > >> > at java.lang.Thread.run(Thread.java:748)
>> > > > > > > > >> >
>> > > > > > > > >> > ... but it still seems to have no problem starting the
>> > next
>> > > > app
>> > > > > > > > (Group)
>> > > > > > > > >> and
>> > > > > > > > >> > things look good for a while...
>> > > > > > > > >> >
>> > > > > > > > >> > ... until suddenly it seems that app instances are
>> > beginning
>> > > > to
>> > > > > > > > >> shutdown...
>> > > > > > > > >> > and I'm not sure why this happens:
>> > > > > > > > >> >
>> > > > > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
>> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>> > > > endpoints
>> > > > > > via
>> > > > > > > > >> > configuration
>> > > > > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
>> > > > > > > > >> c.datastax.driver.core.ClockFactory -
>> > > > > > > > >> > Using native clock to generate timestamps.
>> > > > > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
>> > > > > > > > >> c.datastax.driver.core.CodecRegistry
>> > > > > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
>> > > > > > > > >> java.time.LocalDateTime]
>> > > > > > > > >> > because it collides with previously registered codec
>> > > > > > > > LocalDateTimeCodec
>> > > > > > > > >> > [timestamp <-> java.time.LocalDateTime]
>> > > > > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
>> > > > > > > > c.d.d.c.p.DCAwareRoundRobinPolicy
>> > > > > > > > >> -
>> > > > > > > > >> > Using data-center name 'datacenter1' for
>> > > > DCAwareRoundRobinPolicy
>> > > > > > (if
>> > > > > > > > >> this
>> > > > > > > > >> > is incorrect, please provide the correct datacenter
>> name
>> > > with
>> > > > > > > > >> > DCAwareRoundRobinPolicy constructor)
>> > > > > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
>> > > > > > > > com.datastax.driver.core.Cluster -
>> > > > > > > > >> > New Cassandra host /127.0.0.1:9042 added
>> > > > > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
>> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>> > > > endpoints
>> > > > > > via
>> > > > > > > > >> > configuration
>> > > > > > > > >> > 18:27:51.110 [Thread-15] INFO
>> > > > > > > > >> >
>> o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
>> > > > > Closing
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
>> > > > > > > > >> > :
>> > > > > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
>> > > > > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
>> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
>> > > > endpoints
>> > > > > > via
>> > > > > > > > >> > configuration
>> > > > > > > > >> > 18:27:53.373 [Thread-15] INFO
>> > > > > o.s.c.s.DefaultLifecycleProcessor -
>> > > > > > > > >> Stopping
>> > > > > > > > >> > beans in phase 2147483647
>> > > > > > > > >> > 18:27:54.331 [Thread-15] INFO
>> > > > > o.s.c.s.DefaultLifecycleProcessor -
>> > > > > > > > >> Stopping
>> > > > > > > > >> > beans in phase 0
>> > > > > > > > >> > 18:27:54.736 [Thread-15] INFO
>> > > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
>> > > > > > > > >> > Unregistering JMX-exposed beans on shutdown
>> > > > > > > > >> > 18:27:54.740 [Thread-15] INFO
>> > > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
>> > > > > > > > >> > Unregistering JMX-exposed beans
>> > > > > > > > >> > 18:27:54.798 [Thread-15] INFO
>> > > > > o.s.j.e.a.AnnotationMBeanExporter -
>> > > > > > > > >> > Unregistering JMX-exposed beans on shutdown
>> > > > > > > > >> > 18:27:54.800 [Thread-15] INFO
>> > > > > o.s.j.e.a.AnnotationMBeanExporter -
>> > > > > > > > >> > Unregistering JMX-exposed beans
>> > > > > > > > >> > 18:27:54.937 [Thread-15] INFO
>> > > > > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean -
>> Closing
>> > JPA
>> > > > > > > > >> > EntityManagerFactory for persistence unit 'metaPU'
>> > > > > > > > >> > 18:27:58.376 [async-processor-1] INFO
>> > > > > > > > >> o.f.c.internal.util.VersionPrinter -
>> > > > > > > > >> > Flyway 3.2.1 by Boxfuse
>> > > > > > > > >> > 18:27:58.677 [async-processor-1] INFO
>> > > > > > > > >> o.f.c.i.dbsupport.DbSupportFactory -
>> > > > > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL
>> 10.3)
>> > > > > > > > >> > 18:28:00.620 [async-processor-1] INFO
>> > > > > > > > >> o.f.core.internal.command.DbValidate
>> > > > > > > > >> > - Validated 2 migrations (execution time 00:01.169s)
>> > > > > > > > >> > 18:28:00.901 [Thread-15] INFO
>> > > > > o.e.jetty.server.AbstractConnector
>> > > > > > -
>> > > > > > > > >> Stopped
>> > > > > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{
>> 0.0.0.0:2020
>> > }
>> > > > > > > > >> > 18:28:01.268 [Thread-15] INFO
>> > > > > > o.e.j.server.handler.ContextHandler -
>> > > > > > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
>> > > > > > > > >> >
>> > > > > > > > >> > ... the list is actually a lot longer... in the rest of
>> > the
>> > > > log
>> > > > > > you
>> > > > > > > > can
>> > > > > > > > >> see
>> > > > > > > > >> > that one by one all the apps are shutting down.
>> > > > > > > > >> >
>> > > > > > > > >> > The last (error) entries I see are these:
>> > > > > > > > >> >
>> > > > > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR
>> > > > > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not
>> > refresh
>> > > > JMS
>> > > > > > > > >> Connection
>> > > > > > > > >> > for destination 'cheques-v1' - retrying using
>> > > > > > > > >> FixedBackOff{interval=5000,
>> > > > > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error
>> > > while
>> > > > > > > > >> attempting to
>> > > > > > > > >> > retrieve a connection from the pool; nested exception
>> is
>> > > > > > > > >> > javax.jms.JMSException: Could not connect to broker
>> URL:
>> > > > > > > > >> > tcp://localhost:61616. Reason:
>> java.net.ConnectException:
>> > > > > > Connection
>> > > > > > > > >> > refused (Connection refused)
>> > > > > > > > >> >
>> > > > > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in
>> the
>> > > > > process
>> > > > > > of
>> > > > > > > > the
>> > > > > > > > >> > previous errors.
>> > > > > > > > >> >
>> > > > > > > > >> > So far from here... any input on the above highly
>> > > appreciated.
>> > > > > > > > Overall I
>> > > > > > > > >> > think we are close.
>> > > > > > > > >> >
>> > > > > > > > >> > Cheers,
>> > > > > > > > >> >
>> > > > > > > > >> > Aleks
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <
>> > > mage@apache.org
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > One thing we maybe can do, given we are part of the
>> > Apache
>> > > > > > Family,
>> > > > > > > > is
>> > > > > > > > >> to
>> > > > > > > > >> > > ask the Cassandra community for some suggestions.
>> > > > > > > > >> > >
>> > > > > > > > >> > > Cheers
>> > > > > > > > >> > >
>> > > > > > > > >> > > Markus
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <
>> > > > mage@apache.org
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >> > >
>> > > > > > > > >> > > > Hey all,
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > we are not running Cassandra in a container, we
>> have
>> > > > > dedicated
>> > > > > > > VMs
>> > > > > > > > >> to
>> > > > > > > > >> > run
>> > > > > > > > >> > > > our cluster.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Given this we are not the best persons to help
>> here,
>> > > > sorry.
>> > > > > (;
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Cheers
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Markus
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
>> > > > > edcable@mifos.org>
>> > > > > > > > >> wrote:
>> > > > > > > > >> > > >
>> > > > > > > > >> > > >> Adding Patric directly to this thread too so he
>> can
>> > > give
>> > > > > his
>> > > > > > > > input
>> > > > > > > > >> > > >> regarding to the challenges with Cassandra.
>> > > > > > > > >> > > >>
>> > > > > > > > >> > > >> Ed
>> > > > > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar
>> > Vidakovic <
>> > > > > > > > >> > > >> cheetah@monkeysintown.com> wrote:
>> > > > > > > > >> > > >>
>> > > > > > > > >> > > >>> Hi Victor,
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> .... this is the relevant part of the
>> > > docker-compose.yml
>> > > > > > file:
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> cassandra0:
>> > > > > > > > >> > > >>> image: cassandra:3.11.1
>> > > > > > > > >> > > >>> container_name: cassandra0
>> > > > > > > > >> > > >>> ports:
>> > > > > > > > >> > > >>> - 9042:9042
>> > > > > > > > >> > > >>> - 9160:9160
>> > > > > > > > >> > > >>> - 7199:7199
>> > > > > > > > >> > > >>> - 8778:8778
>> > > > > > > > >> > > >>> volumes:
>> > > > > > > > >> > > >>> - ./cassandra:/etc/cassandra
>> > > > > > > > >> > > >>> environment:
>> > > > > > > > >> > > >>> - CASSANDRA_START_RPC=true
>> > > > > > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0
>> > > > > > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster
>> > > > > > > > >> > > >>> ulimits:
>> > > > > > > > >> > > >>> memlock: -1
>> > > > > > > > >> > > >>> nproc: 32768
>> > > > > > > > >> > > >>> nofile: 100000
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> ... and just to be complete... here's the Docker
>> > > service
>> > > > > > > > >> > configuration
>> > > > > > > > >> > > >>> ("/lib/systemd/system/docker.service"):
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> [Unit]
>> > > > > > > > >> > > >>> Description=Docker Application Container Engine
>> > > > > > > > >> > > >>> Documentation=https://docs.docker.com
>> > > > > > > > >> > > >>> After=network-online.target docker.socket
>> > > > > firewalld.service
>> > > > > > > > >> > > >>> Wants=network-online.target
>> > > > > > > > >> > > >>> Requires=docker.socket
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> [Service]
>> > > > > > > > >> > > >>> Type=notify
>> > > > > > > > >> > > >>> # the default is not to use systemd for cgroups
>> > > because
>> > > > > the
>> > > > > > > > >> delegate
>> > > > > > > > >> > > >>> issues
>> > > > > > > > >> > > >>> still
>> > > > > > > > >> > > >>> # exists and systemd currently does not support
>> the
>> > > > cgroup
>> > > > > > > > feature
>> > > > > > > > >> > set
>> > > > > > > > >> > > >>> required
>> > > > > > > > >> > > >>> # for containers run by docker
>> > > > > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
>> > > > > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
>> > > > > > > > >> > > >>> LimitNOFILE=1048576
>> > > > > > > > >> > > >>> # Having non-zero Limit*s causes performance
>> > problems
>> > > > due
>> > > > > to
>> > > > > > > > >> > accounting
>> > > > > > > > >> > > >>> overhead
>> > > > > > > > >> > > >>> # in the kernel. We recommend using cgroups to do
>> > > > > > > > container-local
>> > > > > > > > >> > > >>> accounting.
>> > > > > > > > >> > > >>> LimitNPROC=infinity
>> > > > > > > > >> > > >>> LimitCORE=infinity
>> > > > > > > > >> > > >>> LimitMEMLOCK=infinity
>> > > > > > > > >> > > >>> # Uncomment TasksMax if your systemd version
>> > supports
>> > > > it.
>> > > > > > > > >> > > >>> # Only systemd 226 and above support this
>> version.
>> > > > > > > > >> > > >>> TasksMax=infinity
>> > > > > > > > >> > > >>> TimeoutStartSec=0
>> > > > > > > > >> > > >>> # set delegate yes so that systemd does not reset
>> > the
>> > > > > > cgroups
>> > > > > > > of
>> > > > > > > > >> > docker
>> > > > > > > > >> > > >>> containers
>> > > > > > > > >> > > >>> Delegate=yes
>> > > > > > > > >> > > >>> # kill only the docker process, not all
>> processes in
>> > > the
>> > > > > > > cgroup
>> > > > > > > > >> > > >>> KillMode=process
>> > > > > > > > >> > > >>> # restart the docker process if it exits
>> prematurely
>> > > > > > > > >> > > >>> Restart=on-failure
>> > > > > > > > >> > > >>> StartLimitBurst=3
>> > > > > > > > >> > > >>> StartLimitInterval=60s
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> [Install]
>> > > > > > > > >> > > >>> WantedBy=multi-user.target
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> ... the one tweak I did there is to set
>> LimitMEMLOCK
>> > > to
>> > > > > > > > >> infinity...
>> > > > > > > > >> > do
>> > > > > > > > >> > > >>> you
>> > > > > > > > >> > > >>> think the problems are file handle related?
>> Should I
>> > > set
>> > > > > > > > >> LimitNOFILE
>> > > > > > > > >> > > also
>> > > > > > > > >> > > >>> to infinity?
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> Appreciate the help.
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> Cheers,
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> Aleks
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
>> > > > > > > > >> > > >>> victor.romero@fintecheando.mx>
>> > > > > > > > >> > > >>> wrote:
>> > > > > > > > >> > > >>>
>> > > > > > > > >> > > >>> > Hi Aleks,
>> > > > > > > > >> > > >>> >
>> > > > > > > > >> > > >>> > Can you share the ulimits flags that the
>> > cassandra's
>> > > > > > > container
>> > > > > > > > >> is
>> > > > > > > > >> > > using
>> > > > > > > > >> > > >>> > while it is running?
>> > > > > > > > >> > > >>> >
>> > > > > > > > >> > > >>> > The values are being set in the composer file?
>> Or
>> > in
>> > > > the
>> > > > > > > > >> upstart or
>> > > > > > > > >> > > >>> > systemd docker's deamon config files?
>> > > > > > > > >> > > >>> >
>> > > > > > > > >> > > >>> >
>> > > > > > > > >> > > >>> >
>> > > > > > > > >> > > >>> > Enviado desde TypeApp
>> > > > > > > > >> > > >>> >
>> > > > > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM,
>> Aleksandar
>> > > > > > > Vidakovic <
>> > > > > > > > >> > > >>> > cheetah@monkeysintown.com> escribió:
>> > > > > > > > >> > > >>> > >... and I should have attached the Cassandra
>> log
>> > > > > dump...
>> > > > > > so
>> > > > > > > > >> here
>> > > > > > > > >> > it
>> > > > > > > > >> > > >>> > >is...
>> > > > > > > > >> > > >>> > >
>> > > > > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar
>> > > Vidakovic
>> > > > <
>> > > > > > > > >> > > >>> > >cheetah@monkeysintown.com> wrote:
>> > > > > > > > >> > > >>> > >
>> > > > > > > > >> > > >>> > >> Hi all,
>> > > > > > > > >> > > >>> > >>
>> > > > > > > > >> > > >>> > >> ... as you might have noticed I did a
>> couple of
>> > > > > > restarts
>> > > > > > > > >> > today...
>> > > > > > > > >> > > >>> the
>> > > > > > > > >> > > >>> > >> problem I am facing now: I can't get
>> Cassandra
>> > to
>> > > > run
>> > > > > > in
>> > > > > > > a
>> > > > > > > > >> > stable
>> > > > > > > > >> > > >>> > >way.
>> > > > > > > > >> > > >>> > >>
>> > > > > > > > >> > > >>> > >> Things look quite OK for a while when
>> running
>> > the
>> > > > > demo
>> > > > > > > > >> server,
>> > > > > > > > >> > but
>> > > > > > > > >> > > >>> > >then
>> > > > > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts
>> > > > > > misbehaving
>> > > > > > > > with
>> > > > > > > > >> > the
>> > > > > > > > >> > > >>> > >deposit
>> > > > > > > > >> > > >>> > >> microservice startup, sometimes with
>> portfolio
>> > or
>> > > > > > > teller).
>> > > > > > > > I
>> > > > > > > > >> > tried
>> > > > > > > > >> > > >>> to
>> > > > > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set
>> > some
>> > > > > > Docker
>> > > > > > > > >> limits
>> > > > > > > > >> > to
>> > > > > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
>> > > > > > > > >> > > >>> > >>
>> > > > > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe
>> > > someone
>> > > > > can
>> > > > > > > > help
>> > > > > > > > >> out
>> > > > > > > > >> > > >>> > >here? Is
>> > > > > > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB
>> of
>> > > > > memory?
>> > > > > > > > >> > > >>> > >>
>> > > > > > > > >> > > >>> > >> Other than that the setup would be ready to
>> > go...
>> > > > we
>> > > > > > just
>> > > > > > > > >> need
>> > > > > > > > >> > to
>> > > > > > > > >> > > >>> get
>> > > > > > > > >> > > >>> > >the
>> > > > > > > > >> > > >>> > >> database running more reliably.
>> > > > > > > > >> > > >>> > >>
>> > > > > > > > >> > > >>> > >> Cheers,
>> > > > > > > > >> > > >>> > >>
>> > > > > > > > >> > > >>> > >> Aleks
>> > > > > > > > >> > > >>> > >>
>> > > > > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar
>> > > > Vidakovic
>> > > > > <
>> > > > > > > > >> > > >>> > >> cheetah@monkeysintown.com> wrote:
>> > > > > > > > >> > > >>> > >>
>> > > > > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed
>> to
>> > add
>> > > > > some
>> > > > > > > > >> > additional
>> > > > > > > > >> > > >>> > >reverse
>> > > > > > > > >> > > >>> > >>> proxy configuration for the web UI to
>> work)...
>> > > and
>> > > > > > there
>> > > > > > > > >> were
>> > > > > > > > >> > > more
>> > > > > > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
>> > > > > > > > >> > > >>> > >>>
>> > > > > > > > >> > > >>> > >>> Just FYI
>> > > > > > > > >> > > >>> > >>>
>> > > > > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
>> > > > > Vidakovic <
>> > > > > > > > >> > > >>> > >>> cheetah@monkeysintown.com> wrote:
>> > > > > > > > >> > > >>> > >>>
>> > > > > > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can
>> > get
>> > > > back
>> > > > > > to
>> > > > > > > > >> testing
>> > > > > > > > >> > > >>> only
>> > > > > > > > >> > > >>> > >>>> later tonight.
>> > > > > > > > >> > > >>> > >>>>
>> > > > > > > > >> > > >>> > >>>> Cheers
>> > > > > > > > >> > > >>> > >>>>
>> > > > > > > > >> > > >>> > >>>>
>> > > > > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya
>> > > > > > > > >> > > >>> > ><ra...@gmail.com>
>> > > > > > > > >> > > >>> > >>>> wrote:
>> > > > > > > > >> > > >>> > >>>>
>> > > > > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test
>> after
>> > 30
>> > > > min.
>> > > > > > > > >> > > >>> > >>>>>
>> > > > > > > > >> > > >>> > >>>>> Big thanks for this 🙂
>> > > > > > > > >> > > >>> > >>>>>
>> > > > > > > > >> > > >>> > >>>>>
>> > > > > > > > >> > > >>> > >>>>>
>> > > > > > > > >> > > >>> > >>>>>
>> > > > > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM
>> Aleksandar
>> > > > > > Vidakovic <
>> > > > > > > > >> > > >>> > >>>>> cheetah@monkeysintown.com> wrote:
>> > > > > > > > >> > > >>> > >>>>>
>> > > > > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted.
>> > > > > > > > >> > > >>> > >>>>> >
>> > > > > > > > >> > > >>> > >>>>> > ... and as I said: the services are
>> still
>> > > > > > > starting...
>> > > > > > > > >> and
>> > > > > > > > >> > the
>> > > > > > > > >> > > >>> > >best
>> > > > > > > > >> > > >>> > >>>>> estimate
>> > > > > > > > >> > > >>> > >>>>> > I have right now (as already
>> mentioned):
>> > > 30min
>> > > > > > > > >> > > >>> > >>>>> >
>> > > > > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan
>> > > Maurya <
>> > > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
>> > > > > > > > >> > > >>> > >>>>> > wrote:
>> > > > > > > > >> > > >>> > >>>>> >
>> > > > > > > > >> > > >>> > >>>>> > > [image: image.png]
>> > > > > > > > >> > > >>> > >>>>> > >
>> > > > > > > > >> > > >>> > >>>>> > >
>> > > > > > > > >> > > >>> > >>>>> > >
>> > > > > > > > >> > > >>> > >>>>> > >
>> > > > > > > > >> > > >>> > >>>>> > >
>> > > > > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan
>> > > > Maurya <
>> > > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
>> > > > > > > > >> > > >>> > >>>>> > > wrote:
>> > > > > > > > >> > > >>> > >>>>> > >
>> > > > > > > > >> > > >>> > >>>>> > >> [image: image.png]
>> > > > > > > > >> > > >>> > >>>>> > >> I am getting this.
>> > > > > > > > >> > > >>> > >>>>> > >>
>> > > > > > > > >> > > >>> > >>>>> > >>
>> > > > > > > > >> > > >>> > >>>>> > >>
>> > > > > > > > >> > > >>> > >>>>> > >>
>> > > > > > > > >> > > >>> > >>>>> > >>
>> > > > > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM
>> > > Aleksandar
>> > > > > > > > Vidakovic
>> > > > > > > > >> <
>> > > > > > > > >> > > >>> > >>>>> > >> cheetah@monkeysintown.com> wrote:
>> > > > > > > > >> > > >>> > >>>>> > >>
>> > > > > > > > >> > > >>> > >>>>> > >>> Hi all,
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost)
>> > > ready...
>> > > > > it
>> > > > > > > took
>> > > > > > > > >> me a
>> > > > > > > > >> > > >>> > >moment
>> > > > > > > > >> > > >>> > >>>>> and a
>> > > > > > > > >> > > >>> > >>>>> > >>> couple
>> > > > > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot
>> > > > > > failures...
>> > > > > > > > the
>> > > > > > > > >> > > >>> services
>> > > > > > > > >> > > >>> > >are
>> > > > > > > > >> > > >>> > >>>>> quite
>> > > > > > > > >> > > >>> > >>>>> > >>> resource hungry and default
>> settings
>> > > won't
>> > > > > do
>> > > > > > > it.
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> The services are still starting and
>> > this
>> > > > > will
>> > > > > > > > take a
>> > > > > > > > >> > > while
>> > > > > > > > >> > > >>> > >(my
>> > > > > > > > >> > > >>> > >>>>> best
>> > > > > > > > >> > > >>> > >>>>> > guess
>> > > > > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
>> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org/login
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> Credentials:
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> Tenant : playground
>> > > > > > > > >> > > >>> > >>>>> > >>> Username : operator
>> > > > > > > > >> > > >>> > >>>>> > >>> Password : init1@l
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to
>> > > compile
>> > > > > this
>> > > > > > > app
>> > > > > > > > >> is
>> > > > > > > > >> > > with
>> > > > > > > > >> > > >>> > >NodeJS
>> > > > > > > > >> > > >>> > >>>>> > 8.11.1
>> > > > > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't
>> > > work).
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> Identity Service:
>> > > > > > > > >> > > >>> > >
>> http://fineract-vm.apache.org:2021/identity/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Office Service:
>> > > > > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Customer Service:
>> > > > > > > > >> > > >>> > >
>> http://fineract-vm.apache.org:2024/customer/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Accounting Service:
>> > > > > > > > >> > > >>> > >>>>>
>> > > > http://fineract-vm.apache.org:2025/accounting/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Portfolio Service:
>> > > > > > > > >> > > >>> > >>>>>
>> > > http://fineract-vm.apache.org:2026/portfolio/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Deposit Service:
>> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Teller Service:
>> > > > > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Reporting Service:
>> > > > > > > > >> > > >>> > >>>>>
>> > > http://fineract-vm.apache.org:2029/reporting/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Cheque Service:
>> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1
>> > > > > > > > >> > > >>> > >>>>> > >>> Payroll Service:
>> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes
>> > > quite
>> > > > a
>> > > > > > > > >> while... if
>> > > > > > > > >> > > you
>> > > > > > > > >> > > >>> > >>>>> encounter
>> > > > > > > > >> > > >>> > >>>>> > >>> connection problems retry a couple
>> of
>> > > > > minutes
>> > > > > > > > later
>> > > > > > > > >> to
>> > > > > > > > >> > > >>> > >ensure I
>> > > > > > > > >> > > >>> > >>>>> am not
>> > > > > > > > >> > > >>> > >>>>> > >>> currently working on something.
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the
>> > backend
>> > > > > > services
>> > > > > > > > >> once
>> > > > > > > > >> > per
>> > > > > > > > >> > > >>> > >day or
>> > > > > > > > >> > > >>> > >>>>> so to
>> > > > > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to
>> > > figure
>> > > > > out
>> > > > > > > how
>> > > > > > > > we
>> > > > > > > > >> > want
>> > > > > > > > >> > > >>> to
>> > > > > > > > >> > > >>> > >>>>> handle
>> > > > > > > > >> > > >>> > >>>>> > this
>> > > > > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where
>> to
>> > put
>> > > > > this
>> > > > > > > demo
>> > > > > > > > >> > server
>> > > > > > > > >> > > >>> > >>>>> configuration
>> > > > > > > > >> > > >>> > >>>>> > >>> then let me know here.
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any
>> > > problems
>> > > > (I
>> > > > > > > have
>> > > > > > > > >> not
>> > > > > > > > >> > > >>> > >extensively
>> > > > > > > > >> > > >>> > >>>>> > tested
>> > > > > > > > >> > > >>> > >>>>> > >>> it
>> > > > > > > > >> > > >>> > >>>>> > >>> yet).
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> Cheers,
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> Aleks
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM
>> > > > Aleksandar
>> > > > > > > > >> Vidakovic <
>> > > > > > > > >> > > >>> > >>>>> > >>> cheetah@monkeysintown.com> wrote:
>> > > > > > > > >> > > >>> > >>>>> > >>>
>> > > > > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem
>> to be
>> > > > > > starting
>> > > > > > > > fine
>> > > > > > > > >> > > now...
>> > > > > > > > >> > > >>> > >just
>> > > > > > > > >> > > >>> > >>>>> have
>> > > > > > > > >> > >
>>
>
>
> --
> *Ed Cable*
> President/CEO, Mifos Initiative
> edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
>
> *Collectively Creating a World of 3 Billion Maries | *http://mifos.org
> <http://facebook.com/mifos> <http://www.twitter.com/mifos>
>
>
--
*Ed Cable*
President/CEO, Mifos Initiative
edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
*Collectively Creating a World of 3 Billion Maries | *http://mifos.org
<http://facebook.com/mifos> <http://www.twitter.com/mifos>
Re: Status of Demo Server on Apache VM Infrastructure
Posted by Ed Cable <ed...@mifos.org>.
@Avik Ganguly <av...@gmail.com> were you able to take up any of
the suggested tasks by Myrle?
@Aleksandar Vidakovic <ch...@monkeysintown.com> Can you provide an update
on the status of deploying Fineract CN on the Apache VMs?
Thanks,
Ed
On Thu, Oct 25, 2018 at 3:39 AM Myrle Krantz <my...@apache.org> wrote:
> Hi Avik,
>
> You've asked several questions here, which is why it took me a while to get
> to answering.
>
> Before I go on, allow me to clarify my use of certain terms in the context
> of Fineract CN:
> * unit test -- a test for just one or two classes. Doesn't require
> spinning up the service. Doesn't require resources like a DB or user
> interaction. Is fast enough to be made part of the build. These tests are
> included in the same module and package as the class which they are
> intended to test. (This is a definition which is generally agreed upon in
> the industry.)
> * component test -- a test for a service. Requires spinning up just that
> service. Requires the use of an embedded DB. Individual tests might be
> fast, but the testsuite is rather expensive to run because startup times
> are impacted by running two DBs and spinning up the service. These tests
> are included in the same repository as the service they are intended to
> test. External services are mocked. (The industry has no widely agreed
> upon definition for component tests; this is my definition.)
> * integration test -- a test for multiple services. These tests have their
> own repositories.
>
> demo-server serves three purposes:
> * It illustrates the steps need to start and provision the services.
> * It serves as an integration test for one possible the provisioning
> workflow.
> * It makes it possible to start up the services locally so that a UI
> developer can program against the latest changes.
>
> demo-server is not intended to be a used in production for multiple
> reasons:
> * It's slow. It starts everything, even stuff you may not need, and it
> provisions everything. It then waits for everything to register itself
> with Eureka. It lowers the times for the eureka server to speed up
> startup, but this results in to many cycles being used for Eureka updates.
> * It's not secure. For example: identity.token.refresh.secureCookie is set
> to false to make http transport possible.
>
> If you're creating a new service, I don't expect that you'll need to run
> demo-server very often. Your emphasis should be on your component tests.
> These can run in just a minute or two, and you don't have to worry (much)
> about JWT tokens, headers, and the like inside of these tests.
>
> If demo-server doesn't serve your purposes I see a few possibilities:
> * create a private fork of the demo-server, then create the setup you
> prefer.
> * submit a PR with another "test" in demo-server which starts the services
> you need. I'd be happy to help you with this.
> * submit a PR which makes the starting of various services in the
> demo-service optional. I'd be happy to help you with this as well.
>
> One idea for speeding up local deployments (which is intended only to
> enable developers and not to be a production deployment method) is to start
> all the services in one process. This would reduce service startup times
> by folding them all into one. It would also replace inter-process
> communication during provisioning with in-process communication. I expect
> that we could get startup under 5 minutes. It would also make it possible
> to do component-testing on services which are dependent on other services
> without extensive, error-prone mocking. Writing tests for portfolio
> probably cost me more time than writing the code, and debugging mocks isn't
> fun. But those benefits would only apply for development environments.
> That's what FINCN-25 is about.
>
> Rhythm isn't resource-hungry. Not sure where that came from...
>
> I don't believe Cassandra is the problem here, but I haven't examined that
> closely. Please prove me wrong. I'd love to see a fix like that, if it
> really is that easy. : o)
>
> Best Regards,
> Myrle
>
>
>
> On Wed, Oct 10, 2018 at 6:24 PM Avik Ganguly <av...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > I am assuming that the suggested form of development for Fineract CN is
> to
> > write unit tests to test the code which you just wrote locally and then
> > push the code to a remote cluster or a massive VM and wait 40+ minutes to
> > run the API / integration tests there. A lot of the confusion regarding
> the
> > stacktraces in identity, rhythm, timeouts and auto-shutdown might or
> might
> > not have anything to do with the resource crunch in my local. (i7, 8GB,
> > SSD)
> >
> > Maybe a FAQ section in "How to build Fineract CN" to help devs with local
> > setup would go a long way to bring in contributions to the core modules.
> >
> > 1. Change demo server orchestration logic from running Identity,
> > Organization, Customer, Accounting, Portfolio, Deposit, Teller,
> Reporting,
> > Cheque, Payroll, Group and Notification services to running only
> Identity,
> > Organization, Customer and working set of your choice (Portfolio and
> > Accounting OR Deposit and Accounting OR Group and Portfolio)
> >
> > 2. Leave out Rhythm from orchestration logic for now. A JIRA ticket
> > tracking ability of Rhythm to run with low resource footprint will
> probably
> > help.
> >
> >
> > @Myrle, can you elaborate on FINCN-25? Is the motivation of this ticket
> to
> > reduce resource footprint only or also to reduce the startup time from 40
> > minutes to a reasonable couple of minutes? Do you have any debugging tips
> > with respect to the slow startup caused by services is due to registering
> > with eureka or provisioning or sequential startup?
> >
> > Maybe we should also follow up in Cassandra mailing list regarding
> > configuring Cassandra to run within 8GB of RAM and include the same
> config
> > as part of the FAQ.
> >
> >
> > Regards,
> > Avik.
> > ᐧ
> >
> > On Wed, Sep 19, 2018 at 2:23 AM Courage Angeh <co...@gmail.com>
> > wrote:
> >
> > > Thanks for the update. Ed.
> > >
> > > If you need an extra hand please let me know Aleks.
> > >
> > > On Tue, Sep 18, 2018, 4:27 PM Ed Cable <ed...@mifos.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Just to bring this thread back up to top of everyone's inbox. Aleks
> > > > estimates he has about a day left to get the demo server up and
> running
> > > is
> > > > hoping to tackle it by tomorrow. Here are the two approaches he
> > suggested
> > > > now that we have the second VM allocated:
> > > >
> > > > - continue with the JUnit based startup sequence of the demo module;
> > the
> > > > one thing that I would need to change is to split up the startup
> > sequence
> > > > and recompile
> > > > - create a set of Docker Compose files to do the startup on the 2 VM
> > > > instances (maybe I could borrow some stuff from Courage)
> > > >
> > > > I would prefer the latter, because you have more fine grained control
> > and
> > > > it's easier move service instances around... and you don't have to
> > > > recompile anything.
> > > >
> > > > Ed
> > > >
> > > > On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <my...@apache.org>
> wrote:
> > > >
> > > > > Hey Aleks,
> > > > >
> > > > > I don’t see anything in that log that would explain the behavior
> > you’re
> > > > > observing. Markus did most of the work on deposits. Markus? Do you
> > have
> > > > any
> > > > > ideas?
> > > > >
> > > > > Best Regards,
> > > > > Myrle
> > > > >
> > > > >
> > > > >
> > > > > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
> > > > > cheetah@monkeysintown.com>
> > > > > wrote:
> > > > >
> > > > > > @Myrle: with your proposed changes things look a bit better. I've
> > > > > disabled
> > > > > > Rhythm as you suggested and get past all microservice startups
> > with a
> > > > lot
> > > > > > less exceptions (one is coming repeatedly from Netty native
> > > transport,
> > > > > > because epoll is not available).
> > > > > >
> > > > > > The one exception (actually it happened on 2 occasions) that is
> > still
> > > > > > there:
> > > > > >
> > > > > > [snip]
> > > > > >
> > > > > > 13:33:45.080 [main] INFO
> > > > > o.a.f.c.d.s.DepositAccountManagementApplication -
> > > > > > Started DepositAccountManagementApplication in 55.187 seconds
> (JVM
> > > > > running
> > > > > > for 62.772)
> > > > > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > > > configuration
> > > > > > Resolving artifact
> > > > > >
> org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > Resolving metadata
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > from /root/.m2/repository (enhanced)
> > > > > > Resolved metadata
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > from /root/.m2/repository (enhanced)
> > > > > > Resolved artifact
> > > > > >
> org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > from
> > > > > > /root/.m2/repository (enhanced)
> > > > > > 13:34:56.800 [DiscoveryClient-0] ERROR
> > > > c.n.discovery.TimedSupervisorTask
> > > > > -
> > > > > > task supervisor timed out
> > > > > > java.util.concurrent.TimeoutException: null
> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > > at
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > 13:34:56.801 [DiscoveryClient-1] ERROR
> > > > c.n.discovery.TimedSupervisorTask
> > > > > -
> > > > > > task supervisor timed out
> > > > > > java.util.concurrent.TimeoutException: null
> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > > at
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > 13:34:56.916 [DiscoveryClient-2] ERROR
> > > > c.n.discovery.TimedSupervisorTask
> > > > > -
> > > > > > task supervisor timed out
> > > > > > java.util.concurrent.TimeoutException: null
> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > > at
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > 13:35:08.586 [main] INFO
> > o.s.c.a.AnnotationConfigApplicationContext
> > > -
> > > > > > Refreshing
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > :
> > > > > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context
> > > hierarchy
> > > > > > 13:35:10.182 [main] INFO
> > > > o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
> > > > > -
> > > > > > JSR-330 'javax.inject.Inject' annotation found and supported for
> > > > > autowiring
> > > > > > 13:35:10.371 [main] INFO
> > > > > >
> o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker
> > -
> > > > Bean
> > > > > > 'configurationPropertiesRebinderAutoConfiguration' of type [class
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
> > > > > > is not eligible for getting processed by all BeanPostProcessors
> > (for
> > > > > > example: not eligible for auto-proxying)
> > > > > > 13:35:10.435 [background-preinit] INFO
> > > > > o.h.validator.internal.util.Version
> > > > > > - HV000001: Hibernate Validator 5.3.0.Final
> > > > > >
> > > > > > [/snip]
> > > > > >
> > > > > > It looks like the whole startup procedure goes through and I can
> > see
> > > > some
> > > > > > kind of "summary" that lists all REST endpoints. But it stays in
> > that
> > > > > state
> > > > > > for only a minute or so and then a cascade of shutdowns are
> > triggered
> > > > > > again. The only thing that stays alive is Eureka which complains
> > > about
> > > > > the
> > > > > > inaccessible services.
> > > > > >
> > > > > > Any idea what else we could do other than just ordering an
> > additional
> > > > > > machine?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Aleks
> > > > > >
> > > > > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <my...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Hey Aleks,
> > > > > > >
> > > > > > > It looks like that exception is coming from Rhythm. I suspect
> > it's
> > > > not
> > > > > > the
> > > > > > > cause of your difficulties, since you say your other services
> are
> > > > going
> > > > > > > down too. You can run most of the Fineract server without
> > Rhythm.
> > > > You
> > > > > > > could edit the demo-server script to leave it out for the
> > purposes
> > > of
> > > > > > > testing. The consequence will be a lack of interest
> calculations
> > > for
> > > > > > > disbursed loans, but for finding out what the real problem is,
> > it'd
> > > > be
> > > > > > good
> > > > > > > to get this one out of the way.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Myrle
> > > > > > >
> > > > > > > (FYI: I've removed the non-list subscribers from the to. Most
> of
> > > > them
> > > > > > are
> > > > > > > already subscribed, and those who aren't, don't want to be.)
> > > > > > >
> > > > > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
> > > > > > > cheetah@monkeysintown.com> wrote:
> > > > > > >
> > > > > > > > I managed to further improve the startup procedure (aka less
> > > > > exceptions
> > > > > > > > than before)... there are still some timeouts, but not as
> many
> > as
> > > > > > before.
> > > > > > > >
> > > > > > > > The demo server still dies after this exception:
> > > > > > > >
> > > > > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > > > configuration
> > > > > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > > > configuration
> > > > > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > > > configuration
> > > > > > > > 23:28:45.990 [pool-9-thread-1] ERROR
> > > > > > > o.s.s.s.TaskUtils$LoggingErrorHandler
> > > > > > > > - Unexpected error occurred in scheduled task.
> > > > > > > >
> > org.springframework.transaction.CannotCreateTransactionException:
> > > > > Could
> > > > > > > not
> > > > > > > > open JPA EntityManager for transaction; nested exception is
> > > > > > > > javax.persistence.PersistenceException:
> > > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > > > resultset:
> > > > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > > > Query is : set autocommit=0
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
> > > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> > > > > > > > at
> > > > > > >
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > > at
> > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > Caused by: javax.persistence.PersistenceException:
> > > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > > > resultset:
> > > > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > > > Query is : set autocommit=0
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > >
> > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
> > > > > > > > ... 20 common frames omitted
> > > > > > > > Caused by: java.sql.SQLNonTransientConnectionException: Could
> > not
> > > > > read
> > > > > > > > resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > > > Query is : set autocommit=0
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > >
> > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
> > > > > > > > ... 22 common frames omitted
> > > > > > > > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException:
> > > Could
> > > > > not
> > > > > > > > read resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > > > Query is : set autocommit=0
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
> > > > > > > > ... 31 common frames omitted
> > > > > > > > Caused by: java.io.EOFException: unexpected end of stream,
> > read 0
> > > > > bytes
> > > > > > > > from 4
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
> > > > > > > > ... 33 common frames omitted
> > > > > > > > 23:29:05.500 [qtp6422064-20] INFO
> > > > > o.a.f.c.l.c.ServiceExceptionFilter -
> > > > > > > > Responding with a service error ServiceError{code=409,
> > > > > > > message='Application
> > > > > > > > identity-v1 already exists!'}
> > > > > > > > 23:29:21.282 [Thread-15] INFO
> > > > > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> > Closing
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > > > :
> > > > > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > > > 23:29:23.717 [Thread-15] INFO
> > o.s.c.s.DefaultLifecycleProcessor
> > > -
> > > > > > > Stopping
> > > > > > > > beans in phase 2147483647
> > > > > > > > 23:29:24.267 [Thread-15] INFO
> > o.s.c.s.DefaultLifecycleProcessor
> > > -
> > > > > > > Stopping
> > > > > > > > beans in phase 0
> > > > > > > > 23:29:24.557 [Thread-15] INFO
> > > o.s.b.a.e.jmx.EndpointMBeanExporter
> > > > -
> > > > > > > > Unregistering JMX-exposed beans on shutdown
> > > > > > > > 23:29:24.579 [Thread-15] INFO
> > > o.s.b.a.e.jmx.EndpointMBeanExporter
> > > > -
> > > > > > > > Unregistering JMX-exposed beans
> > > > > > > >
> > > > > > > > The only thing I've left: start the databases (aka Cassandra)
> > on
> > > a
> > > > > > > separate
> > > > > > > > machine (e. g. on Digital Ocean) and see how this is working
> > > then.
> > > > > > > >
> > > > > > > > ... FYI.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Aleks
> > > > > > > >
> > > > > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
> > > > > > > > cheetah@monkeysintown.com> wrote:
> > > > > > > >
> > > > > > > > > @Rajan: it's not running, because the demo server dies...
> See
> > > > logs
> > > > > > from
> > > > > > > > > previous message...
> > > > > > > > >
> > > > > > > > > We just have to figure that one out and we should be good
> to
> > > go.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
> > > > > rajanmaurya154@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Thanks, Alex,
> > > > > > > > >>
> > > > > > > > >> I can see web app is running but I can't log in with the
> > > > > credentials
> > > > > > > > that
> > > > > > > > >> you shared earlier
> > > > > > > > >>
> > > > > > > > >> Tenant : playground
> > > > > > > > >> Username : operator
> > > > > > > > >> Password : init1@l
> > > > > > > > >>
> > > > > > > > >> Thanks
> > > > > > > > >>
> > > > > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
> > > > > > > > >> cheetah@monkeysintown.com> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi all,
> > > > > > > > >> >
> > > > > > > > >> > ... sweeeet! I got a considerable step further.
> > > > > > > > >> >
> > > > > > > > >> > @Markus: thanks for the conversation... gave me an idea
> > how
> > > to
> > > > > fix
> > > > > > > > (most
> > > > > > > > >> > of) it!
> > > > > > > > >> >
> > > > > > > > >> > The problem was - as suspected - the memory (or lack
> of).
> > > The
> > > > VM
> > > > > > > > >> instance
> > > > > > > > >> > at Apache has net 30G available. 16G are needed for
> > > Cassandra
> > > > > and
> > > > > > in
> > > > > > > > my
> > > > > > > > >> > previous tests I tried to run the demo server with up to
> > > 16G.
> > > > > The
> > > > > > > > >> problem
> > > > > > > > >> > is that the demo server needs considerably more memory
> > than
> > > > 16G.
> > > > > > > > >> According
> > > > > > > > >> > to my observation it's around 23-24G. Even if you limit
> > the
> > > > demo
> > > > > > > > server
> > > > > > > > >> > memory allocation with "-Xmx" to 12G (for example) it
> > would
> > > > just
> > > > > > > > >> continue
> > > > > > > > >> > to consume more memory.
> > > > > > > > >> >
> > > > > > > > >> > My solution: add a swap file. Not ideal in terms of
> > > > performance,
> > > > > > but
> > > > > > > > it
> > > > > > > > >> > gets the whole thing going... at least mostly (please
> read
> > > > on).
> > > > > > > > >> >
> > > > > > > > >> > Current status:
> > > > > > > > >> >
> > > > > > > > >> > - Cassandra (and all other Docker containers) are
> still
> > > > > > running;
> > > > > > > > >> > Cassandra died in my previous attempts at the latest
> > when
> > > > the
> > > > > > > > Teller
> > > > > > > > >> > application started, but with the swap file it's
> still
> > > > > running
> > > > > > > > >> > - all Spring Boot apps are starting now
> > > > > > > > >> > - I get once in a while some spurious timeouts (I
> think
> > > > when
> > > > > > > either
> > > > > > > > >> the
> > > > > > > > >> > Eureka server is contacted or the config server)
> > > > > > > > >> > - there are also sometimes exceptions concerning
> MySQL
> > > > > > > connections,
> > > > > > > > >> but
> > > > > > > > >> > doesn't seem to matter overall
> > > > > > > > >> > - I could issue an authentication request with
> Postman
> > to
> > > > > > > > >> > http://fineract-vm.apache.org:2021/identity/v1; just
> > to
> > > > test
> > > > > > if
> > > > > > > > >> > anything
> > > > > > > > >> > is responding
> > > > > > > > >> > - PROBLEM: but the demo server (the Spring Boot apps)
> > > > > > eventually
> > > > > > > > dies
> > > > > > > > >> > after a couple of minutes; I don't think that the
> > memory
> > > is
> > > > > the
> > > > > > > > >> problem
> > > > > > > > >> > at
> > > > > > > > >> > this point; I think it's more of a timeout problem
> > > > concerning
> > > > > > the
> > > > > > > > >> > communication with Eureka and/or the config server
> > (maybe
> > > > > > because
> > > > > > > > the
> > > > > > > > >> > system is a bit slow)
> > > > > > > > >> >
> > > > > > > > >> > ... and here the memory footprint ("free -h") when
> > > everything
> > > > is
> > > > > > > > >> running,
> > > > > > > > >> > just FYI:
> > > > > > > > >> >
> > > > > > > > >> > total used free shared
> > > > > > buff/cache
> > > > > > > > >> > available
> > > > > > > > >> > Mem: 31G 31G 226M 8.4M
> > > > > > 169M
> > > > > > > > >> > 47M
> > > > > > > > >> > Swap: 15G 6.8G 9.2G
> > > > > > > > >> >
> > > > > > > > >> > ... after the Payroll application is started (approx.) I
> > see
> > > > > these
> > > > > > > > >> > exceptions appearing in the logs (a whole series of
> them):
> > > > > > > > >> >
> > > > > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > > endpoints
> > > > > > via
> > > > > > > > >> > configuration
> > > > > > > > >> > Resolving artifact
> > > > > > > > >> >
> > > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > > > >> > Resolving metadata
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > > > >> > Resolved metadata
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > > > >> > Resolved artifact
> > > > > > > > >> >
> > > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > > > from
> > > > > > > > >> > /root/.m2/repository (enhanced)
> > > > > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
> > > > > > > > >> c.n.discovery.TimedSupervisorTask -
> > > > > > > > >> > task supervisor timed out
> > > > > > > > >> > java.util.concurrent.TimeoutException: null
> > > > > > > > >> > at
> > java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > > > > >> > at
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > > > > >> > at
> > > > > > > > >>
> > > > > > >
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > > >> > at
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > > >> > at
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > > > >> > at
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > > > >> > at
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > > >> > at
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > > >> > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > >> >
> > > > > > > > >> > ... but it still seems to have no problem starting the
> > next
> > > > app
> > > > > > > > (Group)
> > > > > > > > >> and
> > > > > > > > >> > things look good for a while...
> > > > > > > > >> >
> > > > > > > > >> > ... until suddenly it seems that app instances are
> > beginning
> > > > to
> > > > > > > > >> shutdown...
> > > > > > > > >> > and I'm not sure why this happens:
> > > > > > > > >> >
> > > > > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > > endpoints
> > > > > > via
> > > > > > > > >> > configuration
> > > > > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
> > > > > > > > >> c.datastax.driver.core.ClockFactory -
> > > > > > > > >> > Using native clock to generate timestamps.
> > > > > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
> > > > > > > > >> c.datastax.driver.core.CodecRegistry
> > > > > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
> > > > > > > > >> java.time.LocalDateTime]
> > > > > > > > >> > because it collides with previously registered codec
> > > > > > > > LocalDateTimeCodec
> > > > > > > > >> > [timestamp <-> java.time.LocalDateTime]
> > > > > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
> > > > > > > > c.d.d.c.p.DCAwareRoundRobinPolicy
> > > > > > > > >> -
> > > > > > > > >> > Using data-center name 'datacenter1' for
> > > > DCAwareRoundRobinPolicy
> > > > > > (if
> > > > > > > > >> this
> > > > > > > > >> > is incorrect, please provide the correct datacenter name
> > > with
> > > > > > > > >> > DCAwareRoundRobinPolicy constructor)
> > > > > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
> > > > > > > > com.datastax.driver.core.Cluster -
> > > > > > > > >> > New Cassandra host /127.0.0.1:9042 added
> > > > > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > > endpoints
> > > > > > via
> > > > > > > > >> > configuration
> > > > > > > > >> > 18:27:51.110 [Thread-15] INFO
> > > > > > > > >> > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext
> -
> > > > > Closing
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > > > >> > :
> > > > > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > > endpoints
> > > > > > via
> > > > > > > > >> > configuration
> > > > > > > > >> > 18:27:53.373 [Thread-15] INFO
> > > > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > > > >> Stopping
> > > > > > > > >> > beans in phase 2147483647
> > > > > > > > >> > 18:27:54.331 [Thread-15] INFO
> > > > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > > > >> Stopping
> > > > > > > > >> > beans in phase 0
> > > > > > > > >> > 18:27:54.736 [Thread-15] INFO
> > > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > > > >> > 18:27:54.740 [Thread-15] INFO
> > > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > > > >> > Unregistering JMX-exposed beans
> > > > > > > > >> > 18:27:54.798 [Thread-15] INFO
> > > > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > > > >> > 18:27:54.800 [Thread-15] INFO
> > > > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > > > >> > Unregistering JMX-exposed beans
> > > > > > > > >> > 18:27:54.937 [Thread-15] INFO
> > > > > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean - Closing
> > JPA
> > > > > > > > >> > EntityManagerFactory for persistence unit 'metaPU'
> > > > > > > > >> > 18:27:58.376 [async-processor-1] INFO
> > > > > > > > >> o.f.c.internal.util.VersionPrinter -
> > > > > > > > >> > Flyway 3.2.1 by Boxfuse
> > > > > > > > >> > 18:27:58.677 [async-processor-1] INFO
> > > > > > > > >> o.f.c.i.dbsupport.DbSupportFactory -
> > > > > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL
> 10.3)
> > > > > > > > >> > 18:28:00.620 [async-processor-1] INFO
> > > > > > > > >> o.f.core.internal.command.DbValidate
> > > > > > > > >> > - Validated 2 migrations (execution time 00:01.169s)
> > > > > > > > >> > 18:28:00.901 [Thread-15] INFO
> > > > > o.e.jetty.server.AbstractConnector
> > > > > > -
> > > > > > > > >> Stopped
> > > > > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{
> 0.0.0.0:2020
> > }
> > > > > > > > >> > 18:28:01.268 [Thread-15] INFO
> > > > > > o.e.j.server.handler.ContextHandler -
> > > > > > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
> > > > > > > > >> >
> > > > > > > > >> > ... the list is actually a lot longer... in the rest of
> > the
> > > > log
> > > > > > you
> > > > > > > > can
> > > > > > > > >> see
> > > > > > > > >> > that one by one all the apps are shutting down.
> > > > > > > > >> >
> > > > > > > > >> > The last (error) entries I see are these:
> > > > > > > > >> >
> > > > > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR
> > > > > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not
> > refresh
> > > > JMS
> > > > > > > > >> Connection
> > > > > > > > >> > for destination 'cheques-v1' - retrying using
> > > > > > > > >> FixedBackOff{interval=5000,
> > > > > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error
> > > while
> > > > > > > > >> attempting to
> > > > > > > > >> > retrieve a connection from the pool; nested exception is
> > > > > > > > >> > javax.jms.JMSException: Could not connect to broker URL:
> > > > > > > > >> > tcp://localhost:61616. Reason:
> java.net.ConnectException:
> > > > > > Connection
> > > > > > > > >> > refused (Connection refused)
> > > > > > > > >> >
> > > > > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in
> the
> > > > > process
> > > > > > of
> > > > > > > > the
> > > > > > > > >> > previous errors.
> > > > > > > > >> >
> > > > > > > > >> > So far from here... any input on the above highly
> > > appreciated.
> > > > > > > > Overall I
> > > > > > > > >> > think we are close.
> > > > > > > > >> >
> > > > > > > > >> > Cheers,
> > > > > > > > >> >
> > > > > > > > >> > Aleks
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <
> > > mage@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > One thing we maybe can do, given we are part of the
> > Apache
> > > > > > Family,
> > > > > > > > is
> > > > > > > > >> to
> > > > > > > > >> > > ask the Cassandra community for some suggestions.
> > > > > > > > >> > >
> > > > > > > > >> > > Cheers
> > > > > > > > >> > >
> > > > > > > > >> > > Markus
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <
> > > > mage@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Hey all,
> > > > > > > > >> > > >
> > > > > > > > >> > > > we are not running Cassandra in a container, we have
> > > > > dedicated
> > > > > > > VMs
> > > > > > > > >> to
> > > > > > > > >> > run
> > > > > > > > >> > > > our cluster.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Given this we are not the best persons to help here,
> > > > sorry.
> > > > > (;
> > > > > > > > >> > > >
> > > > > > > > >> > > > Cheers
> > > > > > > > >> > > >
> > > > > > > > >> > > > Markus
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
> > > > > edcable@mifos.org>
> > > > > > > > >> wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > >> Adding Patric directly to this thread too so he can
> > > give
> > > > > his
> > > > > > > > input
> > > > > > > > >> > > >> regarding to the challenges with Cassandra.
> > > > > > > > >> > > >>
> > > > > > > > >> > > >> Ed
> > > > > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar
> > Vidakovic <
> > > > > > > > >> > > >> cheetah@monkeysintown.com> wrote:
> > > > > > > > >> > > >>
> > > > > > > > >> > > >>> Hi Victor,
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> .... this is the relevant part of the
> > > docker-compose.yml
> > > > > > file:
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> cassandra0:
> > > > > > > > >> > > >>> image: cassandra:3.11.1
> > > > > > > > >> > > >>> container_name: cassandra0
> > > > > > > > >> > > >>> ports:
> > > > > > > > >> > > >>> - 9042:9042
> > > > > > > > >> > > >>> - 9160:9160
> > > > > > > > >> > > >>> - 7199:7199
> > > > > > > > >> > > >>> - 8778:8778
> > > > > > > > >> > > >>> volumes:
> > > > > > > > >> > > >>> - ./cassandra:/etc/cassandra
> > > > > > > > >> > > >>> environment:
> > > > > > > > >> > > >>> - CASSANDRA_START_RPC=true
> > > > > > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0
> > > > > > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster
> > > > > > > > >> > > >>> ulimits:
> > > > > > > > >> > > >>> memlock: -1
> > > > > > > > >> > > >>> nproc: 32768
> > > > > > > > >> > > >>> nofile: 100000
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> ... and just to be complete... here's the Docker
> > > service
> > > > > > > > >> > configuration
> > > > > > > > >> > > >>> ("/lib/systemd/system/docker.service"):
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> [Unit]
> > > > > > > > >> > > >>> Description=Docker Application Container Engine
> > > > > > > > >> > > >>> Documentation=https://docs.docker.com
> > > > > > > > >> > > >>> After=network-online.target docker.socket
> > > > > firewalld.service
> > > > > > > > >> > > >>> Wants=network-online.target
> > > > > > > > >> > > >>> Requires=docker.socket
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> [Service]
> > > > > > > > >> > > >>> Type=notify
> > > > > > > > >> > > >>> # the default is not to use systemd for cgroups
> > > because
> > > > > the
> > > > > > > > >> delegate
> > > > > > > > >> > > >>> issues
> > > > > > > > >> > > >>> still
> > > > > > > > >> > > >>> # exists and systemd currently does not support
> the
> > > > cgroup
> > > > > > > > feature
> > > > > > > > >> > set
> > > > > > > > >> > > >>> required
> > > > > > > > >> > > >>> # for containers run by docker
> > > > > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
> > > > > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
> > > > > > > > >> > > >>> LimitNOFILE=1048576
> > > > > > > > >> > > >>> # Having non-zero Limit*s causes performance
> > problems
> > > > due
> > > > > to
> > > > > > > > >> > accounting
> > > > > > > > >> > > >>> overhead
> > > > > > > > >> > > >>> # in the kernel. We recommend using cgroups to do
> > > > > > > > container-local
> > > > > > > > >> > > >>> accounting.
> > > > > > > > >> > > >>> LimitNPROC=infinity
> > > > > > > > >> > > >>> LimitCORE=infinity
> > > > > > > > >> > > >>> LimitMEMLOCK=infinity
> > > > > > > > >> > > >>> # Uncomment TasksMax if your systemd version
> > supports
> > > > it.
> > > > > > > > >> > > >>> # Only systemd 226 and above support this version.
> > > > > > > > >> > > >>> TasksMax=infinity
> > > > > > > > >> > > >>> TimeoutStartSec=0
> > > > > > > > >> > > >>> # set delegate yes so that systemd does not reset
> > the
> > > > > > cgroups
> > > > > > > of
> > > > > > > > >> > docker
> > > > > > > > >> > > >>> containers
> > > > > > > > >> > > >>> Delegate=yes
> > > > > > > > >> > > >>> # kill only the docker process, not all processes
> in
> > > the
> > > > > > > cgroup
> > > > > > > > >> > > >>> KillMode=process
> > > > > > > > >> > > >>> # restart the docker process if it exits
> prematurely
> > > > > > > > >> > > >>> Restart=on-failure
> > > > > > > > >> > > >>> StartLimitBurst=3
> > > > > > > > >> > > >>> StartLimitInterval=60s
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> [Install]
> > > > > > > > >> > > >>> WantedBy=multi-user.target
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> ... the one tweak I did there is to set
> LimitMEMLOCK
> > > to
> > > > > > > > >> infinity...
> > > > > > > > >> > do
> > > > > > > > >> > > >>> you
> > > > > > > > >> > > >>> think the problems are file handle related?
> Should I
> > > set
> > > > > > > > >> LimitNOFILE
> > > > > > > > >> > > also
> > > > > > > > >> > > >>> to infinity?
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> Appreciate the help.
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> Cheers,
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> Aleks
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
> > > > > > > > >> > > >>> victor.romero@fintecheando.mx>
> > > > > > > > >> > > >>> wrote:
> > > > > > > > >> > > >>>
> > > > > > > > >> > > >>> > Hi Aleks,
> > > > > > > > >> > > >>> >
> > > > > > > > >> > > >>> > Can you share the ulimits flags that the
> > cassandra's
> > > > > > > container
> > > > > > > > >> is
> > > > > > > > >> > > using
> > > > > > > > >> > > >>> > while it is running?
> > > > > > > > >> > > >>> >
> > > > > > > > >> > > >>> > The values are being set in the composer file?
> Or
> > in
> > > > the
> > > > > > > > >> upstart or
> > > > > > > > >> > > >>> > systemd docker's deamon config files?
> > > > > > > > >> > > >>> >
> > > > > > > > >> > > >>> >
> > > > > > > > >> > > >>> >
> > > > > > > > >> > > >>> > Enviado desde TypeApp
> > > > > > > > >> > > >>> >
> > > > > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM,
> Aleksandar
> > > > > > > Vidakovic <
> > > > > > > > >> > > >>> > cheetah@monkeysintown.com> escribió:
> > > > > > > > >> > > >>> > >... and I should have attached the Cassandra
> log
> > > > > dump...
> > > > > > so
> > > > > > > > >> here
> > > > > > > > >> > it
> > > > > > > > >> > > >>> > >is...
> > > > > > > > >> > > >>> > >
> > > > > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar
> > > Vidakovic
> > > > <
> > > > > > > > >> > > >>> > >cheetah@monkeysintown.com> wrote:
> > > > > > > > >> > > >>> > >
> > > > > > > > >> > > >>> > >> Hi all,
> > > > > > > > >> > > >>> > >>
> > > > > > > > >> > > >>> > >> ... as you might have noticed I did a couple
> of
> > > > > > restarts
> > > > > > > > >> > today...
> > > > > > > > >> > > >>> the
> > > > > > > > >> > > >>> > >> problem I am facing now: I can't get
> Cassandra
> > to
> > > > run
> > > > > > in
> > > > > > > a
> > > > > > > > >> > stable
> > > > > > > > >> > > >>> > >way.
> > > > > > > > >> > > >>> > >>
> > > > > > > > >> > > >>> > >> Things look quite OK for a while when running
> > the
> > > > > demo
> > > > > > > > >> server,
> > > > > > > > >> > but
> > > > > > > > >> > > >>> > >then
> > > > > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts
> > > > > > misbehaving
> > > > > > > > with
> > > > > > > > >> > the
> > > > > > > > >> > > >>> > >deposit
> > > > > > > > >> > > >>> > >> microservice startup, sometimes with
> portfolio
> > or
> > > > > > > teller).
> > > > > > > > I
> > > > > > > > >> > tried
> > > > > > > > >> > > >>> to
> > > > > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set
> > some
> > > > > > Docker
> > > > > > > > >> limits
> > > > > > > > >> > to
> > > > > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
> > > > > > > > >> > > >>> > >>
> > > > > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe
> > > someone
> > > > > can
> > > > > > > > help
> > > > > > > > >> out
> > > > > > > > >> > > >>> > >here? Is
> > > > > > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB
> of
> > > > > memory?
> > > > > > > > >> > > >>> > >>
> > > > > > > > >> > > >>> > >> Other than that the setup would be ready to
> > go...
> > > > we
> > > > > > just
> > > > > > > > >> need
> > > > > > > > >> > to
> > > > > > > > >> > > >>> get
> > > > > > > > >> > > >>> > >the
> > > > > > > > >> > > >>> > >> database running more reliably.
> > > > > > > > >> > > >>> > >>
> > > > > > > > >> > > >>> > >> Cheers,
> > > > > > > > >> > > >>> > >>
> > > > > > > > >> > > >>> > >> Aleks
> > > > > > > > >> > > >>> > >>
> > > > > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar
> > > > Vidakovic
> > > > > <
> > > > > > > > >> > > >>> > >> cheetah@monkeysintown.com> wrote:
> > > > > > > > >> > > >>> > >>
> > > > > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed to
> > add
> > > > > some
> > > > > > > > >> > additional
> > > > > > > > >> > > >>> > >reverse
> > > > > > > > >> > > >>> > >>> proxy configuration for the web UI to
> work)...
> > > and
> > > > > > there
> > > > > > > > >> were
> > > > > > > > >> > > more
> > > > > > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
> > > > > > > > >> > > >>> > >>>
> > > > > > > > >> > > >>> > >>> Just FYI
> > > > > > > > >> > > >>> > >>>
> > > > > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
> > > > > Vidakovic <
> > > > > > > > >> > > >>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > > > > >> > > >>> > >>>
> > > > > > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can
> > get
> > > > back
> > > > > > to
> > > > > > > > >> testing
> > > > > > > > >> > > >>> only
> > > > > > > > >> > > >>> > >>>> later tonight.
> > > > > > > > >> > > >>> > >>>>
> > > > > > > > >> > > >>> > >>>> Cheers
> > > > > > > > >> > > >>> > >>>>
> > > > > > > > >> > > >>> > >>>>
> > > > > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya
> > > > > > > > >> > > >>> > ><ra...@gmail.com>
> > > > > > > > >> > > >>> > >>>> wrote:
> > > > > > > > >> > > >>> > >>>>
> > > > > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test after
> > 30
> > > > min.
> > > > > > > > >> > > >>> > >>>>>
> > > > > > > > >> > > >>> > >>>>> Big thanks for this 🙂
> > > > > > > > >> > > >>> > >>>>>
> > > > > > > > >> > > >>> > >>>>>
> > > > > > > > >> > > >>> > >>>>>
> > > > > > > > >> > > >>> > >>>>>
> > > > > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM Aleksandar
> > > > > > Vidakovic <
> > > > > > > > >> > > >>> > >>>>> cheetah@monkeysintown.com> wrote:
> > > > > > > > >> > > >>> > >>>>>
> > > > > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted.
> > > > > > > > >> > > >>> > >>>>> >
> > > > > > > > >> > > >>> > >>>>> > ... and as I said: the services are
> still
> > > > > > > starting...
> > > > > > > > >> and
> > > > > > > > >> > the
> > > > > > > > >> > > >>> > >best
> > > > > > > > >> > > >>> > >>>>> estimate
> > > > > > > > >> > > >>> > >>>>> > I have right now (as already mentioned):
> > > 30min
> > > > > > > > >> > > >>> > >>>>> >
> > > > > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan
> > > Maurya <
> > > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > > > > >> > > >>> > >>>>> > wrote:
> > > > > > > > >> > > >>> > >>>>> >
> > > > > > > > >> > > >>> > >>>>> > > [image: image.png]
> > > > > > > > >> > > >>> > >>>>> > >
> > > > > > > > >> > > >>> > >>>>> > >
> > > > > > > > >> > > >>> > >>>>> > >
> > > > > > > > >> > > >>> > >>>>> > >
> > > > > > > > >> > > >>> > >>>>> > >
> > > > > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan
> > > > Maurya <
> > > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > > > > >> > > >>> > >>>>> > > wrote:
> > > > > > > > >> > > >>> > >>>>> > >
> > > > > > > > >> > > >>> > >>>>> > >> [image: image.png]
> > > > > > > > >> > > >>> > >>>>> > >> I am getting this.
> > > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM
> > > Aleksandar
> > > > > > > > Vidakovic
> > > > > > > > >> <
> > > > > > > > >> > > >>> > >>>>> > >> cheetah@monkeysintown.com> wrote:
> > > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > > >> > > >>> > >>>>> > >>> Hi all,
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost)
> > > ready...
> > > > > it
> > > > > > > took
> > > > > > > > >> me a
> > > > > > > > >> > > >>> > >moment
> > > > > > > > >> > > >>> > >>>>> and a
> > > > > > > > >> > > >>> > >>>>> > >>> couple
> > > > > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot
> > > > > > failures...
> > > > > > > > the
> > > > > > > > >> > > >>> services
> > > > > > > > >> > > >>> > >are
> > > > > > > > >> > > >>> > >>>>> quite
> > > > > > > > >> > > >>> > >>>>> > >>> resource hungry and default settings
> > > won't
> > > > > do
> > > > > > > it.
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> The services are still starting and
> > this
> > > > > will
> > > > > > > > take a
> > > > > > > > >> > > while
> > > > > > > > >> > > >>> > >(my
> > > > > > > > >> > > >>> > >>>>> best
> > > > > > > > >> > > >>> > >>>>> > guess
> > > > > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org/login
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> Credentials:
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> Tenant : playground
> > > > > > > > >> > > >>> > >>>>> > >>> Username : operator
> > > > > > > > >> > > >>> > >>>>> > >>> Password : init1@l
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to
> > > compile
> > > > > this
> > > > > > > app
> > > > > > > > >> is
> > > > > > > > >> > > with
> > > > > > > > >> > > >>> > >NodeJS
> > > > > > > > >> > > >>> > >>>>> > 8.11.1
> > > > > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't
> > > work).
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> Identity Service:
> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2021/identity/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Office Service:
> > > > > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Customer Service:
> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2024/customer/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Accounting Service:
> > > > > > > > >> > > >>> > >>>>>
> > > > http://fineract-vm.apache.org:2025/accounting/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Portfolio Service:
> > > > > > > > >> > > >>> > >>>>>
> > > http://fineract-vm.apache.org:2026/portfolio/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Deposit Service:
> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Teller Service:
> > > > > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Reporting Service:
> > > > > > > > >> > > >>> > >>>>>
> > > http://fineract-vm.apache.org:2029/reporting/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Cheque Service:
> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1
> > > > > > > > >> > > >>> > >>>>> > >>> Payroll Service:
> > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes
> > > quite
> > > > a
> > > > > > > > >> while... if
> > > > > > > > >> > > you
> > > > > > > > >> > > >>> > >>>>> encounter
> > > > > > > > >> > > >>> > >>>>> > >>> connection problems retry a couple
> of
> > > > > minutes
> > > > > > > > later
> > > > > > > > >> to
> > > > > > > > >> > > >>> > >ensure I
> > > > > > > > >> > > >>> > >>>>> am not
> > > > > > > > >> > > >>> > >>>>> > >>> currently working on something.
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the
> > backend
> > > > > > services
> > > > > > > > >> once
> > > > > > > > >> > per
> > > > > > > > >> > > >>> > >day or
> > > > > > > > >> > > >>> > >>>>> so to
> > > > > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to
> > > figure
> > > > > out
> > > > > > > how
> > > > > > > > we
> > > > > > > > >> > want
> > > > > > > > >> > > >>> to
> > > > > > > > >> > > >>> > >>>>> handle
> > > > > > > > >> > > >>> > >>>>> > this
> > > > > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where to
> > put
> > > > > this
> > > > > > > demo
> > > > > > > > >> > server
> > > > > > > > >> > > >>> > >>>>> configuration
> > > > > > > > >> > > >>> > >>>>> > >>> then let me know here.
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any
> > > problems
> > > > (I
> > > > > > > have
> > > > > > > > >> not
> > > > > > > > >> > > >>> > >extensively
> > > > > > > > >> > > >>> > >>>>> > tested
> > > > > > > > >> > > >>> > >>>>> > >>> it
> > > > > > > > >> > > >>> > >>>>> > >>> yet).
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> Cheers,
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> Aleks
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM
> > > > Aleksandar
> > > > > > > > >> Vidakovic <
> > > > > > > > >> > > >>> > >>>>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem to
> be
> > > > > > starting
> > > > > > > > fine
> > > > > > > > >> > > now...
> > > > > > > > >> > > >>> > >just
> > > > > > > > >> > > >>> > >>>>> have
> > > > > > > > >> > >
>
--
*Ed Cable*
President/CEO, Mifos Initiative
edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
*Collectively Creating a World of 3 Billion Maries | *http://mifos.org
<http://facebook.com/mifos> <http://www.twitter.com/mifos>
Re: Status of Demo Server on Apache VM Infrastructure
Posted by Myrle Krantz <my...@apache.org>.
Hi Avik,
You've asked several questions here, which is why it took me a while to get
to answering.
Before I go on, allow me to clarify my use of certain terms in the context
of Fineract CN:
* unit test -- a test for just one or two classes. Doesn't require
spinning up the service. Doesn't require resources like a DB or user
interaction. Is fast enough to be made part of the build. These tests are
included in the same module and package as the class which they are
intended to test. (This is a definition which is generally agreed upon in
the industry.)
* component test -- a test for a service. Requires spinning up just that
service. Requires the use of an embedded DB. Individual tests might be
fast, but the testsuite is rather expensive to run because startup times
are impacted by running two DBs and spinning up the service. These tests
are included in the same repository as the service they are intended to
test. External services are mocked. (The industry has no widely agreed
upon definition for component tests; this is my definition.)
* integration test -- a test for multiple services. These tests have their
own repositories.
demo-server serves three purposes:
* It illustrates the steps need to start and provision the services.
* It serves as an integration test for one possible the provisioning
workflow.
* It makes it possible to start up the services locally so that a UI
developer can program against the latest changes.
demo-server is not intended to be a used in production for multiple reasons:
* It's slow. It starts everything, even stuff you may not need, and it
provisions everything. It then waits for everything to register itself
with Eureka. It lowers the times for the eureka server to speed up
startup, but this results in to many cycles being used for Eureka updates.
* It's not secure. For example: identity.token.refresh.secureCookie is set
to false to make http transport possible.
If you're creating a new service, I don't expect that you'll need to run
demo-server very often. Your emphasis should be on your component tests.
These can run in just a minute or two, and you don't have to worry (much)
about JWT tokens, headers, and the like inside of these tests.
If demo-server doesn't serve your purposes I see a few possibilities:
* create a private fork of the demo-server, then create the setup you
prefer.
* submit a PR with another "test" in demo-server which starts the services
you need. I'd be happy to help you with this.
* submit a PR which makes the starting of various services in the
demo-service optional. I'd be happy to help you with this as well.
One idea for speeding up local deployments (which is intended only to
enable developers and not to be a production deployment method) is to start
all the services in one process. This would reduce service startup times
by folding them all into one. It would also replace inter-process
communication during provisioning with in-process communication. I expect
that we could get startup under 5 minutes. It would also make it possible
to do component-testing on services which are dependent on other services
without extensive, error-prone mocking. Writing tests for portfolio
probably cost me more time than writing the code, and debugging mocks isn't
fun. But those benefits would only apply for development environments.
That's what FINCN-25 is about.
Rhythm isn't resource-hungry. Not sure where that came from...
I don't believe Cassandra is the problem here, but I haven't examined that
closely. Please prove me wrong. I'd love to see a fix like that, if it
really is that easy. : o)
Best Regards,
Myrle
On Wed, Oct 10, 2018 at 6:24 PM Avik Ganguly <av...@gmail.com>
wrote:
> Hi guys,
>
> I am assuming that the suggested form of development for Fineract CN is to
> write unit tests to test the code which you just wrote locally and then
> push the code to a remote cluster or a massive VM and wait 40+ minutes to
> run the API / integration tests there. A lot of the confusion regarding the
> stacktraces in identity, rhythm, timeouts and auto-shutdown might or might
> not have anything to do with the resource crunch in my local. (i7, 8GB,
> SSD)
>
> Maybe a FAQ section in "How to build Fineract CN" to help devs with local
> setup would go a long way to bring in contributions to the core modules.
>
> 1. Change demo server orchestration logic from running Identity,
> Organization, Customer, Accounting, Portfolio, Deposit, Teller, Reporting,
> Cheque, Payroll, Group and Notification services to running only Identity,
> Organization, Customer and working set of your choice (Portfolio and
> Accounting OR Deposit and Accounting OR Group and Portfolio)
>
> 2. Leave out Rhythm from orchestration logic for now. A JIRA ticket
> tracking ability of Rhythm to run with low resource footprint will probably
> help.
>
>
> @Myrle, can you elaborate on FINCN-25? Is the motivation of this ticket to
> reduce resource footprint only or also to reduce the startup time from 40
> minutes to a reasonable couple of minutes? Do you have any debugging tips
> with respect to the slow startup caused by services is due to registering
> with eureka or provisioning or sequential startup?
>
> Maybe we should also follow up in Cassandra mailing list regarding
> configuring Cassandra to run within 8GB of RAM and include the same config
> as part of the FAQ.
>
>
> Regards,
> Avik.
> ᐧ
>
> On Wed, Sep 19, 2018 at 2:23 AM Courage Angeh <co...@gmail.com>
> wrote:
>
> > Thanks for the update. Ed.
> >
> > If you need an extra hand please let me know Aleks.
> >
> > On Tue, Sep 18, 2018, 4:27 PM Ed Cable <ed...@mifos.org> wrote:
> >
> > > Hi all,
> > >
> > > Just to bring this thread back up to top of everyone's inbox. Aleks
> > > estimates he has about a day left to get the demo server up and running
> > is
> > > hoping to tackle it by tomorrow. Here are the two approaches he
> suggested
> > > now that we have the second VM allocated:
> > >
> > > - continue with the JUnit based startup sequence of the demo module;
> the
> > > one thing that I would need to change is to split up the startup
> sequence
> > > and recompile
> > > - create a set of Docker Compose files to do the startup on the 2 VM
> > > instances (maybe I could borrow some stuff from Courage)
> > >
> > > I would prefer the latter, because you have more fine grained control
> and
> > > it's easier move service instances around... and you don't have to
> > > recompile anything.
> > >
> > > Ed
> > >
> > > On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <my...@apache.org> wrote:
> > >
> > > > Hey Aleks,
> > > >
> > > > I don’t see anything in that log that would explain the behavior
> you’re
> > > > observing. Markus did most of the work on deposits. Markus? Do you
> have
> > > any
> > > > ideas?
> > > >
> > > > Best Regards,
> > > > Myrle
> > > >
> > > >
> > > >
> > > > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
> > > > cheetah@monkeysintown.com>
> > > > wrote:
> > > >
> > > > > @Myrle: with your proposed changes things look a bit better. I've
> > > > disabled
> > > > > Rhythm as you suggested and get past all microservice startups
> with a
> > > lot
> > > > > less exceptions (one is coming repeatedly from Netty native
> > transport,
> > > > > because epoll is not available).
> > > > >
> > > > > The one exception (actually it happened on 2 occasions) that is
> still
> > > > > there:
> > > > >
> > > > > [snip]
> > > > >
> > > > > 13:33:45.080 [main] INFO
> > > > o.a.f.c.d.s.DepositAccountManagementApplication -
> > > > > Started DepositAccountManagementApplication in 55.187 seconds (JVM
> > > > running
> > > > > for 62.772)
> > > > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> via
> > > > > configuration
> > > > > Resolving artifact
> > > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > Resolving metadata
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > from /root/.m2/repository (enhanced)
> > > > > Resolved metadata
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > from /root/.m2/repository (enhanced)
> > > > > Resolved artifact
> > > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > from
> > > > > /root/.m2/repository (enhanced)
> > > > > 13:34:56.800 [DiscoveryClient-0] ERROR
> > > c.n.discovery.TimedSupervisorTask
> > > > -
> > > > > task supervisor timed out
> > > > > java.util.concurrent.TimeoutException: null
> > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > at
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 13:34:56.801 [DiscoveryClient-1] ERROR
> > > c.n.discovery.TimedSupervisorTask
> > > > -
> > > > > task supervisor timed out
> > > > > java.util.concurrent.TimeoutException: null
> > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > at
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 13:34:56.916 [DiscoveryClient-2] ERROR
> > > c.n.discovery.TimedSupervisorTask
> > > > -
> > > > > task supervisor timed out
> > > > > java.util.concurrent.TimeoutException: null
> > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > at
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 13:35:08.586 [main] INFO
> o.s.c.a.AnnotationConfigApplicationContext
> > -
> > > > > Refreshing
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > :
> > > > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context
> > hierarchy
> > > > > 13:35:10.182 [main] INFO
> > > o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
> > > > -
> > > > > JSR-330 'javax.inject.Inject' annotation found and supported for
> > > > autowiring
> > > > > 13:35:10.371 [main] INFO
> > > > > o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker
> -
> > > Bean
> > > > > 'configurationPropertiesRebinderAutoConfiguration' of type [class
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
> > > > > is not eligible for getting processed by all BeanPostProcessors
> (for
> > > > > example: not eligible for auto-proxying)
> > > > > 13:35:10.435 [background-preinit] INFO
> > > > o.h.validator.internal.util.Version
> > > > > - HV000001: Hibernate Validator 5.3.0.Final
> > > > >
> > > > > [/snip]
> > > > >
> > > > > It looks like the whole startup procedure goes through and I can
> see
> > > some
> > > > > kind of "summary" that lists all REST endpoints. But it stays in
> that
> > > > state
> > > > > for only a minute or so and then a cascade of shutdowns are
> triggered
> > > > > again. The only thing that stays alive is Eureka which complains
> > about
> > > > the
> > > > > inaccessible services.
> > > > >
> > > > > Any idea what else we could do other than just ordering an
> additional
> > > > > machine?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Aleks
> > > > >
> > > > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <my...@apache.org>
> > wrote:
> > > > >
> > > > > > Hey Aleks,
> > > > > >
> > > > > > It looks like that exception is coming from Rhythm. I suspect
> it's
> > > not
> > > > > the
> > > > > > cause of your difficulties, since you say your other services are
> > > going
> > > > > > down too. You can run most of the Fineract server without
> Rhythm.
> > > You
> > > > > > could edit the demo-server script to leave it out for the
> purposes
> > of
> > > > > > testing. The consequence will be a lack of interest calculations
> > for
> > > > > > disbursed loans, but for finding out what the real problem is,
> it'd
> > > be
> > > > > good
> > > > > > to get this one out of the way.
> > > > > >
> > > > > > Best Regards,
> > > > > > Myrle
> > > > > >
> > > > > > (FYI: I've removed the non-list subscribers from the to. Most of
> > > them
> > > > > are
> > > > > > already subscribed, and those who aren't, don't want to be.)
> > > > > >
> > > > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
> > > > > > cheetah@monkeysintown.com> wrote:
> > > > > >
> > > > > > > I managed to further improve the startup procedure (aka less
> > > > exceptions
> > > > > > > than before)... there are still some timeouts, but not as many
> as
> > > > > before.
> > > > > > >
> > > > > > > The demo server still dies after this exception:
> > > > > > >
> > > > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > > > configuration
> > > > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > > > configuration
> > > > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > > > configuration
> > > > > > > 23:28:45.990 [pool-9-thread-1] ERROR
> > > > > > o.s.s.s.TaskUtils$LoggingErrorHandler
> > > > > > > - Unexpected error occurred in scheduled task.
> > > > > > >
> org.springframework.transaction.CannotCreateTransactionException:
> > > > Could
> > > > > > not
> > > > > > > open JPA EntityManager for transaction; nested exception is
> > > > > > > javax.persistence.PersistenceException:
> > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > > resultset:
> > > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > > Query is : set autocommit=0
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
> > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> > > > > > > at
> > > > > >
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > Caused by: javax.persistence.PersistenceException:
> > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > > resultset:
> > > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > > Query is : set autocommit=0
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
> > > > > > > at
> > > > > > >
> > > > >
> > >
> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
> > > > > > > ... 20 common frames omitted
> > > > > > > Caused by: java.sql.SQLNonTransientConnectionException: Could
> not
> > > > read
> > > > > > > resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > > Query is : set autocommit=0
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
> > > > > > > at
> > > > > > >
> > > > >
> > >
> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
> > > > > > > ... 22 common frames omitted
> > > > > > > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException:
> > Could
> > > > not
> > > > > > > read resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > > Query is : set autocommit=0
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
> > > > > > > ... 31 common frames omitted
> > > > > > > Caused by: java.io.EOFException: unexpected end of stream,
> read 0
> > > > bytes
> > > > > > > from 4
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
> > > > > > > ... 33 common frames omitted
> > > > > > > 23:29:05.500 [qtp6422064-20] INFO
> > > > o.a.f.c.l.c.ServiceExceptionFilter -
> > > > > > > Responding with a service error ServiceError{code=409,
> > > > > > message='Application
> > > > > > > identity-v1 already exists!'}
> > > > > > > 23:29:21.282 [Thread-15] INFO
> > > > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> Closing
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > > :
> > > > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > > 23:29:23.717 [Thread-15] INFO
> o.s.c.s.DefaultLifecycleProcessor
> > -
> > > > > > Stopping
> > > > > > > beans in phase 2147483647
> > > > > > > 23:29:24.267 [Thread-15] INFO
> o.s.c.s.DefaultLifecycleProcessor
> > -
> > > > > > Stopping
> > > > > > > beans in phase 0
> > > > > > > 23:29:24.557 [Thread-15] INFO
> > o.s.b.a.e.jmx.EndpointMBeanExporter
> > > -
> > > > > > > Unregistering JMX-exposed beans on shutdown
> > > > > > > 23:29:24.579 [Thread-15] INFO
> > o.s.b.a.e.jmx.EndpointMBeanExporter
> > > -
> > > > > > > Unregistering JMX-exposed beans
> > > > > > >
> > > > > > > The only thing I've left: start the databases (aka Cassandra)
> on
> > a
> > > > > > separate
> > > > > > > machine (e. g. on Digital Ocean) and see how this is working
> > then.
> > > > > > >
> > > > > > > ... FYI.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Aleks
> > > > > > >
> > > > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
> > > > > > > cheetah@monkeysintown.com> wrote:
> > > > > > >
> > > > > > > > @Rajan: it's not running, because the demo server dies... See
> > > logs
> > > > > from
> > > > > > > > previous message...
> > > > > > > >
> > > > > > > > We just have to figure that one out and we should be good to
> > go.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
> > > > rajanmaurya154@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks, Alex,
> > > > > > > >>
> > > > > > > >> I can see web app is running but I can't log in with the
> > > > credentials
> > > > > > > that
> > > > > > > >> you shared earlier
> > > > > > > >>
> > > > > > > >> Tenant : playground
> > > > > > > >> Username : operator
> > > > > > > >> Password : init1@l
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >>
> > > > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
> > > > > > > >> cheetah@monkeysintown.com> wrote:
> > > > > > > >>
> > > > > > > >> > Hi all,
> > > > > > > >> >
> > > > > > > >> > ... sweeeet! I got a considerable step further.
> > > > > > > >> >
> > > > > > > >> > @Markus: thanks for the conversation... gave me an idea
> how
> > to
> > > > fix
> > > > > > > (most
> > > > > > > >> > of) it!
> > > > > > > >> >
> > > > > > > >> > The problem was - as suspected - the memory (or lack of).
> > The
> > > VM
> > > > > > > >> instance
> > > > > > > >> > at Apache has net 30G available. 16G are needed for
> > Cassandra
> > > > and
> > > > > in
> > > > > > > my
> > > > > > > >> > previous tests I tried to run the demo server with up to
> > 16G.
> > > > The
> > > > > > > >> problem
> > > > > > > >> > is that the demo server needs considerably more memory
> than
> > > 16G.
> > > > > > > >> According
> > > > > > > >> > to my observation it's around 23-24G. Even if you limit
> the
> > > demo
> > > > > > > server
> > > > > > > >> > memory allocation with "-Xmx" to 12G (for example) it
> would
> > > just
> > > > > > > >> continue
> > > > > > > >> > to consume more memory.
> > > > > > > >> >
> > > > > > > >> > My solution: add a swap file. Not ideal in terms of
> > > performance,
> > > > > but
> > > > > > > it
> > > > > > > >> > gets the whole thing going... at least mostly (please read
> > > on).
> > > > > > > >> >
> > > > > > > >> > Current status:
> > > > > > > >> >
> > > > > > > >> > - Cassandra (and all other Docker containers) are still
> > > > > running;
> > > > > > > >> > Cassandra died in my previous attempts at the latest
> when
> > > the
> > > > > > > Teller
> > > > > > > >> > application started, but with the swap file it's still
> > > > running
> > > > > > > >> > - all Spring Boot apps are starting now
> > > > > > > >> > - I get once in a while some spurious timeouts (I think
> > > when
> > > > > > either
> > > > > > > >> the
> > > > > > > >> > Eureka server is contacted or the config server)
> > > > > > > >> > - there are also sometimes exceptions concerning MySQL
> > > > > > connections,
> > > > > > > >> but
> > > > > > > >> > doesn't seem to matter overall
> > > > > > > >> > - I could issue an authentication request with Postman
> to
> > > > > > > >> > http://fineract-vm.apache.org:2021/identity/v1; just
> to
> > > test
> > > > > if
> > > > > > > >> > anything
> > > > > > > >> > is responding
> > > > > > > >> > - PROBLEM: but the demo server (the Spring Boot apps)
> > > > > eventually
> > > > > > > dies
> > > > > > > >> > after a couple of minutes; I don't think that the
> memory
> > is
> > > > the
> > > > > > > >> problem
> > > > > > > >> > at
> > > > > > > >> > this point; I think it's more of a timeout problem
> > > concerning
> > > > > the
> > > > > > > >> > communication with Eureka and/or the config server
> (maybe
> > > > > because
> > > > > > > the
> > > > > > > >> > system is a bit slow)
> > > > > > > >> >
> > > > > > > >> > ... and here the memory footprint ("free -h") when
> > everything
> > > is
> > > > > > > >> running,
> > > > > > > >> > just FYI:
> > > > > > > >> >
> > > > > > > >> > total used free shared
> > > > > buff/cache
> > > > > > > >> > available
> > > > > > > >> > Mem: 31G 31G 226M 8.4M
> > > > > 169M
> > > > > > > >> > 47M
> > > > > > > >> > Swap: 15G 6.8G 9.2G
> > > > > > > >> >
> > > > > > > >> > ... after the Payroll application is started (approx.) I
> see
> > > > these
> > > > > > > >> > exceptions appearing in the logs (a whole series of them):
> > > > > > > >> >
> > > > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > endpoints
> > > > > via
> > > > > > > >> > configuration
> > > > > > > >> > Resolving artifact
> > > > > > > >> >
> > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > > >> > Resolving metadata
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > > >> > Resolved metadata
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > > >> > Resolved artifact
> > > > > > > >> >
> > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > > from
> > > > > > > >> > /root/.m2/repository (enhanced)
> > > > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
> > > > > > > >> c.n.discovery.TimedSupervisorTask -
> > > > > > > >> > task supervisor timed out
> > > > > > > >> > java.util.concurrent.TimeoutException: null
> > > > > > > >> > at
> java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > > > >> > at
> > > > > > > >>
> > > > > >
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > >> > at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > >> > at java.lang.Thread.run(Thread.java:748)
> > > > > > > >> >
> > > > > > > >> > ... but it still seems to have no problem starting the
> next
> > > app
> > > > > > > (Group)
> > > > > > > >> and
> > > > > > > >> > things look good for a while...
> > > > > > > >> >
> > > > > > > >> > ... until suddenly it seems that app instances are
> beginning
> > > to
> > > > > > > >> shutdown...
> > > > > > > >> > and I'm not sure why this happens:
> > > > > > > >> >
> > > > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > endpoints
> > > > > via
> > > > > > > >> > configuration
> > > > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
> > > > > > > >> c.datastax.driver.core.ClockFactory -
> > > > > > > >> > Using native clock to generate timestamps.
> > > > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
> > > > > > > >> c.datastax.driver.core.CodecRegistry
> > > > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
> > > > > > > >> java.time.LocalDateTime]
> > > > > > > >> > because it collides with previously registered codec
> > > > > > > LocalDateTimeCodec
> > > > > > > >> > [timestamp <-> java.time.LocalDateTime]
> > > > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
> > > > > > > c.d.d.c.p.DCAwareRoundRobinPolicy
> > > > > > > >> -
> > > > > > > >> > Using data-center name 'datacenter1' for
> > > DCAwareRoundRobinPolicy
> > > > > (if
> > > > > > > >> this
> > > > > > > >> > is incorrect, please provide the correct datacenter name
> > with
> > > > > > > >> > DCAwareRoundRobinPolicy constructor)
> > > > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
> > > > > > > com.datastax.driver.core.Cluster -
> > > > > > > >> > New Cassandra host /127.0.0.1:9042 added
> > > > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > endpoints
> > > > > via
> > > > > > > >> > configuration
> > > > > > > >> > 18:27:51.110 [Thread-15] INFO
> > > > > > > >> > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> > > > Closing
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > > >> > :
> > > > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > endpoints
> > > > > via
> > > > > > > >> > configuration
> > > > > > > >> > 18:27:53.373 [Thread-15] INFO
> > > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > > >> Stopping
> > > > > > > >> > beans in phase 2147483647
> > > > > > > >> > 18:27:54.331 [Thread-15] INFO
> > > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > > >> Stopping
> > > > > > > >> > beans in phase 0
> > > > > > > >> > 18:27:54.736 [Thread-15] INFO
> > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > > >> > 18:27:54.740 [Thread-15] INFO
> > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > > >> > Unregistering JMX-exposed beans
> > > > > > > >> > 18:27:54.798 [Thread-15] INFO
> > > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > > >> > 18:27:54.800 [Thread-15] INFO
> > > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > > >> > Unregistering JMX-exposed beans
> > > > > > > >> > 18:27:54.937 [Thread-15] INFO
> > > > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean - Closing
> JPA
> > > > > > > >> > EntityManagerFactory for persistence unit 'metaPU'
> > > > > > > >> > 18:27:58.376 [async-processor-1] INFO
> > > > > > > >> o.f.c.internal.util.VersionPrinter -
> > > > > > > >> > Flyway 3.2.1 by Boxfuse
> > > > > > > >> > 18:27:58.677 [async-processor-1] INFO
> > > > > > > >> o.f.c.i.dbsupport.DbSupportFactory -
> > > > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL 10.3)
> > > > > > > >> > 18:28:00.620 [async-processor-1] INFO
> > > > > > > >> o.f.core.internal.command.DbValidate
> > > > > > > >> > - Validated 2 migrations (execution time 00:01.169s)
> > > > > > > >> > 18:28:00.901 [Thread-15] INFO
> > > > o.e.jetty.server.AbstractConnector
> > > > > -
> > > > > > > >> Stopped
> > > > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{0.0.0.0:2020
> }
> > > > > > > >> > 18:28:01.268 [Thread-15] INFO
> > > > > o.e.j.server.handler.ContextHandler -
> > > > > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
> > > > > > > >> >
> > > > > > > >> > ... the list is actually a lot longer... in the rest of
> the
> > > log
> > > > > you
> > > > > > > can
> > > > > > > >> see
> > > > > > > >> > that one by one all the apps are shutting down.
> > > > > > > >> >
> > > > > > > >> > The last (error) entries I see are these:
> > > > > > > >> >
> > > > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR
> > > > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not
> refresh
> > > JMS
> > > > > > > >> Connection
> > > > > > > >> > for destination 'cheques-v1' - retrying using
> > > > > > > >> FixedBackOff{interval=5000,
> > > > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error
> > while
> > > > > > > >> attempting to
> > > > > > > >> > retrieve a connection from the pool; nested exception is
> > > > > > > >> > javax.jms.JMSException: Could not connect to broker URL:
> > > > > > > >> > tcp://localhost:61616. Reason: java.net.ConnectException:
> > > > > Connection
> > > > > > > >> > refused (Connection refused)
> > > > > > > >> >
> > > > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in the
> > > > process
> > > > > of
> > > > > > > the
> > > > > > > >> > previous errors.
> > > > > > > >> >
> > > > > > > >> > So far from here... any input on the above highly
> > appreciated.
> > > > > > > Overall I
> > > > > > > >> > think we are close.
> > > > > > > >> >
> > > > > > > >> > Cheers,
> > > > > > > >> >
> > > > > > > >> > Aleks
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <
> > mage@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > One thing we maybe can do, given we are part of the
> Apache
> > > > > Family,
> > > > > > > is
> > > > > > > >> to
> > > > > > > >> > > ask the Cassandra community for some suggestions.
> > > > > > > >> > >
> > > > > > > >> > > Cheers
> > > > > > > >> > >
> > > > > > > >> > > Markus
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <
> > > mage@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Hey all,
> > > > > > > >> > > >
> > > > > > > >> > > > we are not running Cassandra in a container, we have
> > > > dedicated
> > > > > > VMs
> > > > > > > >> to
> > > > > > > >> > run
> > > > > > > >> > > > our cluster.
> > > > > > > >> > > >
> > > > > > > >> > > > Given this we are not the best persons to help here,
> > > sorry.
> > > > (;
> > > > > > > >> > > >
> > > > > > > >> > > > Cheers
> > > > > > > >> > > >
> > > > > > > >> > > > Markus
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
> > > > edcable@mifos.org>
> > > > > > > >> wrote:
> > > > > > > >> > > >
> > > > > > > >> > > >> Adding Patric directly to this thread too so he can
> > give
> > > > his
> > > > > > > input
> > > > > > > >> > > >> regarding to the challenges with Cassandra.
> > > > > > > >> > > >>
> > > > > > > >> > > >> Ed
> > > > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar
> Vidakovic <
> > > > > > > >> > > >> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>
> > > > > > > >> > > >>> Hi Victor,
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> .... this is the relevant part of the
> > docker-compose.yml
> > > > > file:
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> cassandra0:
> > > > > > > >> > > >>> image: cassandra:3.11.1
> > > > > > > >> > > >>> container_name: cassandra0
> > > > > > > >> > > >>> ports:
> > > > > > > >> > > >>> - 9042:9042
> > > > > > > >> > > >>> - 9160:9160
> > > > > > > >> > > >>> - 7199:7199
> > > > > > > >> > > >>> - 8778:8778
> > > > > > > >> > > >>> volumes:
> > > > > > > >> > > >>> - ./cassandra:/etc/cassandra
> > > > > > > >> > > >>> environment:
> > > > > > > >> > > >>> - CASSANDRA_START_RPC=true
> > > > > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0
> > > > > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster
> > > > > > > >> > > >>> ulimits:
> > > > > > > >> > > >>> memlock: -1
> > > > > > > >> > > >>> nproc: 32768
> > > > > > > >> > > >>> nofile: 100000
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> ... and just to be complete... here's the Docker
> > service
> > > > > > > >> > configuration
> > > > > > > >> > > >>> ("/lib/systemd/system/docker.service"):
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> [Unit]
> > > > > > > >> > > >>> Description=Docker Application Container Engine
> > > > > > > >> > > >>> Documentation=https://docs.docker.com
> > > > > > > >> > > >>> After=network-online.target docker.socket
> > > > firewalld.service
> > > > > > > >> > > >>> Wants=network-online.target
> > > > > > > >> > > >>> Requires=docker.socket
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> [Service]
> > > > > > > >> > > >>> Type=notify
> > > > > > > >> > > >>> # the default is not to use systemd for cgroups
> > because
> > > > the
> > > > > > > >> delegate
> > > > > > > >> > > >>> issues
> > > > > > > >> > > >>> still
> > > > > > > >> > > >>> # exists and systemd currently does not support the
> > > cgroup
> > > > > > > feature
> > > > > > > >> > set
> > > > > > > >> > > >>> required
> > > > > > > >> > > >>> # for containers run by docker
> > > > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
> > > > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
> > > > > > > >> > > >>> LimitNOFILE=1048576
> > > > > > > >> > > >>> # Having non-zero Limit*s causes performance
> problems
> > > due
> > > > to
> > > > > > > >> > accounting
> > > > > > > >> > > >>> overhead
> > > > > > > >> > > >>> # in the kernel. We recommend using cgroups to do
> > > > > > > container-local
> > > > > > > >> > > >>> accounting.
> > > > > > > >> > > >>> LimitNPROC=infinity
> > > > > > > >> > > >>> LimitCORE=infinity
> > > > > > > >> > > >>> LimitMEMLOCK=infinity
> > > > > > > >> > > >>> # Uncomment TasksMax if your systemd version
> supports
> > > it.
> > > > > > > >> > > >>> # Only systemd 226 and above support this version.
> > > > > > > >> > > >>> TasksMax=infinity
> > > > > > > >> > > >>> TimeoutStartSec=0
> > > > > > > >> > > >>> # set delegate yes so that systemd does not reset
> the
> > > > > cgroups
> > > > > > of
> > > > > > > >> > docker
> > > > > > > >> > > >>> containers
> > > > > > > >> > > >>> Delegate=yes
> > > > > > > >> > > >>> # kill only the docker process, not all processes in
> > the
> > > > > > cgroup
> > > > > > > >> > > >>> KillMode=process
> > > > > > > >> > > >>> # restart the docker process if it exits prematurely
> > > > > > > >> > > >>> Restart=on-failure
> > > > > > > >> > > >>> StartLimitBurst=3
> > > > > > > >> > > >>> StartLimitInterval=60s
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> [Install]
> > > > > > > >> > > >>> WantedBy=multi-user.target
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> ... the one tweak I did there is to set LimitMEMLOCK
> > to
> > > > > > > >> infinity...
> > > > > > > >> > do
> > > > > > > >> > > >>> you
> > > > > > > >> > > >>> think the problems are file handle related? Should I
> > set
> > > > > > > >> LimitNOFILE
> > > > > > > >> > > also
> > > > > > > >> > > >>> to infinity?
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> Appreciate the help.
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> Cheers,
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> Aleks
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
> > > > > > > >> > > >>> victor.romero@fintecheando.mx>
> > > > > > > >> > > >>> wrote:
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> > Hi Aleks,
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> > Can you share the ulimits flags that the
> cassandra's
> > > > > > container
> > > > > > > >> is
> > > > > > > >> > > using
> > > > > > > >> > > >>> > while it is running?
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> > The values are being set in the composer file? Or
> in
> > > the
> > > > > > > >> upstart or
> > > > > > > >> > > >>> > systemd docker's deamon config files?
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> > Enviado desde TypeApp
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM, Aleksandar
> > > > > > Vidakovic <
> > > > > > > >> > > >>> > cheetah@monkeysintown.com> escribió:
> > > > > > > >> > > >>> > >... and I should have attached the Cassandra log
> > > > dump...
> > > > > so
> > > > > > > >> here
> > > > > > > >> > it
> > > > > > > >> > > >>> > >is...
> > > > > > > >> > > >>> > >
> > > > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar
> > Vidakovic
> > > <
> > > > > > > >> > > >>> > >cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >
> > > > > > > >> > > >>> > >> Hi all,
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> ... as you might have noticed I did a couple of
> > > > > restarts
> > > > > > > >> > today...
> > > > > > > >> > > >>> the
> > > > > > > >> > > >>> > >> problem I am facing now: I can't get Cassandra
> to
> > > run
> > > > > in
> > > > > > a
> > > > > > > >> > stable
> > > > > > > >> > > >>> > >way.
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> Things look quite OK for a while when running
> the
> > > > demo
> > > > > > > >> server,
> > > > > > > >> > but
> > > > > > > >> > > >>> > >then
> > > > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts
> > > > > misbehaving
> > > > > > > with
> > > > > > > >> > the
> > > > > > > >> > > >>> > >deposit
> > > > > > > >> > > >>> > >> microservice startup, sometimes with portfolio
> or
> > > > > > teller).
> > > > > > > I
> > > > > > > >> > tried
> > > > > > > >> > > >>> to
> > > > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set
> some
> > > > > Docker
> > > > > > > >> limits
> > > > > > > >> > to
> > > > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe
> > someone
> > > > can
> > > > > > > help
> > > > > > > >> out
> > > > > > > >> > > >>> > >here? Is
> > > > > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB of
> > > > memory?
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> Other than that the setup would be ready to
> go...
> > > we
> > > > > just
> > > > > > > >> need
> > > > > > > >> > to
> > > > > > > >> > > >>> get
> > > > > > > >> > > >>> > >the
> > > > > > > >> > > >>> > >> database running more reliably.
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> Cheers,
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> Aleks
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar
> > > Vidakovic
> > > > <
> > > > > > > >> > > >>> > >> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed to
> add
> > > > some
> > > > > > > >> > additional
> > > > > > > >> > > >>> > >reverse
> > > > > > > >> > > >>> > >>> proxy configuration for the web UI to work)...
> > and
> > > > > there
> > > > > > > >> were
> > > > > > > >> > > more
> > > > > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
> > > > > > > >> > > >>> > >>>
> > > > > > > >> > > >>> > >>> Just FYI
> > > > > > > >> > > >>> > >>>
> > > > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
> > > > Vidakovic <
> > > > > > > >> > > >>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>>
> > > > > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can
> get
> > > back
> > > > > to
> > > > > > > >> testing
> > > > > > > >> > > >>> only
> > > > > > > >> > > >>> > >>>> later tonight.
> > > > > > > >> > > >>> > >>>>
> > > > > > > >> > > >>> > >>>> Cheers
> > > > > > > >> > > >>> > >>>>
> > > > > > > >> > > >>> > >>>>
> > > > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya
> > > > > > > >> > > >>> > ><ra...@gmail.com>
> > > > > > > >> > > >>> > >>>> wrote:
> > > > > > > >> > > >>> > >>>>
> > > > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test after
> 30
> > > min.
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>> Big thanks for this 🙂
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM Aleksandar
> > > > > Vidakovic <
> > > > > > > >> > > >>> > >>>>> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted.
> > > > > > > >> > > >>> > >>>>> >
> > > > > > > >> > > >>> > >>>>> > ... and as I said: the services are still
> > > > > > starting...
> > > > > > > >> and
> > > > > > > >> > the
> > > > > > > >> > > >>> > >best
> > > > > > > >> > > >>> > >>>>> estimate
> > > > > > > >> > > >>> > >>>>> > I have right now (as already mentioned):
> > 30min
> > > > > > > >> > > >>> > >>>>> >
> > > > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan
> > Maurya <
> > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > > > >> > > >>> > >>>>> > wrote:
> > > > > > > >> > > >>> > >>>>> >
> > > > > > > >> > > >>> > >>>>> > > [image: image.png]
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan
> > > Maurya <
> > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > > > >> > > >>> > >>>>> > > wrote:
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >> [image: image.png]
> > > > > > > >> > > >>> > >>>>> > >> I am getting this.
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM
> > Aleksandar
> > > > > > > Vidakovic
> > > > > > > >> <
> > > > > > > >> > > >>> > >>>>> > >> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>> Hi all,
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost)
> > ready...
> > > > it
> > > > > > took
> > > > > > > >> me a
> > > > > > > >> > > >>> > >moment
> > > > > > > >> > > >>> > >>>>> and a
> > > > > > > >> > > >>> > >>>>> > >>> couple
> > > > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot
> > > > > failures...
> > > > > > > the
> > > > > > > >> > > >>> services
> > > > > > > >> > > >>> > >are
> > > > > > > >> > > >>> > >>>>> quite
> > > > > > > >> > > >>> > >>>>> > >>> resource hungry and default settings
> > won't
> > > > do
> > > > > > it.
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> The services are still starting and
> this
> > > > will
> > > > > > > take a
> > > > > > > >> > > while
> > > > > > > >> > > >>> > >(my
> > > > > > > >> > > >>> > >>>>> best
> > > > > > > >> > > >>> > >>>>> > guess
> > > > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org/login
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Credentials:
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Tenant : playground
> > > > > > > >> > > >>> > >>>>> > >>> Username : operator
> > > > > > > >> > > >>> > >>>>> > >>> Password : init1@l
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to
> > compile
> > > > this
> > > > > > app
> > > > > > > >> is
> > > > > > > >> > > with
> > > > > > > >> > > >>> > >NodeJS
> > > > > > > >> > > >>> > >>>>> > 8.11.1
> > > > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't
> > work).
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Identity Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2021/identity/v1
> > > > > > > >> > > >>> > >>>>> > >>> Office Service:
> > > > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
> > > > > > > >> > > >>> > >>>>> > >>> Customer Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2024/customer/v1
> > > > > > > >> > > >>> > >>>>> > >>> Accounting Service:
> > > > > > > >> > > >>> > >>>>>
> > > http://fineract-vm.apache.org:2025/accounting/v1
> > > > > > > >> > > >>> > >>>>> > >>> Portfolio Service:
> > > > > > > >> > > >>> > >>>>>
> > http://fineract-vm.apache.org:2026/portfolio/v1
> > > > > > > >> > > >>> > >>>>> > >>> Deposit Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1
> > > > > > > >> > > >>> > >>>>> > >>> Teller Service:
> > > > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
> > > > > > > >> > > >>> > >>>>> > >>> Reporting Service:
> > > > > > > >> > > >>> > >>>>>
> > http://fineract-vm.apache.org:2029/reporting/v1
> > > > > > > >> > > >>> > >>>>> > >>> Cheque Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1
> > > > > > > >> > > >>> > >>>>> > >>> Payroll Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes
> > quite
> > > a
> > > > > > > >> while... if
> > > > > > > >> > > you
> > > > > > > >> > > >>> > >>>>> encounter
> > > > > > > >> > > >>> > >>>>> > >>> connection problems retry a couple of
> > > > minutes
> > > > > > > later
> > > > > > > >> to
> > > > > > > >> > > >>> > >ensure I
> > > > > > > >> > > >>> > >>>>> am not
> > > > > > > >> > > >>> > >>>>> > >>> currently working on something.
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the
> backend
> > > > > services
> > > > > > > >> once
> > > > > > > >> > per
> > > > > > > >> > > >>> > >day or
> > > > > > > >> > > >>> > >>>>> so to
> > > > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to
> > figure
> > > > out
> > > > > > how
> > > > > > > we
> > > > > > > >> > want
> > > > > > > >> > > >>> to
> > > > > > > >> > > >>> > >>>>> handle
> > > > > > > >> > > >>> > >>>>> > this
> > > > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where to
> put
> > > > this
> > > > > > demo
> > > > > > > >> > server
> > > > > > > >> > > >>> > >>>>> configuration
> > > > > > > >> > > >>> > >>>>> > >>> then let me know here.
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any
> > problems
> > > (I
> > > > > > have
> > > > > > > >> not
> > > > > > > >> > > >>> > >extensively
> > > > > > > >> > > >>> > >>>>> > tested
> > > > > > > >> > > >>> > >>>>> > >>> it
> > > > > > > >> > > >>> > >>>>> > >>> yet).
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Cheers,
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Aleks
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM
> > > Aleksandar
> > > > > > > >> Vidakovic <
> > > > > > > >> > > >>> > >>>>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem to be
> > > > > starting
> > > > > > > fine
> > > > > > > >> > > now...
> > > > > > > >> > > >>> > >just
> > > > > > > >> > > >>> > >>>>> have
> > > > > > > >> > >
Re: Status of Demo Server on Apache VM Infrastructure
Posted by Isaac Kamga <is...@mifos.org>.
Hello Avik,
On Wed, Oct 10, 2018 at 5:24 PM Avik Ganguly <av...@gmail.com>
wrote:
> Hi guys,
>
> I am assuming that the suggested form of development for Fineract CN is to
> write unit tests to test the code which you just wrote locally and then
> push the code to a remote cluster or a massive VM and wait 40+ minutes to
> run the API / integration tests there. A lot of the confusion regarding the
> stacktraces in identity, rhythm, timeouts and auto-shutdown might or might
> not have anything to do with the resource crunch in my local. (i7, 8GB,
> SSD)
>
> Maybe a FAQ section in "How to build Fineract CN" to help devs with local
> setup would go a long way to bring in contributions to the core modules.
>
With the right specifications on your local (i7, 16GB, SSD), you could test
your changes instead of having to use a remote VM. So far, what we've done
is to ask developers like Graham and Karthik with new knowledge to edit
their experiences into the page.
A FAQ section in
https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN
is very welcome and helpful. Maybe you could start by synthesizing the
above paragraph into the first Q and A.
>
> 1. Change demo server orchestration logic from running Identity,
> Organization, Customer, Accounting, Portfolio, Deposit, Teller, Reporting,
> Cheque, Payroll, Group and Notification services to running only Identity,
> Organization, Customer and working set of your choice (Portfolio and
> Accounting OR Deposit and Accounting OR Group and Portfolio)
>
> 2. Leave out Rhythm from orchestration logic for now. A JIRA ticket
> tracking ability of Rhythm to run with low resource footprint will probably
> help.
>
>
> @Myrle, can you elaborate on FINCN-25? Is the motivation of this ticket to
> reduce resource footprint only or also to reduce the startup time from 40
> minutes to a reasonable couple of minutes? Do you have any debugging tips
> with respect to the slow startup caused by services is due to registering
> with eureka or provisioning or sequential startup?
>
> Maybe we should also follow up in Cassandra mailing list regarding
> configuring Cassandra to run within 8GB of RAM and include the same config
> as part of the FAQ.
>
>
> Regards,
> Avik.
> ᐧ
>
> On Wed, Sep 19, 2018 at 2:23 AM Courage Angeh <co...@gmail.com>
> wrote:
>
> > Thanks for the update. Ed.
> >
> > If you need an extra hand please let me know Aleks.
> >
> > On Tue, Sep 18, 2018, 4:27 PM Ed Cable <ed...@mifos.org> wrote:
> >
> > > Hi all,
> > >
> > > Just to bring this thread back up to top of everyone's inbox. Aleks
> > > estimates he has about a day left to get the demo server up and running
> > is
> > > hoping to tackle it by tomorrow. Here are the two approaches he
> suggested
> > > now that we have the second VM allocated:
> > >
> > > - continue with the JUnit based startup sequence of the demo module;
> the
> > > one thing that I would need to change is to split up the startup
> sequence
> > > and recompile
> > > - create a set of Docker Compose files to do the startup on the 2 VM
> > > instances (maybe I could borrow some stuff from Courage)
> > >
> > > I would prefer the latter, because you have more fine grained control
> and
> > > it's easier move service instances around... and you don't have to
> > > recompile anything.
> > >
> > > Ed
> > >
> > > On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <my...@apache.org> wrote:
> > >
> > > > Hey Aleks,
> > > >
> > > > I don’t see anything in that log that would explain the behavior
> you’re
> > > > observing. Markus did most of the work on deposits. Markus? Do you
> have
> > > any
> > > > ideas?
> > > >
> > > > Best Regards,
> > > > Myrle
> > > >
> > > >
> > > >
> > > > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
> > > > cheetah@monkeysintown.com>
> > > > wrote:
> > > >
> > > > > @Myrle: with your proposed changes things look a bit better. I've
> > > > disabled
> > > > > Rhythm as you suggested and get past all microservice startups
> with a
> > > lot
> > > > > less exceptions (one is coming repeatedly from Netty native
> > transport,
> > > > > because epoll is not available).
> > > > >
> > > > > The one exception (actually it happened on 2 occasions) that is
> still
> > > > > there:
> > > > >
> > > > > [snip]
> > > > >
> > > > > 13:33:45.080 [main] INFO
> > > > o.a.f.c.d.s.DepositAccountManagementApplication -
> > > > > Started DepositAccountManagementApplication in 55.187 seconds (JVM
> > > > running
> > > > > for 62.772)
> > > > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> via
> > > > > configuration
> > > > > Resolving artifact
> > > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > Resolving metadata
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > from /root/.m2/repository (enhanced)
> > > > > Resolved metadata
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > from /root/.m2/repository (enhanced)
> > > > > Resolved artifact
> > > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > from
> > > > > /root/.m2/repository (enhanced)
> > > > > 13:34:56.800 [DiscoveryClient-0] ERROR
> > > c.n.discovery.TimedSupervisorTask
> > > > -
> > > > > task supervisor timed out
> > > > > java.util.concurrent.TimeoutException: null
> > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > at
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 13:34:56.801 [DiscoveryClient-1] ERROR
> > > c.n.discovery.TimedSupervisorTask
> > > > -
> > > > > task supervisor timed out
> > > > > java.util.concurrent.TimeoutException: null
> > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > at
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 13:34:56.916 [DiscoveryClient-2] ERROR
> > > c.n.discovery.TimedSupervisorTask
> > > > -
> > > > > task supervisor timed out
> > > > > java.util.concurrent.TimeoutException: null
> > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > at
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 13:35:08.586 [main] INFO
> o.s.c.a.AnnotationConfigApplicationContext
> > -
> > > > > Refreshing
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > :
> > > > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context
> > hierarchy
> > > > > 13:35:10.182 [main] INFO
> > > o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
> > > > -
> > > > > JSR-330 'javax.inject.Inject' annotation found and supported for
> > > > autowiring
> > > > > 13:35:10.371 [main] INFO
> > > > > o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker
> -
> > > Bean
> > > > > 'configurationPropertiesRebinderAutoConfiguration' of type [class
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
> > > > > is not eligible for getting processed by all BeanPostProcessors
> (for
> > > > > example: not eligible for auto-proxying)
> > > > > 13:35:10.435 [background-preinit] INFO
> > > > o.h.validator.internal.util.Version
> > > > > - HV000001: Hibernate Validator 5.3.0.Final
> > > > >
> > > > > [/snip]
> > > > >
> > > > > It looks like the whole startup procedure goes through and I can
> see
> > > some
> > > > > kind of "summary" that lists all REST endpoints. But it stays in
> that
> > > > state
> > > > > for only a minute or so and then a cascade of shutdowns are
> triggered
> > > > > again. The only thing that stays alive is Eureka which complains
> > about
> > > > the
> > > > > inaccessible services.
> > > > >
> > > > > Any idea what else we could do other than just ordering an
> additional
> > > > > machine?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Aleks
> > > > >
> > > > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <my...@apache.org>
> > wrote:
> > > > >
> > > > > > Hey Aleks,
> > > > > >
> > > > > > It looks like that exception is coming from Rhythm. I suspect
> it's
> > > not
> > > > > the
> > > > > > cause of your difficulties, since you say your other services are
> > > going
> > > > > > down too. You can run most of the Fineract server without
> Rhythm.
> > > You
> > > > > > could edit the demo-server script to leave it out for the
> purposes
> > of
> > > > > > testing. The consequence will be a lack of interest calculations
> > for
> > > > > > disbursed loans, but for finding out what the real problem is,
> it'd
> > > be
> > > > > good
> > > > > > to get this one out of the way.
> > > > > >
> > > > > > Best Regards,
> > > > > > Myrle
> > > > > >
> > > > > > (FYI: I've removed the non-list subscribers from the to. Most of
> > > them
> > > > > are
> > > > > > already subscribed, and those who aren't, don't want to be.)
> > > > > >
> > > > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
> > > > > > cheetah@monkeysintown.com> wrote:
> > > > > >
> > > > > > > I managed to further improve the startup procedure (aka less
> > > > exceptions
> > > > > > > than before)... there are still some timeouts, but not as many
> as
> > > > > before.
> > > > > > >
> > > > > > > The demo server still dies after this exception:
> > > > > > >
> > > > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > > > configuration
> > > > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > > > configuration
> > > > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > > > configuration
> > > > > > > 23:28:45.990 [pool-9-thread-1] ERROR
> > > > > > o.s.s.s.TaskUtils$LoggingErrorHandler
> > > > > > > - Unexpected error occurred in scheduled task.
> > > > > > >
> org.springframework.transaction.CannotCreateTransactionException:
> > > > Could
> > > > > > not
> > > > > > > open JPA EntityManager for transaction; nested exception is
> > > > > > > javax.persistence.PersistenceException:
> > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > > resultset:
> > > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > > Query is : set autocommit=0
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
> > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> > > > > > > at
> > > > > >
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > Caused by: javax.persistence.PersistenceException:
> > > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > > resultset:
> > > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > > Query is : set autocommit=0
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
> > > > > > > at
> > > > > > >
> > > > >
> > >
> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
> > > > > > > ... 20 common frames omitted
> > > > > > > Caused by: java.sql.SQLNonTransientConnectionException: Could
> not
> > > > read
> > > > > > > resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > > Query is : set autocommit=0
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
> > > > > > > at
> > > > > > >
> > > > >
> > >
> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
> > > > > > > ... 22 common frames omitted
> > > > > > > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException:
> > Could
> > > > not
> > > > > > > read resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > > Query is : set autocommit=0
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
> > > > > > > ... 31 common frames omitted
> > > > > > > Caused by: java.io.EOFException: unexpected end of stream,
> read 0
> > > > bytes
> > > > > > > from 4
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
> > > > > > > ... 33 common frames omitted
> > > > > > > 23:29:05.500 [qtp6422064-20] INFO
> > > > o.a.f.c.l.c.ServiceExceptionFilter -
> > > > > > > Responding with a service error ServiceError{code=409,
> > > > > > message='Application
> > > > > > > identity-v1 already exists!'}
> > > > > > > 23:29:21.282 [Thread-15] INFO
> > > > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> Closing
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > > :
> > > > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > > 23:29:23.717 [Thread-15] INFO
> o.s.c.s.DefaultLifecycleProcessor
> > -
> > > > > > Stopping
> > > > > > > beans in phase 2147483647
> > > > > > > 23:29:24.267 [Thread-15] INFO
> o.s.c.s.DefaultLifecycleProcessor
> > -
> > > > > > Stopping
> > > > > > > beans in phase 0
> > > > > > > 23:29:24.557 [Thread-15] INFO
> > o.s.b.a.e.jmx.EndpointMBeanExporter
> > > -
> > > > > > > Unregistering JMX-exposed beans on shutdown
> > > > > > > 23:29:24.579 [Thread-15] INFO
> > o.s.b.a.e.jmx.EndpointMBeanExporter
> > > -
> > > > > > > Unregistering JMX-exposed beans
> > > > > > >
> > > > > > > The only thing I've left: start the databases (aka Cassandra)
> on
> > a
> > > > > > separate
> > > > > > > machine (e. g. on Digital Ocean) and see how this is working
> > then.
> > > > > > >
> > > > > > > ... FYI.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Aleks
> > > > > > >
> > > > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
> > > > > > > cheetah@monkeysintown.com> wrote:
> > > > > > >
> > > > > > > > @Rajan: it's not running, because the demo server dies... See
> > > logs
> > > > > from
> > > > > > > > previous message...
> > > > > > > >
> > > > > > > > We just have to figure that one out and we should be good to
> > go.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
> > > > rajanmaurya154@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks, Alex,
> > > > > > > >>
> > > > > > > >> I can see web app is running but I can't log in with the
> > > > credentials
> > > > > > > that
> > > > > > > >> you shared earlier
> > > > > > > >>
> > > > > > > >> Tenant : playground
> > > > > > > >> Username : operator
> > > > > > > >> Password : init1@l
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >>
> > > > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
> > > > > > > >> cheetah@monkeysintown.com> wrote:
> > > > > > > >>
> > > > > > > >> > Hi all,
> > > > > > > >> >
> > > > > > > >> > ... sweeeet! I got a considerable step further.
> > > > > > > >> >
> > > > > > > >> > @Markus: thanks for the conversation... gave me an idea
> how
> > to
> > > > fix
> > > > > > > (most
> > > > > > > >> > of) it!
> > > > > > > >> >
> > > > > > > >> > The problem was - as suspected - the memory (or lack of).
> > The
> > > VM
> > > > > > > >> instance
> > > > > > > >> > at Apache has net 30G available. 16G are needed for
> > Cassandra
> > > > and
> > > > > in
> > > > > > > my
> > > > > > > >> > previous tests I tried to run the demo server with up to
> > 16G.
> > > > The
> > > > > > > >> problem
> > > > > > > >> > is that the demo server needs considerably more memory
> than
> > > 16G.
> > > > > > > >> According
> > > > > > > >> > to my observation it's around 23-24G. Even if you limit
> the
> > > demo
> > > > > > > server
> > > > > > > >> > memory allocation with "-Xmx" to 12G (for example) it
> would
> > > just
> > > > > > > >> continue
> > > > > > > >> > to consume more memory.
> > > > > > > >> >
> > > > > > > >> > My solution: add a swap file. Not ideal in terms of
> > > performance,
> > > > > but
> > > > > > > it
> > > > > > > >> > gets the whole thing going... at least mostly (please read
> > > on).
> > > > > > > >> >
> > > > > > > >> > Current status:
> > > > > > > >> >
> > > > > > > >> > - Cassandra (and all other Docker containers) are still
> > > > > running;
> > > > > > > >> > Cassandra died in my previous attempts at the latest
> when
> > > the
> > > > > > > Teller
> > > > > > > >> > application started, but with the swap file it's still
> > > > running
> > > > > > > >> > - all Spring Boot apps are starting now
> > > > > > > >> > - I get once in a while some spurious timeouts (I think
> > > when
> > > > > > either
> > > > > > > >> the
> > > > > > > >> > Eureka server is contacted or the config server)
> > > > > > > >> > - there are also sometimes exceptions concerning MySQL
> > > > > > connections,
> > > > > > > >> but
> > > > > > > >> > doesn't seem to matter overall
> > > > > > > >> > - I could issue an authentication request with Postman
> to
> > > > > > > >> > http://fineract-vm.apache.org:2021/identity/v1; just
> to
> > > test
> > > > > if
> > > > > > > >> > anything
> > > > > > > >> > is responding
> > > > > > > >> > - PROBLEM: but the demo server (the Spring Boot apps)
> > > > > eventually
> > > > > > > dies
> > > > > > > >> > after a couple of minutes; I don't think that the
> memory
> > is
> > > > the
> > > > > > > >> problem
> > > > > > > >> > at
> > > > > > > >> > this point; I think it's more of a timeout problem
> > > concerning
> > > > > the
> > > > > > > >> > communication with Eureka and/or the config server
> (maybe
> > > > > because
> > > > > > > the
> > > > > > > >> > system is a bit slow)
> > > > > > > >> >
> > > > > > > >> > ... and here the memory footprint ("free -h") when
> > everything
> > > is
> > > > > > > >> running,
> > > > > > > >> > just FYI:
> > > > > > > >> >
> > > > > > > >> > total used free shared
> > > > > buff/cache
> > > > > > > >> > available
> > > > > > > >> > Mem: 31G 31G 226M 8.4M
> > > > > 169M
> > > > > > > >> > 47M
> > > > > > > >> > Swap: 15G 6.8G 9.2G
> > > > > > > >> >
> > > > > > > >> > ... after the Payroll application is started (approx.) I
> see
> > > > these
> > > > > > > >> > exceptions appearing in the logs (a whole series of them):
> > > > > > > >> >
> > > > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > endpoints
> > > > > via
> > > > > > > >> > configuration
> > > > > > > >> > Resolving artifact
> > > > > > > >> >
> > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > > >> > Resolving metadata
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > > >> > Resolved metadata
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > > >> > Resolved artifact
> > > > > > > >> >
> > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > > from
> > > > > > > >> > /root/.m2/repository (enhanced)
> > > > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
> > > > > > > >> c.n.discovery.TimedSupervisorTask -
> > > > > > > >> > task supervisor timed out
> > > > > > > >> > java.util.concurrent.TimeoutException: null
> > > > > > > >> > at
> java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > > > >> > at
> > > > > > > >>
> > > > > >
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > >> > at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > >> > at java.lang.Thread.run(Thread.java:748)
> > > > > > > >> >
> > > > > > > >> > ... but it still seems to have no problem starting the
> next
> > > app
> > > > > > > (Group)
> > > > > > > >> and
> > > > > > > >> > things look good for a while...
> > > > > > > >> >
> > > > > > > >> > ... until suddenly it seems that app instances are
> beginning
> > > to
> > > > > > > >> shutdown...
> > > > > > > >> > and I'm not sure why this happens:
> > > > > > > >> >
> > > > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > endpoints
> > > > > via
> > > > > > > >> > configuration
> > > > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
> > > > > > > >> c.datastax.driver.core.ClockFactory -
> > > > > > > >> > Using native clock to generate timestamps.
> > > > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
> > > > > > > >> c.datastax.driver.core.CodecRegistry
> > > > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
> > > > > > > >> java.time.LocalDateTime]
> > > > > > > >> > because it collides with previously registered codec
> > > > > > > LocalDateTimeCodec
> > > > > > > >> > [timestamp <-> java.time.LocalDateTime]
> > > > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
> > > > > > > c.d.d.c.p.DCAwareRoundRobinPolicy
> > > > > > > >> -
> > > > > > > >> > Using data-center name 'datacenter1' for
> > > DCAwareRoundRobinPolicy
> > > > > (if
> > > > > > > >> this
> > > > > > > >> > is incorrect, please provide the correct datacenter name
> > with
> > > > > > > >> > DCAwareRoundRobinPolicy constructor)
> > > > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
> > > > > > > com.datastax.driver.core.Cluster -
> > > > > > > >> > New Cassandra host /127.0.0.1:9042 added
> > > > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > endpoints
> > > > > via
> > > > > > > >> > configuration
> > > > > > > >> > 18:27:51.110 [Thread-15] INFO
> > > > > > > >> > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> > > > Closing
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > > >> > :
> > > > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > > endpoints
> > > > > via
> > > > > > > >> > configuration
> > > > > > > >> > 18:27:53.373 [Thread-15] INFO
> > > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > > >> Stopping
> > > > > > > >> > beans in phase 2147483647
> > > > > > > >> > 18:27:54.331 [Thread-15] INFO
> > > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > > >> Stopping
> > > > > > > >> > beans in phase 0
> > > > > > > >> > 18:27:54.736 [Thread-15] INFO
> > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > > >> > 18:27:54.740 [Thread-15] INFO
> > > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > > >> > Unregistering JMX-exposed beans
> > > > > > > >> > 18:27:54.798 [Thread-15] INFO
> > > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > > >> > 18:27:54.800 [Thread-15] INFO
> > > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > > >> > Unregistering JMX-exposed beans
> > > > > > > >> > 18:27:54.937 [Thread-15] INFO
> > > > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean - Closing
> JPA
> > > > > > > >> > EntityManagerFactory for persistence unit 'metaPU'
> > > > > > > >> > 18:27:58.376 [async-processor-1] INFO
> > > > > > > >> o.f.c.internal.util.VersionPrinter -
> > > > > > > >> > Flyway 3.2.1 by Boxfuse
> > > > > > > >> > 18:27:58.677 [async-processor-1] INFO
> > > > > > > >> o.f.c.i.dbsupport.DbSupportFactory -
> > > > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL 10.3)
> > > > > > > >> > 18:28:00.620 [async-processor-1] INFO
> > > > > > > >> o.f.core.internal.command.DbValidate
> > > > > > > >> > - Validated 2 migrations (execution time 00:01.169s)
> > > > > > > >> > 18:28:00.901 [Thread-15] INFO
> > > > o.e.jetty.server.AbstractConnector
> > > > > -
> > > > > > > >> Stopped
> > > > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{0.0.0.0:2020
> }
> > > > > > > >> > 18:28:01.268 [Thread-15] INFO
> > > > > o.e.j.server.handler.ContextHandler -
> > > > > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
> > > > > > > >> >
> > > > > > > >> > ... the list is actually a lot longer... in the rest of
> the
> > > log
> > > > > you
> > > > > > > can
> > > > > > > >> see
> > > > > > > >> > that one by one all the apps are shutting down.
> > > > > > > >> >
> > > > > > > >> > The last (error) entries I see are these:
> > > > > > > >> >
> > > > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR
> > > > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not
> refresh
> > > JMS
> > > > > > > >> Connection
> > > > > > > >> > for destination 'cheques-v1' - retrying using
> > > > > > > >> FixedBackOff{interval=5000,
> > > > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error
> > while
> > > > > > > >> attempting to
> > > > > > > >> > retrieve a connection from the pool; nested exception is
> > > > > > > >> > javax.jms.JMSException: Could not connect to broker URL:
> > > > > > > >> > tcp://localhost:61616. Reason: java.net.ConnectException:
> > > > > Connection
> > > > > > > >> > refused (Connection refused)
> > > > > > > >> >
> > > > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in the
> > > > process
> > > > > of
> > > > > > > the
> > > > > > > >> > previous errors.
> > > > > > > >> >
> > > > > > > >> > So far from here... any input on the above highly
> > appreciated.
> > > > > > > Overall I
> > > > > > > >> > think we are close.
> > > > > > > >> >
> > > > > > > >> > Cheers,
> > > > > > > >> >
> > > > > > > >> > Aleks
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <
> > mage@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > One thing we maybe can do, given we are part of the
> Apache
> > > > > Family,
> > > > > > > is
> > > > > > > >> to
> > > > > > > >> > > ask the Cassandra community for some suggestions.
> > > > > > > >> > >
> > > > > > > >> > > Cheers
> > > > > > > >> > >
> > > > > > > >> > > Markus
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <
> > > mage@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Hey all,
> > > > > > > >> > > >
> > > > > > > >> > > > we are not running Cassandra in a container, we have
> > > > dedicated
> > > > > > VMs
> > > > > > > >> to
> > > > > > > >> > run
> > > > > > > >> > > > our cluster.
> > > > > > > >> > > >
> > > > > > > >> > > > Given this we are not the best persons to help here,
> > > sorry.
> > > > (;
> > > > > > > >> > > >
> > > > > > > >> > > > Cheers
> > > > > > > >> > > >
> > > > > > > >> > > > Markus
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
> > > > edcable@mifos.org>
> > > > > > > >> wrote:
> > > > > > > >> > > >
> > > > > > > >> > > >> Adding Patric directly to this thread too so he can
> > give
> > > > his
> > > > > > > input
> > > > > > > >> > > >> regarding to the challenges with Cassandra.
> > > > > > > >> > > >>
> > > > > > > >> > > >> Ed
> > > > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar
> Vidakovic <
> > > > > > > >> > > >> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>
> > > > > > > >> > > >>> Hi Victor,
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> .... this is the relevant part of the
> > docker-compose.yml
> > > > > file:
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> cassandra0:
> > > > > > > >> > > >>> image: cassandra:3.11.1
> > > > > > > >> > > >>> container_name: cassandra0
> > > > > > > >> > > >>> ports:
> > > > > > > >> > > >>> - 9042:9042
> > > > > > > >> > > >>> - 9160:9160
> > > > > > > >> > > >>> - 7199:7199
> > > > > > > >> > > >>> - 8778:8778
> > > > > > > >> > > >>> volumes:
> > > > > > > >> > > >>> - ./cassandra:/etc/cassandra
> > > > > > > >> > > >>> environment:
> > > > > > > >> > > >>> - CASSANDRA_START_RPC=true
> > > > > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0
> > > > > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster
> > > > > > > >> > > >>> ulimits:
> > > > > > > >> > > >>> memlock: -1
> > > > > > > >> > > >>> nproc: 32768
> > > > > > > >> > > >>> nofile: 100000
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> ... and just to be complete... here's the Docker
> > service
> > > > > > > >> > configuration
> > > > > > > >> > > >>> ("/lib/systemd/system/docker.service"):
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> [Unit]
> > > > > > > >> > > >>> Description=Docker Application Container Engine
> > > > > > > >> > > >>> Documentation=https://docs.docker.com
> > > > > > > >> > > >>> After=network-online.target docker.socket
> > > > firewalld.service
> > > > > > > >> > > >>> Wants=network-online.target
> > > > > > > >> > > >>> Requires=docker.socket
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> [Service]
> > > > > > > >> > > >>> Type=notify
> > > > > > > >> > > >>> # the default is not to use systemd for cgroups
> > because
> > > > the
> > > > > > > >> delegate
> > > > > > > >> > > >>> issues
> > > > > > > >> > > >>> still
> > > > > > > >> > > >>> # exists and systemd currently does not support the
> > > cgroup
> > > > > > > feature
> > > > > > > >> > set
> > > > > > > >> > > >>> required
> > > > > > > >> > > >>> # for containers run by docker
> > > > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
> > > > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
> > > > > > > >> > > >>> LimitNOFILE=1048576
> > > > > > > >> > > >>> # Having non-zero Limit*s causes performance
> problems
> > > due
> > > > to
> > > > > > > >> > accounting
> > > > > > > >> > > >>> overhead
> > > > > > > >> > > >>> # in the kernel. We recommend using cgroups to do
> > > > > > > container-local
> > > > > > > >> > > >>> accounting.
> > > > > > > >> > > >>> LimitNPROC=infinity
> > > > > > > >> > > >>> LimitCORE=infinity
> > > > > > > >> > > >>> LimitMEMLOCK=infinity
> > > > > > > >> > > >>> # Uncomment TasksMax if your systemd version
> supports
> > > it.
> > > > > > > >> > > >>> # Only systemd 226 and above support this version.
> > > > > > > >> > > >>> TasksMax=infinity
> > > > > > > >> > > >>> TimeoutStartSec=0
> > > > > > > >> > > >>> # set delegate yes so that systemd does not reset
> the
> > > > > cgroups
> > > > > > of
> > > > > > > >> > docker
> > > > > > > >> > > >>> containers
> > > > > > > >> > > >>> Delegate=yes
> > > > > > > >> > > >>> # kill only the docker process, not all processes in
> > the
> > > > > > cgroup
> > > > > > > >> > > >>> KillMode=process
> > > > > > > >> > > >>> # restart the docker process if it exits prematurely
> > > > > > > >> > > >>> Restart=on-failure
> > > > > > > >> > > >>> StartLimitBurst=3
> > > > > > > >> > > >>> StartLimitInterval=60s
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> [Install]
> > > > > > > >> > > >>> WantedBy=multi-user.target
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> ... the one tweak I did there is to set LimitMEMLOCK
> > to
> > > > > > > >> infinity...
> > > > > > > >> > do
> > > > > > > >> > > >>> you
> > > > > > > >> > > >>> think the problems are file handle related? Should I
> > set
> > > > > > > >> LimitNOFILE
> > > > > > > >> > > also
> > > > > > > >> > > >>> to infinity?
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> Appreciate the help.
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> Cheers,
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> Aleks
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
> > > > > > > >> > > >>> victor.romero@fintecheando.mx>
> > > > > > > >> > > >>> wrote:
> > > > > > > >> > > >>>
> > > > > > > >> > > >>> > Hi Aleks,
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> > Can you share the ulimits flags that the
> cassandra's
> > > > > > container
> > > > > > > >> is
> > > > > > > >> > > using
> > > > > > > >> > > >>> > while it is running?
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> > The values are being set in the composer file? Or
> in
> > > the
> > > > > > > >> upstart or
> > > > > > > >> > > >>> > systemd docker's deamon config files?
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> > Enviado desde TypeApp
> > > > > > > >> > > >>> >
> > > > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM, Aleksandar
> > > > > > Vidakovic <
> > > > > > > >> > > >>> > cheetah@monkeysintown.com> escribió:
> > > > > > > >> > > >>> > >... and I should have attached the Cassandra log
> > > > dump...
> > > > > so
> > > > > > > >> here
> > > > > > > >> > it
> > > > > > > >> > > >>> > >is...
> > > > > > > >> > > >>> > >
> > > > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar
> > Vidakovic
> > > <
> > > > > > > >> > > >>> > >cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >
> > > > > > > >> > > >>> > >> Hi all,
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> ... as you might have noticed I did a couple of
> > > > > restarts
> > > > > > > >> > today...
> > > > > > > >> > > >>> the
> > > > > > > >> > > >>> > >> problem I am facing now: I can't get Cassandra
> to
> > > run
> > > > > in
> > > > > > a
> > > > > > > >> > stable
> > > > > > > >> > > >>> > >way.
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> Things look quite OK for a while when running
> the
> > > > demo
> > > > > > > >> server,
> > > > > > > >> > but
> > > > > > > >> > > >>> > >then
> > > > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts
> > > > > misbehaving
> > > > > > > with
> > > > > > > >> > the
> > > > > > > >> > > >>> > >deposit
> > > > > > > >> > > >>> > >> microservice startup, sometimes with portfolio
> or
> > > > > > teller).
> > > > > > > I
> > > > > > > >> > tried
> > > > > > > >> > > >>> to
> > > > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set
> some
> > > > > Docker
> > > > > > > >> limits
> > > > > > > >> > to
> > > > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe
> > someone
> > > > can
> > > > > > > help
> > > > > > > >> out
> > > > > > > >> > > >>> > >here? Is
> > > > > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB of
> > > > memory?
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> Other than that the setup would be ready to
> go...
> > > we
> > > > > just
> > > > > > > >> need
> > > > > > > >> > to
> > > > > > > >> > > >>> get
> > > > > > > >> > > >>> > >the
> > > > > > > >> > > >>> > >> database running more reliably.
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> Cheers,
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> Aleks
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar
> > > Vidakovic
> > > > <
> > > > > > > >> > > >>> > >> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>
> > > > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed to
> add
> > > > some
> > > > > > > >> > additional
> > > > > > > >> > > >>> > >reverse
> > > > > > > >> > > >>> > >>> proxy configuration for the web UI to work)...
> > and
> > > > > there
> > > > > > > >> were
> > > > > > > >> > > more
> > > > > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
> > > > > > > >> > > >>> > >>>
> > > > > > > >> > > >>> > >>> Just FYI
> > > > > > > >> > > >>> > >>>
> > > > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
> > > > Vidakovic <
> > > > > > > >> > > >>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>>
> > > > > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can
> get
> > > back
> > > > > to
> > > > > > > >> testing
> > > > > > > >> > > >>> only
> > > > > > > >> > > >>> > >>>> later tonight.
> > > > > > > >> > > >>> > >>>>
> > > > > > > >> > > >>> > >>>> Cheers
> > > > > > > >> > > >>> > >>>>
> > > > > > > >> > > >>> > >>>>
> > > > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya
> > > > > > > >> > > >>> > ><ra...@gmail.com>
> > > > > > > >> > > >>> > >>>> wrote:
> > > > > > > >> > > >>> > >>>>
> > > > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test after
> 30
> > > min.
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>> Big thanks for this 🙂
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM Aleksandar
> > > > > Vidakovic <
> > > > > > > >> > > >>> > >>>>> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>>>>
> > > > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted.
> > > > > > > >> > > >>> > >>>>> >
> > > > > > > >> > > >>> > >>>>> > ... and as I said: the services are still
> > > > > > starting...
> > > > > > > >> and
> > > > > > > >> > the
> > > > > > > >> > > >>> > >best
> > > > > > > >> > > >>> > >>>>> estimate
> > > > > > > >> > > >>> > >>>>> > I have right now (as already mentioned):
> > 30min
> > > > > > > >> > > >>> > >>>>> >
> > > > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan
> > Maurya <
> > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > > > >> > > >>> > >>>>> > wrote:
> > > > > > > >> > > >>> > >>>>> >
> > > > > > > >> > > >>> > >>>>> > > [image: image.png]
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan
> > > Maurya <
> > > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > > > >> > > >>> > >>>>> > > wrote:
> > > > > > > >> > > >>> > >>>>> > >
> > > > > > > >> > > >>> > >>>>> > >> [image: image.png]
> > > > > > > >> > > >>> > >>>>> > >> I am getting this.
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM
> > Aleksandar
> > > > > > > Vidakovic
> > > > > > > >> <
> > > > > > > >> > > >>> > >>>>> > >> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>>>> > >>
> > > > > > > >> > > >>> > >>>>> > >>> Hi all,
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost)
> > ready...
> > > > it
> > > > > > took
> > > > > > > >> me a
> > > > > > > >> > > >>> > >moment
> > > > > > > >> > > >>> > >>>>> and a
> > > > > > > >> > > >>> > >>>>> > >>> couple
> > > > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot
> > > > > failures...
> > > > > > > the
> > > > > > > >> > > >>> services
> > > > > > > >> > > >>> > >are
> > > > > > > >> > > >>> > >>>>> quite
> > > > > > > >> > > >>> > >>>>> > >>> resource hungry and default settings
> > won't
> > > > do
> > > > > > it.
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> The services are still starting and
> this
> > > > will
> > > > > > > take a
> > > > > > > >> > > while
> > > > > > > >> > > >>> > >(my
> > > > > > > >> > > >>> > >>>>> best
> > > > > > > >> > > >>> > >>>>> > guess
> > > > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org/login
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Credentials:
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Tenant : playground
> > > > > > > >> > > >>> > >>>>> > >>> Username : operator
> > > > > > > >> > > >>> > >>>>> > >>> Password : init1@l
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to
> > compile
> > > > this
> > > > > > app
> > > > > > > >> is
> > > > > > > >> > > with
> > > > > > > >> > > >>> > >NodeJS
> > > > > > > >> > > >>> > >>>>> > 8.11.1
> > > > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't
> > work).
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Identity Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2021/identity/v1
> > > > > > > >> > > >>> > >>>>> > >>> Office Service:
> > > > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
> > > > > > > >> > > >>> > >>>>> > >>> Customer Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2024/customer/v1
> > > > > > > >> > > >>> > >>>>> > >>> Accounting Service:
> > > > > > > >> > > >>> > >>>>>
> > > http://fineract-vm.apache.org:2025/accounting/v1
> > > > > > > >> > > >>> > >>>>> > >>> Portfolio Service:
> > > > > > > >> > > >>> > >>>>>
> > http://fineract-vm.apache.org:2026/portfolio/v1
> > > > > > > >> > > >>> > >>>>> > >>> Deposit Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1
> > > > > > > >> > > >>> > >>>>> > >>> Teller Service:
> > > > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
> > > > > > > >> > > >>> > >>>>> > >>> Reporting Service:
> > > > > > > >> > > >>> > >>>>>
> > http://fineract-vm.apache.org:2029/reporting/v1
> > > > > > > >> > > >>> > >>>>> > >>> Cheque Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1
> > > > > > > >> > > >>> > >>>>> > >>> Payroll Service:
> > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes
> > quite
> > > a
> > > > > > > >> while... if
> > > > > > > >> > > you
> > > > > > > >> > > >>> > >>>>> encounter
> > > > > > > >> > > >>> > >>>>> > >>> connection problems retry a couple of
> > > > minutes
> > > > > > > later
> > > > > > > >> to
> > > > > > > >> > > >>> > >ensure I
> > > > > > > >> > > >>> > >>>>> am not
> > > > > > > >> > > >>> > >>>>> > >>> currently working on something.
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the
> backend
> > > > > services
> > > > > > > >> once
> > > > > > > >> > per
> > > > > > > >> > > >>> > >day or
> > > > > > > >> > > >>> > >>>>> so to
> > > > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to
> > figure
> > > > out
> > > > > > how
> > > > > > > we
> > > > > > > >> > want
> > > > > > > >> > > >>> to
> > > > > > > >> > > >>> > >>>>> handle
> > > > > > > >> > > >>> > >>>>> > this
> > > > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where to
> put
> > > > this
> > > > > > demo
> > > > > > > >> > server
> > > > > > > >> > > >>> > >>>>> configuration
> > > > > > > >> > > >>> > >>>>> > >>> then let me know here.
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any
> > problems
> > > (I
> > > > > > have
> > > > > > > >> not
> > > > > > > >> > > >>> > >extensively
> > > > > > > >> > > >>> > >>>>> > tested
> > > > > > > >> > > >>> > >>>>> > >>> it
> > > > > > > >> > > >>> > >>>>> > >>> yet).
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Cheers,
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> Aleks
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM
> > > Aleksandar
> > > > > > > >> Vidakovic <
> > > > > > > >> > > >>> > >>>>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > > > >> > > >>> > >>>>> > >>>
> > > > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem to be
> > > > > starting
> > > > > > > fine
> > > > > > > >> > > now...
> > > > > > > >> > > >>> > >just
> > > > > > > >> > > >>> > >>>>> have
> > > > > > > >> > >
Re: Status of Demo Server on Apache VM Infrastructure
Posted by Avik Ganguly <av...@gmail.com>.
Hi guys,
I am assuming that the suggested form of development for Fineract CN is to
write unit tests to test the code which you just wrote locally and then
push the code to a remote cluster or a massive VM and wait 40+ minutes to
run the API / integration tests there. A lot of the confusion regarding the
stacktraces in identity, rhythm, timeouts and auto-shutdown might or might
not have anything to do with the resource crunch in my local. (i7, 8GB, SSD)
Maybe a FAQ section in "How to build Fineract CN" to help devs with local
setup would go a long way to bring in contributions to the core modules.
1. Change demo server orchestration logic from running Identity,
Organization, Customer, Accounting, Portfolio, Deposit, Teller, Reporting,
Cheque, Payroll, Group and Notification services to running only Identity,
Organization, Customer and working set of your choice (Portfolio and
Accounting OR Deposit and Accounting OR Group and Portfolio)
2. Leave out Rhythm from orchestration logic for now. A JIRA ticket
tracking ability of Rhythm to run with low resource footprint will probably
help.
@Myrle, can you elaborate on FINCN-25? Is the motivation of this ticket to
reduce resource footprint only or also to reduce the startup time from 40
minutes to a reasonable couple of minutes? Do you have any debugging tips
with respect to the slow startup caused by services is due to registering
with eureka or provisioning or sequential startup?
Maybe we should also follow up in Cassandra mailing list regarding
configuring Cassandra to run within 8GB of RAM and include the same config
as part of the FAQ.
Regards,
Avik.
ᐧ
On Wed, Sep 19, 2018 at 2:23 AM Courage Angeh <co...@gmail.com>
wrote:
> Thanks for the update. Ed.
>
> If you need an extra hand please let me know Aleks.
>
> On Tue, Sep 18, 2018, 4:27 PM Ed Cable <ed...@mifos.org> wrote:
>
> > Hi all,
> >
> > Just to bring this thread back up to top of everyone's inbox. Aleks
> > estimates he has about a day left to get the demo server up and running
> is
> > hoping to tackle it by tomorrow. Here are the two approaches he suggested
> > now that we have the second VM allocated:
> >
> > - continue with the JUnit based startup sequence of the demo module; the
> > one thing that I would need to change is to split up the startup sequence
> > and recompile
> > - create a set of Docker Compose files to do the startup on the 2 VM
> > instances (maybe I could borrow some stuff from Courage)
> >
> > I would prefer the latter, because you have more fine grained control and
> > it's easier move service instances around... and you don't have to
> > recompile anything.
> >
> > Ed
> >
> > On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <my...@apache.org> wrote:
> >
> > > Hey Aleks,
> > >
> > > I don’t see anything in that log that would explain the behavior you’re
> > > observing. Markus did most of the work on deposits. Markus? Do you have
> > any
> > > ideas?
> > >
> > > Best Regards,
> > > Myrle
> > >
> > >
> > >
> > > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
> > > cheetah@monkeysintown.com>
> > > wrote:
> > >
> > > > @Myrle: with your proposed changes things look a bit better. I've
> > > disabled
> > > > Rhythm as you suggested and get past all microservice startups with a
> > lot
> > > > less exceptions (one is coming repeatedly from Netty native
> transport,
> > > > because epoll is not available).
> > > >
> > > > The one exception (actually it happened on 2 occasions) that is still
> > > > there:
> > > >
> > > > [snip]
> > > >
> > > > 13:33:45.080 [main] INFO
> > > o.a.f.c.d.s.DepositAccountManagementApplication -
> > > > Started DepositAccountManagementApplication in 55.187 seconds (JVM
> > > running
> > > > for 62.772)
> > > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
> > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
> > > > configuration
> > > > Resolving artifact
> > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > Resolving metadata
> > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > from /root/.m2/repository (enhanced)
> > > > Resolved metadata
> > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > from /root/.m2/repository (enhanced)
> > > > Resolved artifact
> > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > from
> > > > /root/.m2/repository (enhanced)
> > > > 13:34:56.800 [DiscoveryClient-0] ERROR
> > c.n.discovery.TimedSupervisorTask
> > > -
> > > > task supervisor timed out
> > > > java.util.concurrent.TimeoutException: null
> > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > at
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 13:34:56.801 [DiscoveryClient-1] ERROR
> > c.n.discovery.TimedSupervisorTask
> > > -
> > > > task supervisor timed out
> > > > java.util.concurrent.TimeoutException: null
> > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > at
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 13:34:56.916 [DiscoveryClient-2] ERROR
> > c.n.discovery.TimedSupervisorTask
> > > -
> > > > task supervisor timed out
> > > > java.util.concurrent.TimeoutException: null
> > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > at
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 13:35:08.586 [main] INFO o.s.c.a.AnnotationConfigApplicationContext
> -
> > > > Refreshing
> > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > :
> > > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context
> hierarchy
> > > > 13:35:10.182 [main] INFO
> > o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
> > > -
> > > > JSR-330 'javax.inject.Inject' annotation found and supported for
> > > autowiring
> > > > 13:35:10.371 [main] INFO
> > > > o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker -
> > Bean
> > > > 'configurationPropertiesRebinderAutoConfiguration' of type [class
> > > >
> > > >
> > >
> >
> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
> > > > is not eligible for getting processed by all BeanPostProcessors (for
> > > > example: not eligible for auto-proxying)
> > > > 13:35:10.435 [background-preinit] INFO
> > > o.h.validator.internal.util.Version
> > > > - HV000001: Hibernate Validator 5.3.0.Final
> > > >
> > > > [/snip]
> > > >
> > > > It looks like the whole startup procedure goes through and I can see
> > some
> > > > kind of "summary" that lists all REST endpoints. But it stays in that
> > > state
> > > > for only a minute or so and then a cascade of shutdowns are triggered
> > > > again. The only thing that stays alive is Eureka which complains
> about
> > > the
> > > > inaccessible services.
> > > >
> > > > Any idea what else we could do other than just ordering an additional
> > > > machine?
> > > >
> > > > Cheers,
> > > >
> > > > Aleks
> > > >
> > > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <my...@apache.org>
> wrote:
> > > >
> > > > > Hey Aleks,
> > > > >
> > > > > It looks like that exception is coming from Rhythm. I suspect it's
> > not
> > > > the
> > > > > cause of your difficulties, since you say your other services are
> > going
> > > > > down too. You can run most of the Fineract server without Rhythm.
> > You
> > > > > could edit the demo-server script to leave it out for the purposes
> of
> > > > > testing. The consequence will be a lack of interest calculations
> for
> > > > > disbursed loans, but for finding out what the real problem is, it'd
> > be
> > > > good
> > > > > to get this one out of the way.
> > > > >
> > > > > Best Regards,
> > > > > Myrle
> > > > >
> > > > > (FYI: I've removed the non-list subscribers from the to. Most of
> > them
> > > > are
> > > > > already subscribed, and those who aren't, don't want to be.)
> > > > >
> > > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
> > > > > cheetah@monkeysintown.com> wrote:
> > > > >
> > > > > > I managed to further improve the startup procedure (aka less
> > > exceptions
> > > > > > than before)... there are still some timeouts, but not as many as
> > > > before.
> > > > > >
> > > > > > The demo server still dies after this exception:
> > > > > >
> > > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > > > configuration
> > > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > > > configuration
> > > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > > > configuration
> > > > > > 23:28:45.990 [pool-9-thread-1] ERROR
> > > > > o.s.s.s.TaskUtils$LoggingErrorHandler
> > > > > > - Unexpected error occurred in scheduled task.
> > > > > > org.springframework.transaction.CannotCreateTransactionException:
> > > Could
> > > > > not
> > > > > > open JPA EntityManager for transaction; nested exception is
> > > > > > javax.persistence.PersistenceException:
> > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > resultset:
> > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > Query is : set autocommit=0
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> > > > > > at
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > Caused by: javax.persistence.PersistenceException:
> > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > resultset:
> > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > Query is : set autocommit=0
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
> > > > > > at
> > > > > >
> > > >
> > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
> > > > > > ... 20 common frames omitted
> > > > > > Caused by: java.sql.SQLNonTransientConnectionException: Could not
> > > read
> > > > > > resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > Query is : set autocommit=0
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
> > > > > > at
> > > > > >
> > > >
> > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
> > > > > > ... 22 common frames omitted
> > > > > > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException:
> Could
> > > not
> > > > > > read resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > Query is : set autocommit=0
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
> > > > > > ... 31 common frames omitted
> > > > > > Caused by: java.io.EOFException: unexpected end of stream, read 0
> > > bytes
> > > > > > from 4
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
> > > > > > ... 33 common frames omitted
> > > > > > 23:29:05.500 [qtp6422064-20] INFO
> > > o.a.f.c.l.c.ServiceExceptionFilter -
> > > > > > Responding with a service error ServiceError{code=409,
> > > > > message='Application
> > > > > > identity-v1 already exists!'}
> > > > > > 23:29:21.282 [Thread-15] INFO
> > > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Closing
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > :
> > > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > 23:29:23.717 [Thread-15] INFO o.s.c.s.DefaultLifecycleProcessor
> -
> > > > > Stopping
> > > > > > beans in phase 2147483647
> > > > > > 23:29:24.267 [Thread-15] INFO o.s.c.s.DefaultLifecycleProcessor
> -
> > > > > Stopping
> > > > > > beans in phase 0
> > > > > > 23:29:24.557 [Thread-15] INFO
> o.s.b.a.e.jmx.EndpointMBeanExporter
> > -
> > > > > > Unregistering JMX-exposed beans on shutdown
> > > > > > 23:29:24.579 [Thread-15] INFO
> o.s.b.a.e.jmx.EndpointMBeanExporter
> > -
> > > > > > Unregistering JMX-exposed beans
> > > > > >
> > > > > > The only thing I've left: start the databases (aka Cassandra) on
> a
> > > > > separate
> > > > > > machine (e. g. on Digital Ocean) and see how this is working
> then.
> > > > > >
> > > > > > ... FYI.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Aleks
> > > > > >
> > > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
> > > > > > cheetah@monkeysintown.com> wrote:
> > > > > >
> > > > > > > @Rajan: it's not running, because the demo server dies... See
> > logs
> > > > from
> > > > > > > previous message...
> > > > > > >
> > > > > > > We just have to figure that one out and we should be good to
> go.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
> > > rajanmaurya154@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Thanks, Alex,
> > > > > > >>
> > > > > > >> I can see web app is running but I can't log in with the
> > > credentials
> > > > > > that
> > > > > > >> you shared earlier
> > > > > > >>
> > > > > > >> Tenant : playground
> > > > > > >> Username : operator
> > > > > > >> Password : init1@l
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >>
> > > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
> > > > > > >> cheetah@monkeysintown.com> wrote:
> > > > > > >>
> > > > > > >> > Hi all,
> > > > > > >> >
> > > > > > >> > ... sweeeet! I got a considerable step further.
> > > > > > >> >
> > > > > > >> > @Markus: thanks for the conversation... gave me an idea how
> to
> > > fix
> > > > > > (most
> > > > > > >> > of) it!
> > > > > > >> >
> > > > > > >> > The problem was - as suspected - the memory (or lack of).
> The
> > VM
> > > > > > >> instance
> > > > > > >> > at Apache has net 30G available. 16G are needed for
> Cassandra
> > > and
> > > > in
> > > > > > my
> > > > > > >> > previous tests I tried to run the demo server with up to
> 16G.
> > > The
> > > > > > >> problem
> > > > > > >> > is that the demo server needs considerably more memory than
> > 16G.
> > > > > > >> According
> > > > > > >> > to my observation it's around 23-24G. Even if you limit the
> > demo
> > > > > > server
> > > > > > >> > memory allocation with "-Xmx" to 12G (for example) it would
> > just
> > > > > > >> continue
> > > > > > >> > to consume more memory.
> > > > > > >> >
> > > > > > >> > My solution: add a swap file. Not ideal in terms of
> > performance,
> > > > but
> > > > > > it
> > > > > > >> > gets the whole thing going... at least mostly (please read
> > on).
> > > > > > >> >
> > > > > > >> > Current status:
> > > > > > >> >
> > > > > > >> > - Cassandra (and all other Docker containers) are still
> > > > running;
> > > > > > >> > Cassandra died in my previous attempts at the latest when
> > the
> > > > > > Teller
> > > > > > >> > application started, but with the swap file it's still
> > > running
> > > > > > >> > - all Spring Boot apps are starting now
> > > > > > >> > - I get once in a while some spurious timeouts (I think
> > when
> > > > > either
> > > > > > >> the
> > > > > > >> > Eureka server is contacted or the config server)
> > > > > > >> > - there are also sometimes exceptions concerning MySQL
> > > > > connections,
> > > > > > >> but
> > > > > > >> > doesn't seem to matter overall
> > > > > > >> > - I could issue an authentication request with Postman to
> > > > > > >> > http://fineract-vm.apache.org:2021/identity/v1; just to
> > test
> > > > if
> > > > > > >> > anything
> > > > > > >> > is responding
> > > > > > >> > - PROBLEM: but the demo server (the Spring Boot apps)
> > > > eventually
> > > > > > dies
> > > > > > >> > after a couple of minutes; I don't think that the memory
> is
> > > the
> > > > > > >> problem
> > > > > > >> > at
> > > > > > >> > this point; I think it's more of a timeout problem
> > concerning
> > > > the
> > > > > > >> > communication with Eureka and/or the config server (maybe
> > > > because
> > > > > > the
> > > > > > >> > system is a bit slow)
> > > > > > >> >
> > > > > > >> > ... and here the memory footprint ("free -h") when
> everything
> > is
> > > > > > >> running,
> > > > > > >> > just FYI:
> > > > > > >> >
> > > > > > >> > total used free shared
> > > > buff/cache
> > > > > > >> > available
> > > > > > >> > Mem: 31G 31G 226M 8.4M
> > > > 169M
> > > > > > >> > 47M
> > > > > > >> > Swap: 15G 6.8G 9.2G
> > > > > > >> >
> > > > > > >> > ... after the Payroll application is started (approx.) I see
> > > these
> > > > > > >> > exceptions appearing in the logs (a whole series of them):
> > > > > > >> >
> > > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > >> > configuration
> > > > > > >> > Resolving artifact
> > > > > > >> >
> > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > >> > Resolving metadata
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > >> > Resolved metadata
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > >> > Resolved artifact
> > > > > > >> >
> > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > from
> > > > > > >> > /root/.m2/repository (enhanced)
> > > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
> > > > > > >> c.n.discovery.TimedSupervisorTask -
> > > > > > >> > task supervisor timed out
> > > > > > >> > java.util.concurrent.TimeoutException: null
> > > > > > >> > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > > >> > at
> > > > > > >>
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > >> > at java.lang.Thread.run(Thread.java:748)
> > > > > > >> >
> > > > > > >> > ... but it still seems to have no problem starting the next
> > app
> > > > > > (Group)
> > > > > > >> and
> > > > > > >> > things look good for a while...
> > > > > > >> >
> > > > > > >> > ... until suddenly it seems that app instances are beginning
> > to
> > > > > > >> shutdown...
> > > > > > >> > and I'm not sure why this happens:
> > > > > > >> >
> > > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > >> > configuration
> > > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
> > > > > > >> c.datastax.driver.core.ClockFactory -
> > > > > > >> > Using native clock to generate timestamps.
> > > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
> > > > > > >> c.datastax.driver.core.CodecRegistry
> > > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
> > > > > > >> java.time.LocalDateTime]
> > > > > > >> > because it collides with previously registered codec
> > > > > > LocalDateTimeCodec
> > > > > > >> > [timestamp <-> java.time.LocalDateTime]
> > > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
> > > > > > c.d.d.c.p.DCAwareRoundRobinPolicy
> > > > > > >> -
> > > > > > >> > Using data-center name 'datacenter1' for
> > DCAwareRoundRobinPolicy
> > > > (if
> > > > > > >> this
> > > > > > >> > is incorrect, please provide the correct datacenter name
> with
> > > > > > >> > DCAwareRoundRobinPolicy constructor)
> > > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
> > > > > > com.datastax.driver.core.Cluster -
> > > > > > >> > New Cassandra host /127.0.0.1:9042 added
> > > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > >> > configuration
> > > > > > >> > 18:27:51.110 [Thread-15] INFO
> > > > > > >> > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> > > Closing
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > >> > :
> > > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > >> > configuration
> > > > > > >> > 18:27:53.373 [Thread-15] INFO
> > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > >> Stopping
> > > > > > >> > beans in phase 2147483647
> > > > > > >> > 18:27:54.331 [Thread-15] INFO
> > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > >> Stopping
> > > > > > >> > beans in phase 0
> > > > > > >> > 18:27:54.736 [Thread-15] INFO
> > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > >> > 18:27:54.740 [Thread-15] INFO
> > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > >> > Unregistering JMX-exposed beans
> > > > > > >> > 18:27:54.798 [Thread-15] INFO
> > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > >> > 18:27:54.800 [Thread-15] INFO
> > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > >> > Unregistering JMX-exposed beans
> > > > > > >> > 18:27:54.937 [Thread-15] INFO
> > > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean - Closing JPA
> > > > > > >> > EntityManagerFactory for persistence unit 'metaPU'
> > > > > > >> > 18:27:58.376 [async-processor-1] INFO
> > > > > > >> o.f.c.internal.util.VersionPrinter -
> > > > > > >> > Flyway 3.2.1 by Boxfuse
> > > > > > >> > 18:27:58.677 [async-processor-1] INFO
> > > > > > >> o.f.c.i.dbsupport.DbSupportFactory -
> > > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL 10.3)
> > > > > > >> > 18:28:00.620 [async-processor-1] INFO
> > > > > > >> o.f.core.internal.command.DbValidate
> > > > > > >> > - Validated 2 migrations (execution time 00:01.169s)
> > > > > > >> > 18:28:00.901 [Thread-15] INFO
> > > o.e.jetty.server.AbstractConnector
> > > > -
> > > > > > >> Stopped
> > > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{0.0.0.0:2020}
> > > > > > >> > 18:28:01.268 [Thread-15] INFO
> > > > o.e.j.server.handler.ContextHandler -
> > > > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
> > > > > > >> >
> > > > > > >> > ... the list is actually a lot longer... in the rest of the
> > log
> > > > you
> > > > > > can
> > > > > > >> see
> > > > > > >> > that one by one all the apps are shutting down.
> > > > > > >> >
> > > > > > >> > The last (error) entries I see are these:
> > > > > > >> >
> > > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR
> > > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not refresh
> > JMS
> > > > > > >> Connection
> > > > > > >> > for destination 'cheques-v1' - retrying using
> > > > > > >> FixedBackOff{interval=5000,
> > > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error
> while
> > > > > > >> attempting to
> > > > > > >> > retrieve a connection from the pool; nested exception is
> > > > > > >> > javax.jms.JMSException: Could not connect to broker URL:
> > > > > > >> > tcp://localhost:61616. Reason: java.net.ConnectException:
> > > > Connection
> > > > > > >> > refused (Connection refused)
> > > > > > >> >
> > > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in the
> > > process
> > > > of
> > > > > > the
> > > > > > >> > previous errors.
> > > > > > >> >
> > > > > > >> > So far from here... any input on the above highly
> appreciated.
> > > > > > Overall I
> > > > > > >> > think we are close.
> > > > > > >> >
> > > > > > >> > Cheers,
> > > > > > >> >
> > > > > > >> > Aleks
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <
> mage@apache.org
> > >
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > One thing we maybe can do, given we are part of the Apache
> > > > Family,
> > > > > > is
> > > > > > >> to
> > > > > > >> > > ask the Cassandra community for some suggestions.
> > > > > > >> > >
> > > > > > >> > > Cheers
> > > > > > >> > >
> > > > > > >> > > Markus
> > > > > > >> > >
> > > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <
> > mage@apache.org
> > > >
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hey all,
> > > > > > >> > > >
> > > > > > >> > > > we are not running Cassandra in a container, we have
> > > dedicated
> > > > > VMs
> > > > > > >> to
> > > > > > >> > run
> > > > > > >> > > > our cluster.
> > > > > > >> > > >
> > > > > > >> > > > Given this we are not the best persons to help here,
> > sorry.
> > > (;
> > > > > > >> > > >
> > > > > > >> > > > Cheers
> > > > > > >> > > >
> > > > > > >> > > > Markus
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
> > > edcable@mifos.org>
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > >> Adding Patric directly to this thread too so he can
> give
> > > his
> > > > > > input
> > > > > > >> > > >> regarding to the challenges with Cassandra.
> > > > > > >> > > >>
> > > > > > >> > > >> Ed
> > > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar Vidakovic <
> > > > > > >> > > >> cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>
> > > > > > >> > > >>> Hi Victor,
> > > > > > >> > > >>>
> > > > > > >> > > >>> .... this is the relevant part of the
> docker-compose.yml
> > > > file:
> > > > > > >> > > >>>
> > > > > > >> > > >>> cassandra0:
> > > > > > >> > > >>> image: cassandra:3.11.1
> > > > > > >> > > >>> container_name: cassandra0
> > > > > > >> > > >>> ports:
> > > > > > >> > > >>> - 9042:9042
> > > > > > >> > > >>> - 9160:9160
> > > > > > >> > > >>> - 7199:7199
> > > > > > >> > > >>> - 8778:8778
> > > > > > >> > > >>> volumes:
> > > > > > >> > > >>> - ./cassandra:/etc/cassandra
> > > > > > >> > > >>> environment:
> > > > > > >> > > >>> - CASSANDRA_START_RPC=true
> > > > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0
> > > > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster
> > > > > > >> > > >>> ulimits:
> > > > > > >> > > >>> memlock: -1
> > > > > > >> > > >>> nproc: 32768
> > > > > > >> > > >>> nofile: 100000
> > > > > > >> > > >>>
> > > > > > >> > > >>> ... and just to be complete... here's the Docker
> service
> > > > > > >> > configuration
> > > > > > >> > > >>> ("/lib/systemd/system/docker.service"):
> > > > > > >> > > >>>
> > > > > > >> > > >>> [Unit]
> > > > > > >> > > >>> Description=Docker Application Container Engine
> > > > > > >> > > >>> Documentation=https://docs.docker.com
> > > > > > >> > > >>> After=network-online.target docker.socket
> > > firewalld.service
> > > > > > >> > > >>> Wants=network-online.target
> > > > > > >> > > >>> Requires=docker.socket
> > > > > > >> > > >>>
> > > > > > >> > > >>> [Service]
> > > > > > >> > > >>> Type=notify
> > > > > > >> > > >>> # the default is not to use systemd for cgroups
> because
> > > the
> > > > > > >> delegate
> > > > > > >> > > >>> issues
> > > > > > >> > > >>> still
> > > > > > >> > > >>> # exists and systemd currently does not support the
> > cgroup
> > > > > > feature
> > > > > > >> > set
> > > > > > >> > > >>> required
> > > > > > >> > > >>> # for containers run by docker
> > > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
> > > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
> > > > > > >> > > >>> LimitNOFILE=1048576
> > > > > > >> > > >>> # Having non-zero Limit*s causes performance problems
> > due
> > > to
> > > > > > >> > accounting
> > > > > > >> > > >>> overhead
> > > > > > >> > > >>> # in the kernel. We recommend using cgroups to do
> > > > > > container-local
> > > > > > >> > > >>> accounting.
> > > > > > >> > > >>> LimitNPROC=infinity
> > > > > > >> > > >>> LimitCORE=infinity
> > > > > > >> > > >>> LimitMEMLOCK=infinity
> > > > > > >> > > >>> # Uncomment TasksMax if your systemd version supports
> > it.
> > > > > > >> > > >>> # Only systemd 226 and above support this version.
> > > > > > >> > > >>> TasksMax=infinity
> > > > > > >> > > >>> TimeoutStartSec=0
> > > > > > >> > > >>> # set delegate yes so that systemd does not reset the
> > > > cgroups
> > > > > of
> > > > > > >> > docker
> > > > > > >> > > >>> containers
> > > > > > >> > > >>> Delegate=yes
> > > > > > >> > > >>> # kill only the docker process, not all processes in
> the
> > > > > cgroup
> > > > > > >> > > >>> KillMode=process
> > > > > > >> > > >>> # restart the docker process if it exits prematurely
> > > > > > >> > > >>> Restart=on-failure
> > > > > > >> > > >>> StartLimitBurst=3
> > > > > > >> > > >>> StartLimitInterval=60s
> > > > > > >> > > >>>
> > > > > > >> > > >>> [Install]
> > > > > > >> > > >>> WantedBy=multi-user.target
> > > > > > >> > > >>>
> > > > > > >> > > >>> ... the one tweak I did there is to set LimitMEMLOCK
> to
> > > > > > >> infinity...
> > > > > > >> > do
> > > > > > >> > > >>> you
> > > > > > >> > > >>> think the problems are file handle related? Should I
> set
> > > > > > >> LimitNOFILE
> > > > > > >> > > also
> > > > > > >> > > >>> to infinity?
> > > > > > >> > > >>>
> > > > > > >> > > >>> Appreciate the help.
> > > > > > >> > > >>>
> > > > > > >> > > >>> Cheers,
> > > > > > >> > > >>>
> > > > > > >> > > >>> Aleks
> > > > > > >> > > >>>
> > > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
> > > > > > >> > > >>> victor.romero@fintecheando.mx>
> > > > > > >> > > >>> wrote:
> > > > > > >> > > >>>
> > > > > > >> > > >>> > Hi Aleks,
> > > > > > >> > > >>> >
> > > > > > >> > > >>> > Can you share the ulimits flags that the cassandra's
> > > > > container
> > > > > > >> is
> > > > > > >> > > using
> > > > > > >> > > >>> > while it is running?
> > > > > > >> > > >>> >
> > > > > > >> > > >>> > The values are being set in the composer file? Or in
> > the
> > > > > > >> upstart or
> > > > > > >> > > >>> > systemd docker's deamon config files?
> > > > > > >> > > >>> >
> > > > > > >> > > >>> >
> > > > > > >> > > >>> >
> > > > > > >> > > >>> > Enviado desde TypeApp
> > > > > > >> > > >>> >
> > > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM, Aleksandar
> > > > > Vidakovic <
> > > > > > >> > > >>> > cheetah@monkeysintown.com> escribió:
> > > > > > >> > > >>> > >... and I should have attached the Cassandra log
> > > dump...
> > > > so
> > > > > > >> here
> > > > > > >> > it
> > > > > > >> > > >>> > >is...
> > > > > > >> > > >>> > >
> > > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar
> Vidakovic
> > <
> > > > > > >> > > >>> > >cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>> > >
> > > > > > >> > > >>> > >> Hi all,
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> ... as you might have noticed I did a couple of
> > > > restarts
> > > > > > >> > today...
> > > > > > >> > > >>> the
> > > > > > >> > > >>> > >> problem I am facing now: I can't get Cassandra to
> > run
> > > > in
> > > > > a
> > > > > > >> > stable
> > > > > > >> > > >>> > >way.
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> Things look quite OK for a while when running the
> > > demo
> > > > > > >> server,
> > > > > > >> > but
> > > > > > >> > > >>> > >then
> > > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts
> > > > misbehaving
> > > > > > with
> > > > > > >> > the
> > > > > > >> > > >>> > >deposit
> > > > > > >> > > >>> > >> microservice startup, sometimes with portfolio or
> > > > > teller).
> > > > > > I
> > > > > > >> > tried
> > > > > > >> > > >>> to
> > > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set some
> > > > Docker
> > > > > > >> limits
> > > > > > >> > to
> > > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe
> someone
> > > can
> > > > > > help
> > > > > > >> out
> > > > > > >> > > >>> > >here? Is
> > > > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB of
> > > memory?
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> Other than that the setup would be ready to go...
> > we
> > > > just
> > > > > > >> need
> > > > > > >> > to
> > > > > > >> > > >>> get
> > > > > > >> > > >>> > >the
> > > > > > >> > > >>> > >> database running more reliably.
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> Cheers,
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> Aleks
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar
> > Vidakovic
> > > <
> > > > > > >> > > >>> > >> cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed to add
> > > some
> > > > > > >> > additional
> > > > > > >> > > >>> > >reverse
> > > > > > >> > > >>> > >>> proxy configuration for the web UI to work)...
> and
> > > > there
> > > > > > >> were
> > > > > > >> > > more
> > > > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
> > > > > > >> > > >>> > >>>
> > > > > > >> > > >>> > >>> Just FYI
> > > > > > >> > > >>> > >>>
> > > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
> > > Vidakovic <
> > > > > > >> > > >>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>> > >>>
> > > > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can get
> > back
> > > > to
> > > > > > >> testing
> > > > > > >> > > >>> only
> > > > > > >> > > >>> > >>>> later tonight.
> > > > > > >> > > >>> > >>>>
> > > > > > >> > > >>> > >>>> Cheers
> > > > > > >> > > >>> > >>>>
> > > > > > >> > > >>> > >>>>
> > > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya
> > > > > > >> > > >>> > ><ra...@gmail.com>
> > > > > > >> > > >>> > >>>> wrote:
> > > > > > >> > > >>> > >>>>
> > > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test after 30
> > min.
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>> Big thanks for this 🙂
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM Aleksandar
> > > > Vidakovic <
> > > > > > >> > > >>> > >>>>> cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted.
> > > > > > >> > > >>> > >>>>> >
> > > > > > >> > > >>> > >>>>> > ... and as I said: the services are still
> > > > > starting...
> > > > > > >> and
> > > > > > >> > the
> > > > > > >> > > >>> > >best
> > > > > > >> > > >>> > >>>>> estimate
> > > > > > >> > > >>> > >>>>> > I have right now (as already mentioned):
> 30min
> > > > > > >> > > >>> > >>>>> >
> > > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan
> Maurya <
> > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > > >> > > >>> > >>>>> > wrote:
> > > > > > >> > > >>> > >>>>> >
> > > > > > >> > > >>> > >>>>> > > [image: image.png]
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan
> > Maurya <
> > > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > > >> > > >>> > >>>>> > > wrote:
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >> [image: image.png]
> > > > > > >> > > >>> > >>>>> > >> I am getting this.
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM
> Aleksandar
> > > > > > Vidakovic
> > > > > > >> <
> > > > > > >> > > >>> > >>>>> > >> cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>> Hi all,
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost)
> ready...
> > > it
> > > > > took
> > > > > > >> me a
> > > > > > >> > > >>> > >moment
> > > > > > >> > > >>> > >>>>> and a
> > > > > > >> > > >>> > >>>>> > >>> couple
> > > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot
> > > > failures...
> > > > > > the
> > > > > > >> > > >>> services
> > > > > > >> > > >>> > >are
> > > > > > >> > > >>> > >>>>> quite
> > > > > > >> > > >>> > >>>>> > >>> resource hungry and default settings
> won't
> > > do
> > > > > it.
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> The services are still starting and this
> > > will
> > > > > > take a
> > > > > > >> > > while
> > > > > > >> > > >>> > >(my
> > > > > > >> > > >>> > >>>>> best
> > > > > > >> > > >>> > >>>>> > guess
> > > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org/login
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Credentials:
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Tenant : playground
> > > > > > >> > > >>> > >>>>> > >>> Username : operator
> > > > > > >> > > >>> > >>>>> > >>> Password : init1@l
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to
> compile
> > > this
> > > > > app
> > > > > > >> is
> > > > > > >> > > with
> > > > > > >> > > >>> > >NodeJS
> > > > > > >> > > >>> > >>>>> > 8.11.1
> > > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't
> work).
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Identity Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2021/identity/v1
> > > > > > >> > > >>> > >>>>> > >>> Office Service:
> > > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
> > > > > > >> > > >>> > >>>>> > >>> Customer Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2024/customer/v1
> > > > > > >> > > >>> > >>>>> > >>> Accounting Service:
> > > > > > >> > > >>> > >>>>>
> > http://fineract-vm.apache.org:2025/accounting/v1
> > > > > > >> > > >>> > >>>>> > >>> Portfolio Service:
> > > > > > >> > > >>> > >>>>>
> http://fineract-vm.apache.org:2026/portfolio/v1
> > > > > > >> > > >>> > >>>>> > >>> Deposit Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1
> > > > > > >> > > >>> > >>>>> > >>> Teller Service:
> > > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
> > > > > > >> > > >>> > >>>>> > >>> Reporting Service:
> > > > > > >> > > >>> > >>>>>
> http://fineract-vm.apache.org:2029/reporting/v1
> > > > > > >> > > >>> > >>>>> > >>> Cheque Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1
> > > > > > >> > > >>> > >>>>> > >>> Payroll Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes
> quite
> > a
> > > > > > >> while... if
> > > > > > >> > > you
> > > > > > >> > > >>> > >>>>> encounter
> > > > > > >> > > >>> > >>>>> > >>> connection problems retry a couple of
> > > minutes
> > > > > > later
> > > > > > >> to
> > > > > > >> > > >>> > >ensure I
> > > > > > >> > > >>> > >>>>> am not
> > > > > > >> > > >>> > >>>>> > >>> currently working on something.
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the backend
> > > > services
> > > > > > >> once
> > > > > > >> > per
> > > > > > >> > > >>> > >day or
> > > > > > >> > > >>> > >>>>> so to
> > > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to
> figure
> > > out
> > > > > how
> > > > > > we
> > > > > > >> > want
> > > > > > >> > > >>> to
> > > > > > >> > > >>> > >>>>> handle
> > > > > > >> > > >>> > >>>>> > this
> > > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where to put
> > > this
> > > > > demo
> > > > > > >> > server
> > > > > > >> > > >>> > >>>>> configuration
> > > > > > >> > > >>> > >>>>> > >>> then let me know here.
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any
> problems
> > (I
> > > > > have
> > > > > > >> not
> > > > > > >> > > >>> > >extensively
> > > > > > >> > > >>> > >>>>> > tested
> > > > > > >> > > >>> > >>>>> > >>> it
> > > > > > >> > > >>> > >>>>> > >>> yet).
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Cheers,
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Aleks
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM
> > Aleksandar
> > > > > > >> Vidakovic <
> > > > > > >> > > >>> > >>>>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem to be
> > > > starting
> > > > > > fine
> > > > > > >> > > now...
> > > > > > >> > > >>> > >just
> > > > > > >> > > >>> > >>>>> have
> > > > > > >> > > >>> > >>>>> > to
> > > > > > >> > > >>> > >>>>> > >>> > work on a couple of improvements on
> the
> > > > config
> > > > > > >> files.
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > Currently having another stab to get
> the
> > > > > webapp
> > > > > > >> > > running.
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > BTW: it should be fairly easy to adapt
> > the
> > > > > > >> simplistic
> > > > > > >> > > >>> > >Docker
> > > > > > >> > > >>> > >>>>> Compose
> > > > > > >> > > >>> > >>>>> > >>> file
> > > > > > >> > > >>> > >>>>> > >>> > for a CI pipeline.
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > Will keep you posted.
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > Cheers,
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > Aleks
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > On Mon, Jun 11, 2018 at 9:41 AM
> > Aleksandar
> > > > > > >> Vidakovic
> > > > > > >> > <
> > > > > > >> > > >>> > >>>>> > >>> > cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> >> I am still working on it... Will ping
> > you
> > > > > here
> > > > > > >> when
> > > > > > >> > > it's
> > > > > > >> > > >>> > >ready
> > > > > > >> > > >>> > >>>>> (hope
> > > > > > >> > > >>> > >>>>> > >>> >> soon).
> > > > > > >> > > >>> > >>>>> > >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >> On Mon, Jun 11, 2018, 9:40 AM Rajan
> > > Maurya
> > > > <
> > > > > > >> > > >>> > >>>>> > rajanmaurya154@gmail.com>
> > > > > > >> > > >>> > >>>>> > >>> >> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> Hi Alex,
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>> Is the demo server up, If yes then
> > > what's
> > > > > the
> > > > > > >> base
> > > > > > >> > > url
> > > > > > >> > > >>> > >so I
> > > > > > >> > > >>> > >>>>> can
> > > > > > >> > > >>> > >>>>> > >>> access.
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>> On Fri, Jun 8, 2018 at 2:03 PM Rajan
> > > > Maurya
> > > > > <
> > > > > > >> > > >>> > >>>>> > >>> rajanmaurya154@gmail.com>
> > > > > > >> > > >>> > >>>>> > >>> >>> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>> > Hi Alex,
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> > This is really great news, I will
> > wait
> > > > > until
> > > > > > >> > then.
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> > Thanks
> > > > > > >> > > >>> > >>>>> > >>> >>> > Rajan Maurya
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> > On Fri, Jun 8, 2018 at 11:36 AM
> > > > Aleksandar
> > > > > > >> > > Vidakovic
> > > > > > >> > > >>> <
> > > > > > >> > > >>> > >>>>> > >>> >>> > cheetah@monkeysintown.com> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> >> Hi Rajan,
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> My plan is to get this up and
> > running
> > > > > until
> > > > > > >> > > Monday.
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> Cheers,
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> Aleks
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> On Fri, Jun 8, 2018, 7:23 AM
> Rajan
> > > > > Maurya <
> > > > > > >> > > >>> > >>>>> > >>> rajanmaurya154@gmail.com>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> > Hi Markus, Mark, Myrle, Alex,
> Ed
> > > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > > >> > > >>> > >>>>> > >>> >>> >> > I am mentoring
> fineract-cn-mobile
> > > > GSoC
> > > > > > >> > project.
> > > > > > >> > > I
> > > > > > >> > > >>> > >am
> > > > > > >> > > >>> > >>>>> waiting
> > > > > > >> > > >>> > >>>>> > for
>
Re: Status of Demo Server on Apache VM Infrastructure
Posted by Courage Angeh <co...@gmail.com>.
Thanks for the update. Ed.
If you need an extra hand please let me know Aleks.
On Tue, Sep 18, 2018, 4:27 PM Ed Cable <ed...@mifos.org> wrote:
> Hi all,
>
> Just to bring this thread back up to top of everyone's inbox. Aleks
> estimates he has about a day left to get the demo server up and running is
> hoping to tackle it by tomorrow. Here are the two approaches he suggested
> now that we have the second VM allocated:
>
> - continue with the JUnit based startup sequence of the demo module; the
> one thing that I would need to change is to split up the startup sequence
> and recompile
> - create a set of Docker Compose files to do the startup on the 2 VM
> instances (maybe I could borrow some stuff from Courage)
>
> I would prefer the latter, because you have more fine grained control and
> it's easier move service instances around... and you don't have to
> recompile anything.
>
> Ed
>
> On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <my...@apache.org> wrote:
>
> > Hey Aleks,
> >
> > I don’t see anything in that log that would explain the behavior you’re
> > observing. Markus did most of the work on deposits. Markus? Do you have
> any
> > ideas?
> >
> > Best Regards,
> > Myrle
> >
> >
> >
> > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
> > cheetah@monkeysintown.com>
> > wrote:
> >
> > > @Myrle: with your proposed changes things look a bit better. I've
> > disabled
> > > Rhythm as you suggested and get past all microservice startups with a
> lot
> > > less exceptions (one is coming repeatedly from Netty native transport,
> > > because epoll is not available).
> > >
> > > The one exception (actually it happened on 2 occasions) that is still
> > > there:
> > >
> > > [snip]
> > >
> > > 13:33:45.080 [main] INFO
> > o.a.f.c.d.s.DepositAccountManagementApplication -
> > > Started DepositAccountManagementApplication in 55.187 seconds (JVM
> > running
> > > for 62.772)
> > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
> > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
> > > configuration
> > > Resolving artifact
> > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > Resolving metadata
> > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > from /root/.m2/repository (enhanced)
> > > Resolved metadata
> > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > from /root/.m2/repository (enhanced)
> > > Resolved artifact
> > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> from
> > > /root/.m2/repository (enhanced)
> > > 13:34:56.800 [DiscoveryClient-0] ERROR
> c.n.discovery.TimedSupervisorTask
> > -
> > > task supervisor timed out
> > > java.util.concurrent.TimeoutException: null
> > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > at
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > 13:34:56.801 [DiscoveryClient-1] ERROR
> c.n.discovery.TimedSupervisorTask
> > -
> > > task supervisor timed out
> > > java.util.concurrent.TimeoutException: null
> > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > at
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > 13:34:56.916 [DiscoveryClient-2] ERROR
> c.n.discovery.TimedSupervisorTask
> > -
> > > task supervisor timed out
> > > java.util.concurrent.TimeoutException: null
> > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > at
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > 13:35:08.586 [main] INFO o.s.c.a.AnnotationConfigApplicationContext -
> > > Refreshing
> > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > :
> > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context hierarchy
> > > 13:35:10.182 [main] INFO
> o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
> > -
> > > JSR-330 'javax.inject.Inject' annotation found and supported for
> > autowiring
> > > 13:35:10.371 [main] INFO
> > > o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker -
> Bean
> > > 'configurationPropertiesRebinderAutoConfiguration' of type [class
> > >
> > >
> >
> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
> > > is not eligible for getting processed by all BeanPostProcessors (for
> > > example: not eligible for auto-proxying)
> > > 13:35:10.435 [background-preinit] INFO
> > o.h.validator.internal.util.Version
> > > - HV000001: Hibernate Validator 5.3.0.Final
> > >
> > > [/snip]
> > >
> > > It looks like the whole startup procedure goes through and I can see
> some
> > > kind of "summary" that lists all REST endpoints. But it stays in that
> > state
> > > for only a minute or so and then a cascade of shutdowns are triggered
> > > again. The only thing that stays alive is Eureka which complains about
> > the
> > > inaccessible services.
> > >
> > > Any idea what else we could do other than just ordering an additional
> > > machine?
> > >
> > > Cheers,
> > >
> > > Aleks
> > >
> > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <my...@apache.org> wrote:
> > >
> > > > Hey Aleks,
> > > >
> > > > It looks like that exception is coming from Rhythm. I suspect it's
> not
> > > the
> > > > cause of your difficulties, since you say your other services are
> going
> > > > down too. You can run most of the Fineract server without Rhythm.
> You
> > > > could edit the demo-server script to leave it out for the purposes of
> > > > testing. The consequence will be a lack of interest calculations for
> > > > disbursed loans, but for finding out what the real problem is, it'd
> be
> > > good
> > > > to get this one out of the way.
> > > >
> > > > Best Regards,
> > > > Myrle
> > > >
> > > > (FYI: I've removed the non-list subscribers from the to. Most of
> them
> > > are
> > > > already subscribed, and those who aren't, don't want to be.)
> > > >
> > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
> > > > cheetah@monkeysintown.com> wrote:
> > > >
> > > > > I managed to further improve the startup procedure (aka less
> > exceptions
> > > > > than before)... there are still some timeouts, but not as many as
> > > before.
> > > > >
> > > > > The demo server still dies after this exception:
> > > > >
> > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> via
> > > > > configuration
> > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> via
> > > > > configuration
> > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> via
> > > > > configuration
> > > > > 23:28:45.990 [pool-9-thread-1] ERROR
> > > > o.s.s.s.TaskUtils$LoggingErrorHandler
> > > > > - Unexpected error occurred in scheduled task.
> > > > > org.springframework.transaction.CannotCreateTransactionException:
> > Could
> > > > not
> > > > > open JPA EntityManager for transaction; nested exception is
> > > > > javax.persistence.PersistenceException:
> > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > resultset:
> > > > > unexpected end of stream, read 0 bytes from 4
> > > > > Query is : set autocommit=0
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > Caused by: javax.persistence.PersistenceException:
> > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > resultset:
> > > > > unexpected end of stream, read 0 bytes from 4
> > > > > Query is : set autocommit=0
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
> > > > > at
> > > > >
> > >
> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
> > > > > ... 20 common frames omitted
> > > > > Caused by: java.sql.SQLNonTransientConnectionException: Could not
> > read
> > > > > resultset: unexpected end of stream, read 0 bytes from 4
> > > > > Query is : set autocommit=0
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
> > > > > at
> > > > >
> > >
> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
> > > > > ... 22 common frames omitted
> > > > > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
> > not
> > > > > read resultset: unexpected end of stream, read 0 bytes from 4
> > > > > Query is : set autocommit=0
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
> > > > > ... 31 common frames omitted
> > > > > Caused by: java.io.EOFException: unexpected end of stream, read 0
> > bytes
> > > > > from 4
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
> > > > > ... 33 common frames omitted
> > > > > 23:29:05.500 [qtp6422064-20] INFO
> > o.a.f.c.l.c.ServiceExceptionFilter -
> > > > > Responding with a service error ServiceError{code=409,
> > > > message='Application
> > > > > identity-v1 already exists!'}
> > > > > 23:29:21.282 [Thread-15] INFO
> > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Closing
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > :
> > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
> > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > 23:29:23.717 [Thread-15] INFO o.s.c.s.DefaultLifecycleProcessor -
> > > > Stopping
> > > > > beans in phase 2147483647
> > > > > 23:29:24.267 [Thread-15] INFO o.s.c.s.DefaultLifecycleProcessor -
> > > > Stopping
> > > > > beans in phase 0
> > > > > 23:29:24.557 [Thread-15] INFO o.s.b.a.e.jmx.EndpointMBeanExporter
> -
> > > > > Unregistering JMX-exposed beans on shutdown
> > > > > 23:29:24.579 [Thread-15] INFO o.s.b.a.e.jmx.EndpointMBeanExporter
> -
> > > > > Unregistering JMX-exposed beans
> > > > >
> > > > > The only thing I've left: start the databases (aka Cassandra) on a
> > > > separate
> > > > > machine (e. g. on Digital Ocean) and see how this is working then.
> > > > >
> > > > > ... FYI.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Aleks
> > > > >
> > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
> > > > > cheetah@monkeysintown.com> wrote:
> > > > >
> > > > > > @Rajan: it's not running, because the demo server dies... See
> logs
> > > from
> > > > > > previous message...
> > > > > >
> > > > > > We just have to figure that one out and we should be good to go.
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
> > rajanmaurya154@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Thanks, Alex,
> > > > > >>
> > > > > >> I can see web app is running but I can't log in with the
> > credentials
> > > > > that
> > > > > >> you shared earlier
> > > > > >>
> > > > > >> Tenant : playground
> > > > > >> Username : operator
> > > > > >> Password : init1@l
> > > > > >>
> > > > > >> Thanks
> > > > > >>
> > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
> > > > > >> cheetah@monkeysintown.com> wrote:
> > > > > >>
> > > > > >> > Hi all,
> > > > > >> >
> > > > > >> > ... sweeeet! I got a considerable step further.
> > > > > >> >
> > > > > >> > @Markus: thanks for the conversation... gave me an idea how to
> > fix
> > > > > (most
> > > > > >> > of) it!
> > > > > >> >
> > > > > >> > The problem was - as suspected - the memory (or lack of). The
> VM
> > > > > >> instance
> > > > > >> > at Apache has net 30G available. 16G are needed for Cassandra
> > and
> > > in
> > > > > my
> > > > > >> > previous tests I tried to run the demo server with up to 16G.
> > The
> > > > > >> problem
> > > > > >> > is that the demo server needs considerably more memory than
> 16G.
> > > > > >> According
> > > > > >> > to my observation it's around 23-24G. Even if you limit the
> demo
> > > > > server
> > > > > >> > memory allocation with "-Xmx" to 12G (for example) it would
> just
> > > > > >> continue
> > > > > >> > to consume more memory.
> > > > > >> >
> > > > > >> > My solution: add a swap file. Not ideal in terms of
> performance,
> > > but
> > > > > it
> > > > > >> > gets the whole thing going... at least mostly (please read
> on).
> > > > > >> >
> > > > > >> > Current status:
> > > > > >> >
> > > > > >> > - Cassandra (and all other Docker containers) are still
> > > running;
> > > > > >> > Cassandra died in my previous attempts at the latest when
> the
> > > > > Teller
> > > > > >> > application started, but with the swap file it's still
> > running
> > > > > >> > - all Spring Boot apps are starting now
> > > > > >> > - I get once in a while some spurious timeouts (I think
> when
> > > > either
> > > > > >> the
> > > > > >> > Eureka server is contacted or the config server)
> > > > > >> > - there are also sometimes exceptions concerning MySQL
> > > > connections,
> > > > > >> but
> > > > > >> > doesn't seem to matter overall
> > > > > >> > - I could issue an authentication request with Postman to
> > > > > >> > http://fineract-vm.apache.org:2021/identity/v1; just to
> test
> > > if
> > > > > >> > anything
> > > > > >> > is responding
> > > > > >> > - PROBLEM: but the demo server (the Spring Boot apps)
> > > eventually
> > > > > dies
> > > > > >> > after a couple of minutes; I don't think that the memory is
> > the
> > > > > >> problem
> > > > > >> > at
> > > > > >> > this point; I think it's more of a timeout problem
> concerning
> > > the
> > > > > >> > communication with Eureka and/or the config server (maybe
> > > because
> > > > > the
> > > > > >> > system is a bit slow)
> > > > > >> >
> > > > > >> > ... and here the memory footprint ("free -h") when everything
> is
> > > > > >> running,
> > > > > >> > just FYI:
> > > > > >> >
> > > > > >> > total used free shared
> > > buff/cache
> > > > > >> > available
> > > > > >> > Mem: 31G 31G 226M 8.4M
> > > 169M
> > > > > >> > 47M
> > > > > >> > Swap: 15G 6.8G 9.2G
> > > > > >> >
> > > > > >> > ... after the Payroll application is started (approx.) I see
> > these
> > > > > >> > exceptions appearing in the logs (a whole series of them):
> > > > > >> >
> > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > >> > configuration
> > > > > >> > Resolving artifact
> > > > > >> >
> > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > >> > Resolving metadata
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > >> > from /root/.m2/repository (enhanced)
> > > > > >> > Resolved metadata
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > >> > from /root/.m2/repository (enhanced)
> > > > > >> > Resolved artifact
> > > > > >> >
> > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > from
> > > > > >> > /root/.m2/repository (enhanced)
> > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
> > > > > >> c.n.discovery.TimedSupervisorTask -
> > > > > >> > task supervisor timed out
> > > > > >> > java.util.concurrent.TimeoutException: null
> > > > > >> > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > >> > at
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > >> > at
> > > > > >>
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > >> > at
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > >> > at
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > >> > at
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > >> > at
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > >> > at java.lang.Thread.run(Thread.java:748)
> > > > > >> >
> > > > > >> > ... but it still seems to have no problem starting the next
> app
> > > > > (Group)
> > > > > >> and
> > > > > >> > things look good for a while...
> > > > > >> >
> > > > > >> > ... until suddenly it seems that app instances are beginning
> to
> > > > > >> shutdown...
> > > > > >> > and I'm not sure why this happens:
> > > > > >> >
> > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > >> > configuration
> > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
> > > > > >> c.datastax.driver.core.ClockFactory -
> > > > > >> > Using native clock to generate timestamps.
> > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
> > > > > >> c.datastax.driver.core.CodecRegistry
> > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
> > > > > >> java.time.LocalDateTime]
> > > > > >> > because it collides with previously registered codec
> > > > > LocalDateTimeCodec
> > > > > >> > [timestamp <-> java.time.LocalDateTime]
> > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
> > > > > c.d.d.c.p.DCAwareRoundRobinPolicy
> > > > > >> -
> > > > > >> > Using data-center name 'datacenter1' for
> DCAwareRoundRobinPolicy
> > > (if
> > > > > >> this
> > > > > >> > is incorrect, please provide the correct datacenter name with
> > > > > >> > DCAwareRoundRobinPolicy constructor)
> > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
> > > > > com.datastax.driver.core.Cluster -
> > > > > >> > New Cassandra host /127.0.0.1:9042 added
> > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > >> > configuration
> > > > > >> > 18:27:51.110 [Thread-15] INFO
> > > > > >> > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> > Closing
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > >> > :
> > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> endpoints
> > > via
> > > > > >> > configuration
> > > > > >> > 18:27:53.373 [Thread-15] INFO
> > o.s.c.s.DefaultLifecycleProcessor -
> > > > > >> Stopping
> > > > > >> > beans in phase 2147483647
> > > > > >> > 18:27:54.331 [Thread-15] INFO
> > o.s.c.s.DefaultLifecycleProcessor -
> > > > > >> Stopping
> > > > > >> > beans in phase 0
> > > > > >> > 18:27:54.736 [Thread-15] INFO
> > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > >> > 18:27:54.740 [Thread-15] INFO
> > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > >> > Unregistering JMX-exposed beans
> > > > > >> > 18:27:54.798 [Thread-15] INFO
> > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > >> > 18:27:54.800 [Thread-15] INFO
> > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > >> > Unregistering JMX-exposed beans
> > > > > >> > 18:27:54.937 [Thread-15] INFO
> > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean - Closing JPA
> > > > > >> > EntityManagerFactory for persistence unit 'metaPU'
> > > > > >> > 18:27:58.376 [async-processor-1] INFO
> > > > > >> o.f.c.internal.util.VersionPrinter -
> > > > > >> > Flyway 3.2.1 by Boxfuse
> > > > > >> > 18:27:58.677 [async-processor-1] INFO
> > > > > >> o.f.c.i.dbsupport.DbSupportFactory -
> > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL 10.3)
> > > > > >> > 18:28:00.620 [async-processor-1] INFO
> > > > > >> o.f.core.internal.command.DbValidate
> > > > > >> > - Validated 2 migrations (execution time 00:01.169s)
> > > > > >> > 18:28:00.901 [Thread-15] INFO
> > o.e.jetty.server.AbstractConnector
> > > -
> > > > > >> Stopped
> > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{0.0.0.0:2020}
> > > > > >> > 18:28:01.268 [Thread-15] INFO
> > > o.e.j.server.handler.ContextHandler -
> > > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
> > > > > >> >
> > > > > >> > ... the list is actually a lot longer... in the rest of the
> log
> > > you
> > > > > can
> > > > > >> see
> > > > > >> > that one by one all the apps are shutting down.
> > > > > >> >
> > > > > >> > The last (error) entries I see are these:
> > > > > >> >
> > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR
> > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not refresh
> JMS
> > > > > >> Connection
> > > > > >> > for destination 'cheques-v1' - retrying using
> > > > > >> FixedBackOff{interval=5000,
> > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error while
> > > > > >> attempting to
> > > > > >> > retrieve a connection from the pool; nested exception is
> > > > > >> > javax.jms.JMSException: Could not connect to broker URL:
> > > > > >> > tcp://localhost:61616. Reason: java.net.ConnectException:
> > > Connection
> > > > > >> > refused (Connection refused)
> > > > > >> >
> > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in the
> > process
> > > of
> > > > > the
> > > > > >> > previous errors.
> > > > > >> >
> > > > > >> > So far from here... any input on the above highly appreciated.
> > > > > Overall I
> > > > > >> > think we are close.
> > > > > >> >
> > > > > >> > Cheers,
> > > > > >> >
> > > > > >> > Aleks
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <mage@apache.org
> >
> > > > wrote:
> > > > > >> >
> > > > > >> > > One thing we maybe can do, given we are part of the Apache
> > > Family,
> > > > > is
> > > > > >> to
> > > > > >> > > ask the Cassandra community for some suggestions.
> > > > > >> > >
> > > > > >> > > Cheers
> > > > > >> > >
> > > > > >> > > Markus
> > > > > >> > >
> > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <
> mage@apache.org
> > >
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hey all,
> > > > > >> > > >
> > > > > >> > > > we are not running Cassandra in a container, we have
> > dedicated
> > > > VMs
> > > > > >> to
> > > > > >> > run
> > > > > >> > > > our cluster.
> > > > > >> > > >
> > > > > >> > > > Given this we are not the best persons to help here,
> sorry.
> > (;
> > > > > >> > > >
> > > > > >> > > > Cheers
> > > > > >> > > >
> > > > > >> > > > Markus
> > > > > >> > > >
> > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
> > edcable@mifos.org>
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > >> Adding Patric directly to this thread too so he can give
> > his
> > > > > input
> > > > > >> > > >> regarding to the challenges with Cassandra.
> > > > > >> > > >>
> > > > > >> > > >> Ed
> > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar Vidakovic <
> > > > > >> > > >> cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>
> > > > > >> > > >>> Hi Victor,
> > > > > >> > > >>>
> > > > > >> > > >>> .... this is the relevant part of the docker-compose.yml
> > > file:
> > > > > >> > > >>>
> > > > > >> > > >>> cassandra0:
> > > > > >> > > >>> image: cassandra:3.11.1
> > > > > >> > > >>> container_name: cassandra0
> > > > > >> > > >>> ports:
> > > > > >> > > >>> - 9042:9042
> > > > > >> > > >>> - 9160:9160
> > > > > >> > > >>> - 7199:7199
> > > > > >> > > >>> - 8778:8778
> > > > > >> > > >>> volumes:
> > > > > >> > > >>> - ./cassandra:/etc/cassandra
> > > > > >> > > >>> environment:
> > > > > >> > > >>> - CASSANDRA_START_RPC=true
> > > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0
> > > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster
> > > > > >> > > >>> ulimits:
> > > > > >> > > >>> memlock: -1
> > > > > >> > > >>> nproc: 32768
> > > > > >> > > >>> nofile: 100000
> > > > > >> > > >>>
> > > > > >> > > >>> ... and just to be complete... here's the Docker service
> > > > > >> > configuration
> > > > > >> > > >>> ("/lib/systemd/system/docker.service"):
> > > > > >> > > >>>
> > > > > >> > > >>> [Unit]
> > > > > >> > > >>> Description=Docker Application Container Engine
> > > > > >> > > >>> Documentation=https://docs.docker.com
> > > > > >> > > >>> After=network-online.target docker.socket
> > firewalld.service
> > > > > >> > > >>> Wants=network-online.target
> > > > > >> > > >>> Requires=docker.socket
> > > > > >> > > >>>
> > > > > >> > > >>> [Service]
> > > > > >> > > >>> Type=notify
> > > > > >> > > >>> # the default is not to use systemd for cgroups because
> > the
> > > > > >> delegate
> > > > > >> > > >>> issues
> > > > > >> > > >>> still
> > > > > >> > > >>> # exists and systemd currently does not support the
> cgroup
> > > > > feature
> > > > > >> > set
> > > > > >> > > >>> required
> > > > > >> > > >>> # for containers run by docker
> > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
> > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
> > > > > >> > > >>> LimitNOFILE=1048576
> > > > > >> > > >>> # Having non-zero Limit*s causes performance problems
> due
> > to
> > > > > >> > accounting
> > > > > >> > > >>> overhead
> > > > > >> > > >>> # in the kernel. We recommend using cgroups to do
> > > > > container-local
> > > > > >> > > >>> accounting.
> > > > > >> > > >>> LimitNPROC=infinity
> > > > > >> > > >>> LimitCORE=infinity
> > > > > >> > > >>> LimitMEMLOCK=infinity
> > > > > >> > > >>> # Uncomment TasksMax if your systemd version supports
> it.
> > > > > >> > > >>> # Only systemd 226 and above support this version.
> > > > > >> > > >>> TasksMax=infinity
> > > > > >> > > >>> TimeoutStartSec=0
> > > > > >> > > >>> # set delegate yes so that systemd does not reset the
> > > cgroups
> > > > of
> > > > > >> > docker
> > > > > >> > > >>> containers
> > > > > >> > > >>> Delegate=yes
> > > > > >> > > >>> # kill only the docker process, not all processes in the
> > > > cgroup
> > > > > >> > > >>> KillMode=process
> > > > > >> > > >>> # restart the docker process if it exits prematurely
> > > > > >> > > >>> Restart=on-failure
> > > > > >> > > >>> StartLimitBurst=3
> > > > > >> > > >>> StartLimitInterval=60s
> > > > > >> > > >>>
> > > > > >> > > >>> [Install]
> > > > > >> > > >>> WantedBy=multi-user.target
> > > > > >> > > >>>
> > > > > >> > > >>> ... the one tweak I did there is to set LimitMEMLOCK to
> > > > > >> infinity...
> > > > > >> > do
> > > > > >> > > >>> you
> > > > > >> > > >>> think the problems are file handle related? Should I set
> > > > > >> LimitNOFILE
> > > > > >> > > also
> > > > > >> > > >>> to infinity?
> > > > > >> > > >>>
> > > > > >> > > >>> Appreciate the help.
> > > > > >> > > >>>
> > > > > >> > > >>> Cheers,
> > > > > >> > > >>>
> > > > > >> > > >>> Aleks
> > > > > >> > > >>>
> > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
> > > > > >> > > >>> victor.romero@fintecheando.mx>
> > > > > >> > > >>> wrote:
> > > > > >> > > >>>
> > > > > >> > > >>> > Hi Aleks,
> > > > > >> > > >>> >
> > > > > >> > > >>> > Can you share the ulimits flags that the cassandra's
> > > > container
> > > > > >> is
> > > > > >> > > using
> > > > > >> > > >>> > while it is running?
> > > > > >> > > >>> >
> > > > > >> > > >>> > The values are being set in the composer file? Or in
> the
> > > > > >> upstart or
> > > > > >> > > >>> > systemd docker's deamon config files?
> > > > > >> > > >>> >
> > > > > >> > > >>> >
> > > > > >> > > >>> >
> > > > > >> > > >>> > Enviado desde TypeApp
> > > > > >> > > >>> >
> > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM, Aleksandar
> > > > Vidakovic <
> > > > > >> > > >>> > cheetah@monkeysintown.com> escribió:
> > > > > >> > > >>> > >... and I should have attached the Cassandra log
> > dump...
> > > so
> > > > > >> here
> > > > > >> > it
> > > > > >> > > >>> > >is...
> > > > > >> > > >>> > >
> > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar Vidakovic
> <
> > > > > >> > > >>> > >cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>> > >
> > > > > >> > > >>> > >> Hi all,
> > > > > >> > > >>> > >>
> > > > > >> > > >>> > >> ... as you might have noticed I did a couple of
> > > restarts
> > > > > >> > today...
> > > > > >> > > >>> the
> > > > > >> > > >>> > >> problem I am facing now: I can't get Cassandra to
> run
> > > in
> > > > a
> > > > > >> > stable
> > > > > >> > > >>> > >way.
> > > > > >> > > >>> > >>
> > > > > >> > > >>> > >> Things look quite OK for a while when running the
> > demo
> > > > > >> server,
> > > > > >> > but
> > > > > >> > > >>> > >then
> > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts
> > > misbehaving
> > > > > with
> > > > > >> > the
> > > > > >> > > >>> > >deposit
> > > > > >> > > >>> > >> microservice startup, sometimes with portfolio or
> > > > teller).
> > > > > I
> > > > > >> > tried
> > > > > >> > > >>> to
> > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set some
> > > Docker
> > > > > >> limits
> > > > > >> > to
> > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
> > > > > >> > > >>> > >>
> > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe someone
> > can
> > > > > help
> > > > > >> out
> > > > > >> > > >>> > >here? Is
> > > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB of
> > memory?
> > > > > >> > > >>> > >>
> > > > > >> > > >>> > >> Other than that the setup would be ready to go...
> we
> > > just
> > > > > >> need
> > > > > >> > to
> > > > > >> > > >>> get
> > > > > >> > > >>> > >the
> > > > > >> > > >>> > >> database running more reliably.
> > > > > >> > > >>> > >>
> > > > > >> > > >>> > >> Cheers,
> > > > > >> > > >>> > >>
> > > > > >> > > >>> > >> Aleks
> > > > > >> > > >>> > >>
> > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar
> Vidakovic
> > <
> > > > > >> > > >>> > >> cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>> > >>
> > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed to add
> > some
> > > > > >> > additional
> > > > > >> > > >>> > >reverse
> > > > > >> > > >>> > >>> proxy configuration for the web UI to work)... and
> > > there
> > > > > >> were
> > > > > >> > > more
> > > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
> > > > > >> > > >>> > >>>
> > > > > >> > > >>> > >>> Just FYI
> > > > > >> > > >>> > >>>
> > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
> > Vidakovic <
> > > > > >> > > >>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>> > >>>
> > > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can get
> back
> > > to
> > > > > >> testing
> > > > > >> > > >>> only
> > > > > >> > > >>> > >>>> later tonight.
> > > > > >> > > >>> > >>>>
> > > > > >> > > >>> > >>>> Cheers
> > > > > >> > > >>> > >>>>
> > > > > >> > > >>> > >>>>
> > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya
> > > > > >> > > >>> > ><ra...@gmail.com>
> > > > > >> > > >>> > >>>> wrote:
> > > > > >> > > >>> > >>>>
> > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test after 30
> min.
> > > > > >> > > >>> > >>>>>
> > > > > >> > > >>> > >>>>> Big thanks for this 🙂
> > > > > >> > > >>> > >>>>>
> > > > > >> > > >>> > >>>>>
> > > > > >> > > >>> > >>>>>
> > > > > >> > > >>> > >>>>>
> > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM Aleksandar
> > > Vidakovic <
> > > > > >> > > >>> > >>>>> cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>> > >>>>>
> > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted.
> > > > > >> > > >>> > >>>>> >
> > > > > >> > > >>> > >>>>> > ... and as I said: the services are still
> > > > starting...
> > > > > >> and
> > > > > >> > the
> > > > > >> > > >>> > >best
> > > > > >> > > >>> > >>>>> estimate
> > > > > >> > > >>> > >>>>> > I have right now (as already mentioned): 30min
> > > > > >> > > >>> > >>>>> >
> > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan Maurya <
> > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > >> > > >>> > >>>>> > wrote:
> > > > > >> > > >>> > >>>>> >
> > > > > >> > > >>> > >>>>> > > [image: image.png]
> > > > > >> > > >>> > >>>>> > >
> > > > > >> > > >>> > >>>>> > >
> > > > > >> > > >>> > >>>>> > >
> > > > > >> > > >>> > >>>>> > >
> > > > > >> > > >>> > >>>>> > >
> > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan
> Maurya <
> > > > > >> > > >>> > >>>>> rajanmaurya154@gmail.com>
> > > > > >> > > >>> > >>>>> > > wrote:
> > > > > >> > > >>> > >>>>> > >
> > > > > >> > > >>> > >>>>> > >> [image: image.png]
> > > > > >> > > >>> > >>>>> > >> I am getting this.
> > > > > >> > > >>> > >>>>> > >>
> > > > > >> > > >>> > >>>>> > >>
> > > > > >> > > >>> > >>>>> > >>
> > > > > >> > > >>> > >>>>> > >>
> > > > > >> > > >>> > >>>>> > >>
> > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM Aleksandar
> > > > > Vidakovic
> > > > > >> <
> > > > > >> > > >>> > >>>>> > >> cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>> > >>>>> > >>
> > > > > >> > > >>> > >>>>> > >>> Hi all,
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost) ready...
> > it
> > > > took
> > > > > >> me a
> > > > > >> > > >>> > >moment
> > > > > >> > > >>> > >>>>> and a
> > > > > >> > > >>> > >>>>> > >>> couple
> > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot
> > > failures...
> > > > > the
> > > > > >> > > >>> services
> > > > > >> > > >>> > >are
> > > > > >> > > >>> > >>>>> quite
> > > > > >> > > >>> > >>>>> > >>> resource hungry and default settings won't
> > do
> > > > it.
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> The services are still starting and this
> > will
> > > > > take a
> > > > > >> > > while
> > > > > >> > > >>> > >(my
> > > > > >> > > >>> > >>>>> best
> > > > > >> > > >>> > >>>>> > guess
> > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
> > > > > >> > > >>> > >http://fineract-vm.apache.org/login
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> Credentials:
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> Tenant : playground
> > > > > >> > > >>> > >>>>> > >>> Username : operator
> > > > > >> > > >>> > >>>>> > >>> Password : init1@l
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to compile
> > this
> > > > app
> > > > > >> is
> > > > > >> > > with
> > > > > >> > > >>> > >NodeJS
> > > > > >> > > >>> > >>>>> > 8.11.1
> > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't work).
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> Identity Service:
> > > > > >> > > >>> > >http://fineract-vm.apache.org:2021/identity/v1
> > > > > >> > > >>> > >>>>> > >>> Office Service:
> > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
> > > > > >> > > >>> > >>>>> > >>> Customer Service:
> > > > > >> > > >>> > >http://fineract-vm.apache.org:2024/customer/v1
> > > > > >> > > >>> > >>>>> > >>> Accounting Service:
> > > > > >> > > >>> > >>>>>
> http://fineract-vm.apache.org:2025/accounting/v1
> > > > > >> > > >>> > >>>>> > >>> Portfolio Service:
> > > > > >> > > >>> > >>>>> http://fineract-vm.apache.org:2026/portfolio/v1
> > > > > >> > > >>> > >>>>> > >>> Deposit Service:
> > > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1
> > > > > >> > > >>> > >>>>> > >>> Teller Service:
> > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
> > > > > >> > > >>> > >>>>> > >>> Reporting Service:
> > > > > >> > > >>> > >>>>> http://fineract-vm.apache.org:2029/reporting/v1
> > > > > >> > > >>> > >>>>> > >>> Cheque Service:
> > > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1
> > > > > >> > > >>> > >>>>> > >>> Payroll Service:
> > > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes quite
> a
> > > > > >> while... if
> > > > > >> > > you
> > > > > >> > > >>> > >>>>> encounter
> > > > > >> > > >>> > >>>>> > >>> connection problems retry a couple of
> > minutes
> > > > > later
> > > > > >> to
> > > > > >> > > >>> > >ensure I
> > > > > >> > > >>> > >>>>> am not
> > > > > >> > > >>> > >>>>> > >>> currently working on something.
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the backend
> > > services
> > > > > >> once
> > > > > >> > per
> > > > > >> > > >>> > >day or
> > > > > >> > > >>> > >>>>> so to
> > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to figure
> > out
> > > > how
> > > > > we
> > > > > >> > want
> > > > > >> > > >>> to
> > > > > >> > > >>> > >>>>> handle
> > > > > >> > > >>> > >>>>> > this
> > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where to put
> > this
> > > > demo
> > > > > >> > server
> > > > > >> > > >>> > >>>>> configuration
> > > > > >> > > >>> > >>>>> > >>> then let me know here.
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any problems
> (I
> > > > have
> > > > > >> not
> > > > > >> > > >>> > >extensively
> > > > > >> > > >>> > >>>>> > tested
> > > > > >> > > >>> > >>>>> > >>> it
> > > > > >> > > >>> > >>>>> > >>> yet).
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> Cheers,
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> Aleks
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM
> Aleksandar
> > > > > >> Vidakovic <
> > > > > >> > > >>> > >>>>> > >>> cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>> > >>>>> > >>>
> > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem to be
> > > starting
> > > > > fine
> > > > > >> > > now...
> > > > > >> > > >>> > >just
> > > > > >> > > >>> > >>>>> have
> > > > > >> > > >>> > >>>>> > to
> > > > > >> > > >>> > >>>>> > >>> > work on a couple of improvements on the
> > > config
> > > > > >> files.
> > > > > >> > > >>> > >>>>> > >>> >
> > > > > >> > > >>> > >>>>> > >>> > Currently having another stab to get the
> > > > webapp
> > > > > >> > > running.
> > > > > >> > > >>> > >>>>> > >>> >
> > > > > >> > > >>> > >>>>> > >>> > BTW: it should be fairly easy to adapt
> the
> > > > > >> simplistic
> > > > > >> > > >>> > >Docker
> > > > > >> > > >>> > >>>>> Compose
> > > > > >> > > >>> > >>>>> > >>> file
> > > > > >> > > >>> > >>>>> > >>> > for a CI pipeline.
> > > > > >> > > >>> > >>>>> > >>> >
> > > > > >> > > >>> > >>>>> > >>> > Will keep you posted.
> > > > > >> > > >>> > >>>>> > >>> >
> > > > > >> > > >>> > >>>>> > >>> > Cheers,
> > > > > >> > > >>> > >>>>> > >>> >
> > > > > >> > > >>> > >>>>> > >>> > Aleks
> > > > > >> > > >>> > >>>>> > >>> >
> > > > > >> > > >>> > >>>>> > >>> > On Mon, Jun 11, 2018 at 9:41 AM
> Aleksandar
> > > > > >> Vidakovic
> > > > > >> > <
> > > > > >> > > >>> > >>>>> > >>> > cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>> > >>>>> > >>> >
> > > > > >> > > >>> > >>>>> > >>> >> I am still working on it... Will ping
> you
> > > > here
> > > > > >> when
> > > > > >> > > it's
> > > > > >> > > >>> > >ready
> > > > > >> > > >>> > >>>>> (hope
> > > > > >> > > >>> > >>>>> > >>> >> soon).
> > > > > >> > > >>> > >>>>> > >>> >>
> > > > > >> > > >>> > >>>>> > >>> >> On Mon, Jun 11, 2018, 9:40 AM Rajan
> > Maurya
> > > <
> > > > > >> > > >>> > >>>>> > rajanmaurya154@gmail.com>
> > > > > >> > > >>> > >>>>> > >>> >> wrote:
> > > > > >> > > >>> > >>>>> > >>> >>
> > > > > >> > > >>> > >>>>> > >>> >>> Hi Alex,
> > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > >> > > >>> > >>>>> > >>> >>> Is the demo server up, If yes then
> > what's
> > > > the
> > > > > >> base
> > > > > >> > > url
> > > > > >> > > >>> > >so I
> > > > > >> > > >>> > >>>>> can
> > > > > >> > > >>> > >>>>> > >>> access.
> > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > >> > > >>> > >>>>> > >>> >>> On Fri, Jun 8, 2018 at 2:03 PM Rajan
> > > Maurya
> > > > <
> > > > > >> > > >>> > >>>>> > >>> rajanmaurya154@gmail.com>
> > > > > >> > > >>> > >>>>> > >>> >>> wrote:
> > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > >> > > >>> > >>>>> > >>> >>> > Hi Alex,
> > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > >> > > >>> > >>>>> > >>> >>> > This is really great news, I will
> wait
> > > > until
> > > > > >> > then.
> > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > >> > > >>> > >>>>> > >>> >>> > Thanks
> > > > > >> > > >>> > >>>>> > >>> >>> > Rajan Maurya
> > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > >> > > >>> > >>>>> > >>> >>> > On Fri, Jun 8, 2018 at 11:36 AM
> > > Aleksandar
> > > > > >> > > Vidakovic
> > > > > >> > > >>> <
> > > > > >> > > >>> > >>>>> > >>> >>> > cheetah@monkeysintown.com> wrote:
> > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > >> > > >>> > >>>>> > >>> >>> >> Hi Rajan,
> > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > >> > > >>> > >>>>> > >>> >>> >> My plan is to get this up and
> running
> > > > until
> > > > > >> > > Monday.
> > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > >> > > >>> > >>>>> > >>> >>> >> Cheers,
> > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > >> > > >>> > >>>>> > >>> >>> >> Aleks
> > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > >> > > >>> > >>>>> > >>> >>> >> On Fri, Jun 8, 2018, 7:23 AM Rajan
> > > > Maurya <
> > > > > >> > > >>> > >>>>> > >>> rajanmaurya154@gmail.com>
> > > > > >> > > >>> > >>>>> > >>> >>> >> wrote:
> > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > >> > > >>> > >>>>> > >>> >>> >> > Hi Markus, Mark, Myrle, Alex, Ed
> > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > I am mentoring fineract-cn-mobile
> > > GSoC
> > > > > >> > project.
> > > > > >> > > I
> > > > > >> > > >>> > >am
> > > > > >> > > >>> > >>>>> waiting
> > > > > >> > > >>> > >>>>> > for
> > > > > >> > > >>> > >>>>> > >>> >>> this
> > > > > >> > > >>> > >>>>> > >>> >>> >> demo
> > > > > >> > > >>> > >>>>> > >>> >>> >> > server for a long time to see
> > what's
> > > > most
> > > > > >> > > >>> important
> > > > > >> > > >>> > >to
> > > > > >> > > >>> > >>>>> > implement
> > > > > >> > > >>> > >>>>> > >>> >>> in-app
> > > > > >> > > >>> > >>>>> > >>> >>> >> > first.
> > > > > >> > > >>> > >>>>> > >>> >>> >> > My GSoC students almost finished
> > the
> > > > task
> > > > > >> that
> > > > > >> > > was
> > > > > >> > > >>> > >>>>> necessary.
> > > > > >> > > >>> > >>>>> > We
> > > > > >> > > >>> > >>>>> > >>> >>> have
> > > > > >> > > >>> > >>>>> > >>> >>> >> kinda
> > > > > >> > > >>> > >>>>> > >>> >>> >> > of stuck and We really need this
> > demo
> > > > > >> server
> > > > > >> > and
> > > > > >> > > >>> > >web app
> > > > > >> > > >>> > >>>>> to
> > > > > >> > > >>> > >>>>> > >>> assign
> > > > > >> > > >>> > >>>>> > >>> >>> them
> > > > > >> > > >>> > >>>>> > >>> >>> >> > tasks.
> > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > It would be nice we can get the
> > demo
> > > > > >> server up
> > > > > >> > > >>> > >ASAP.
> > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > Thanks
> > > > > >> > > >>> > >>>>> > >>> >>> >> > Rajan Maurya
> > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > On Fri, Jun 1, 2018 at 6:41 PM
> Mark
> > > van
> > > > > >> Veen <
> > > > > >> > > >>> > >>>>> > >>> >>> mark.vanveen@kuelap.io>
> > > > > >> > > >>> > >>>>> > >>> >>> >> > wrote:
> > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > Hi Aleks,
> > > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > Re your second issue:
> > > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > It looks like that node-sass
> has
> > > not
> > > > > been
> > > > > >> > > >>> > >installed
> > > > > >> > > >>> > >>>>> > correctly.
> > > > > >> > > >>> > >>>>> > >>> >>> Can you
> > > > > >> > > >>> > >>>>> > >>> >>> >> > try
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > to remove the node_modules
> folder
> > > and
> > > > > run
> > > > > >> > npm
> > > > > >> > > >>> > >install
> > > > > >> > > >>> > >>>>> again?
> > > > > >> > > >>> > >>>>> > >>> This
> > > > > >> > > >>> > >>>>> > >>> >>> >> should
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > run without errors.
> > > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > Mark
> > > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > On Thu, May 31, 2018 at 9:05 PM
> > > > Courage
> > > > > >> > Angeh
> > > > > >> > > <
> > > > > >> > > >>> > >>>>> > >>> >>> courageangeh@gmail.com
> > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > wrote:
> > > > > >> > > >>> > >>>>> > >>> >>> >> > >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > > Well, i can it with
> > containerized
> > > > > >> MariaDB
> > > > > >> > > and
> > > > > >> > > >>> > >it
> > > > > >> > > >>> > >>>>> work till
> > > > > >> > > >>> > >>>>> > >>> my
> > > > > >> > > >>> > >>>>> > >>> >>> >> computer
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > ran
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > > out of space (in the middle
> of
> > > > > >> > provisioning)
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > > On Thu, May 31, 2018, 6:29 PM
> > > > Victor
> > > > > >> > Romero
> > > > > >> > > <
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > victor.romero@fintecheando.mx
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > > >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > > wrote:
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > >
> > > > > >> > > >>> > >>>>> > >>> >>> >> > > > > Hi Aleks,
>