You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@fineract.apache.org by Udayan Tandon <ud...@threewheelsunited.com> on 2019/07/25 12:40:15 UTC

MariaDB connection getting reset in microservices

Hi guys,

I have been trying to setup Fineract CN on top of docker swarm. I have largely been successful in doing so and the microservices are up and running with a tenant and a bunch of users created. A constant problem that I have been facing is that after a while a microservice has been up say such as the office microservice it gives an error when it tries to fetch data from MariaDB instance running in another container. This is the error log that it gives:

applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:55:51.125 [main] INFO  o.a.f.cn.office.OfficeApplication - Started OfficeApplication in 14.768 seconds (JVM running for 15.77)
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC username not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC password not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:00:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:05:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:10:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:15:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.795 [qtp1962266146-17] INFO  o.a.f.c.l.c.ServiceExceptionFilter - Unexpected exception caught org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.798 [qtp1962266146-17] WARN  o.e.jetty.servlet.ServletHandler -
applications_office-ms.1.o15rgpqagwmv@Fineract    | org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:225)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.ServiceExceptionFilter.doFilterInternal(ServiceExceptionFilter.java:44)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.anubis.filter.UserContextFilter.doFilterInternal(UserContextFilter.java:51)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:121)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.TenantHeaderFilter.doFilterInternal(TenantHeaderFilter.java:53)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.Server.handle(Server.java:524)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.Thread.run(Thread.java:748)
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.internal.service.OfficeService$$EnhancerBySpringCGLIB$$d825377c.fetchOffices(<generated>)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.rest.controller.OfficeRestController.fetchOffices(OfficeRestController.java:132)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.reflect.Method.invoke(Method.java:498)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 87 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 108 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 110 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 119 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.net.SocketException: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:210)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:166)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 121 common frames omitted

On the MariaDB side of the things it just says "Got an error reading communication packets" and closes a bunch of connections. I have tried changing the "wait_timeout" as mentioned in this issue: https://github.com/docker-library/mariadb/issues/113 and have also increased the max_communication_packet size to no avail. My BoneCP pool settings are the default as follows:

bonecp:
  idleMaxAgeInMinutes: 240
  idleConnectionTestPeriodInMinutes: 60
  maxConnectionsPerPartition: 10
  minConnectionsPerPartition: 1
  partitionCount: 2
  acquireIncrement: 5
  statementsCacheSize: 100

If anyone has any insight on the matter or has experienced the same set of problems before any help would be much appreciated.

Thanks and Regards
Udayan Tandon
https://github.com/udayan12167
https://www.threewheelsunited.com/

RE: RE: MariaDB connection getting reset in microservices

Posted by Udayan Tandon <ud...@threewheelsunited.com>.

Hi,

For anyone facing the same problem as me I managed to fix it by reducing the BoneCP idleMaxAgeInMinutes to 30 mins and idleConnectionTestPeriod to 1 Min. Now these are the first settings I tried and everything seems to be working. I’ll be digging deeper into the limits of these settings later.

Thanks and Regards
Udayan Tandon

________________________________
From: Udayan Tandon <ud...@threewheelsunited.com>
Sent: Wednesday, July 31, 2019 11:35:02 PM
To: dev@fineract.apache.org <de...@fineract.apache.org>
Subject: RE: RE: MariaDB connection getting reset in microservices

Hi Victor,

Adding my Mariadb config file as attachment to this mail for reference. Let me know if you see anything particularly wrong with it.

+ Adding some new details about the error.
So the DB connection errors out after the microservice has been up for a while. Say roughly 30 mins. Then all subsequent API requests that need from MariaDB for that microservice error out with the following exception:

Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Connection reset
applications_office-ms.1.l0j2g0g89bc2@Fineract    | Query is : set autocommit=0
applications_office-ms.1.l0j2g0g89bc2@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
applications_office-ms.1.l0j2g0g89bc2@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
applications_office-ms.1.l0j2g0g89bc2@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
applications_office-ms.1.l0j2g0g89bc2@Fineract    |     ... 118 common frames omitted
applications_office-ms.1.l0j2g0g89bc2@Fineract    | Caused by: java.net.SocketException: Connection reset

If I keep trying API requests to the same microservice multiple times. Then eventually the database connection restores itself and it starts working again to the point when it’s left idle again for a while. Then it dies again.

Thanks and Regards
Udayan Tandon

________________________________
From: VIctor Romero <vi...@fintecheando.mx>
Sent: Tuesday, July 30, 2019 8:13:04 PM
To: dev@fineract.apache.org <de...@fineract.apache.org>
Subject: RE: MariaDB connection getting reset in microservices

I think it is more related to the MariaDb settings or TCP settings at OS level.

Could you please share your maria db configuration file?

Regards

Obtener BlueMail para Android<http://www.bluemail.me/r?b=15431>
En 29 jul 2019, en 2:05 p. m., Udayan Tandon <ud...@threewheelsunited.com>> escribió:


Hi Victor,



  1.  I’m using Java version 1.8. To build every docker image. Basically using the Dockerfiles provided in each repo of each Fineract CN microservice.
  2.  MariaDB 10.4
  3.  MariaDB is running in a container inside a docker swarm on the same host as the microservices which are running in their own separate containers.
  4.  I created a docker network and all the containers run on them and thus MariaDB is connected to other services through this network. The DB connection does happen initially and as I provision services for the tenant it even runs Flyway migrations perfectly. But then starts erroring out and refuses to reconnect unless I restart and re-provision the service.
  5.  Yes it is docker CE v18.09.7 running on an Azure Ubuntu VM. The MariaDB side shows the following logs:
Aborted connection #SNo. to db: 'playground' user: 'root' host: '172.16.238.69' (Got an error reading communication packets)
Where the host IP address is actually the IP of the internal docker network. As I mentioned before I have tried to tinker and corrected all obvious properties on the MariaDB side that can be the cause of that error.


Let me know if you have more questions.


Thanks and Regards
Udayan Tandon



________________________________
From: Victor Manuel Romero Rodriguez <vi...@hotmail.com>
Sent: Monday, July 29, 2019 10:56:10 PM
To: dev@fineract.apache.org <de...@fineract.apache.org>
Subject: Re: MariaDB connection getting reset in microservices

Hi Udayan,

I have questions to try to help further:

1. Which Java version are you using to run the services?
2. Which is the MariaDB version used?
3. MariaDB is running in the same host or in another? in VM, Container or in a bare bone server?
4. Which kind of networking have you set in Docker?
5. It is Docker CE? any event or log is shown in the MariaDB side?

Please let me know.

About our experience have been growing in Fineract-CN and let me explain the way we did it

1st. We have improved the DB connection replacing BoneCP with HikariCP.

2nd. We added a HA Proxy in Front of a Galera Cluster for MariaDB. (We have timeouts but it is expected because the pooling and the timeout set in the HA Proxy, but connection is recovered and it is only a warning, by the way we use the least connection algorithm for balancing).

3rd. Some customers has requested to use the MariaDB services in the cloud (AWS, GCP and Azure).

4th. For high volume/rate of transactions we had had to create connectors and some other stuff in the microservices to make it work with Oracle and TiDB. Which is more complex but they are different use cases.

Regards

Victor Romero


El 25/07/19 a las 7:40, Udayan Tandon escribió:
Hi guys,


I have been trying to setup Fineract CN on top of docker swarm. I have largely been successful in doing so and the microservices are up and running with a tenant and a bunch of users created. A constant problem that I have been facing is that after a while a microservice has been up say such as the office microservice it gives an error when it tries to fetch data from MariaDB instance running in another container. This is the error log that it gives:


applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:55:51.125 [main] INFO  o.a.f.cn.office.OfficeApplication - Started OfficeApplication in 14.768 seconds (JVM running for 15.77)
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC username not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC password not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:00:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:05:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:10:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:15:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.795 [qtp1962266146-17] INFO  o.a.f.c.l.c.ServiceExceptionFilter - Unexpected exception caught org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.798 [qtp1962266146-17] WARN  o.e.jetty.servlet.ServletHandler -
applications_office-ms.1.o15rgpqagwmv@Fineract    | org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:225)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.ServiceExceptionFilter.doFilterInternal(ServiceExceptionFilter.java:44)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.anubis.filter.UserContextFilter.doFilterInternal(UserContextFilter.java:51)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:121)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.TenantHeaderFilter.doFilterInternal(TenantHeaderFilter.java:53)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.Server.handle(Server.java:524)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.Thread.run(Thread.java:748)
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.internal.service.OfficeService$$EnhancerBySpringCGLIB$$d825377c.fetchOffices(<generated>)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.rest.controller.OfficeRestController.fetchOffices(OfficeRestController.java:132)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.reflect.Method.invoke(Method.java:498)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 87 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 108 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 110 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 119 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.net.SocketException: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:210)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:166)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 121 common frames omitted


On the MariaDB side of the things it just says “Got an error reading communication packets” and closes a bunch of connections. I have tried changing the “wait_timeout” as mentioned in this issue: https://github.com/docker-library/mariadb/issues/113 and have also increased the max_communication_packet size to no avail. My BoneCP pool settings are the default as follows:


bonecp:
  idleMaxAgeInMinutes: 240
  idleConnectionTestPeriodInMinutes: 60
  maxConnectionsPerPartition: 10
  minConnectionsPerPartition: 1
  partitionCount: 2
  acquireIncrement: 5
  statementsCacheSize: 100


If anyone has any insight on the matter or has experienced the same set of problems before any help would be much appreciated.


Thanks and Regards
Udayan Tandon
https://github.com/udayan12167
https://www.threewheelsunited.com/

RE: RE: MariaDB connection getting reset in microservices

Posted by Udayan Tandon <ud...@threewheelsunited.com>.

Hi Victor,

Adding my Mariadb config file as attachment to this mail for reference. Let me know if you see anything particularly wrong with it.

+ Adding some new details about the error.
So the DB connection errors out after the microservice has been up for a while. Say roughly 30 mins. Then all subsequent API requests that need from MariaDB for that microservice error out with the following exception:

Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Connection reset
applications_office-ms.1.l0j2g0g89bc2@Fineract    | Query is : set autocommit=0
applications_office-ms.1.l0j2g0g89bc2@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
applications_office-ms.1.l0j2g0g89bc2@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
applications_office-ms.1.l0j2g0g89bc2@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
applications_office-ms.1.l0j2g0g89bc2@Fineract    |     ... 118 common frames omitted
applications_office-ms.1.l0j2g0g89bc2@Fineract    | Caused by: java.net.SocketException: Connection reset

If I keep trying API requests to the same microservice multiple times. Then eventually the database connection restores itself and it starts working again to the point when it’s left idle again for a while. Then it dies again.

Thanks and Regards
Udayan Tandon

________________________________
From: VIctor Romero <vi...@fintecheando.mx>
Sent: Tuesday, July 30, 2019 8:13:04 PM
To: dev@fineract.apache.org <de...@fineract.apache.org>
Subject: RE: MariaDB connection getting reset in microservices

I think it is more related to the MariaDb settings or TCP settings at OS level.

Could you please share your maria db configuration file?

Regards

Obtener BlueMail para Android<http://www.bluemail.me/r?b=15431>
En 29 jul 2019, en 2:05 p. m., Udayan Tandon <ud...@threewheelsunited.com>> escribió:


Hi Victor,



  1.  I’m using Java version 1.8. To build every docker image. Basically using the Dockerfiles provided in each repo of each Fineract CN microservice.
  2.  MariaDB 10.4
  3.  MariaDB is running in a container inside a docker swarm on the same host as the microservices which are running in their own separate containers.
  4.  I created a docker network and all the containers run on them and thus MariaDB is connected to other services through this network. The DB connection does happen initially and as I provision services for the tenant it even runs Flyway migrations perfectly. But then starts erroring out and refuses to reconnect unless I restart and re-provision the service.
  5.  Yes it is docker CE v18.09.7 running on an Azure Ubuntu VM. The MariaDB side shows the following logs:
Aborted connection #SNo. to db: 'playground' user: 'root' host: '172.16.238.69' (Got an error reading communication packets)
Where the host IP address is actually the IP of the internal docker network. As I mentioned before I have tried to tinker and corrected all obvious properties on the MariaDB side that can be the cause of that error.


Let me know if you have more questions.


Thanks and Regards
Udayan Tandon



________________________________
From: Victor Manuel Romero Rodriguez <vi...@hotmail.com>
Sent: Monday, July 29, 2019 10:56:10 PM
To: dev@fineract.apache.org <de...@fineract.apache.org>
Subject: Re: MariaDB connection getting reset in microservices

Hi Udayan,

I have questions to try to help further:

1. Which Java version are you using to run the services?
2. Which is the MariaDB version used?
3. MariaDB is running in the same host or in another? in VM, Container or in a bare bone server?
4. Which kind of networking have you set in Docker?
5. It is Docker CE? any event or log is shown in the MariaDB side?

Please let me know.

About our experience have been growing in Fineract-CN and let me explain the way we did it

1st. We have improved the DB connection replacing BoneCP with HikariCP.

2nd. We added a HA Proxy in Front of a Galera Cluster for MariaDB. (We have timeouts but it is expected because the pooling and the timeout set in the HA Proxy, but connection is recovered and it is only a warning, by the way we use the least connection algorithm for balancing).

3rd. Some customers has requested to use the MariaDB services in the cloud (AWS, GCP and Azure).

4th. For high volume/rate of transactions we had had to create connectors and some other stuff in the microservices to make it work with Oracle and TiDB. Which is more complex but they are different use cases.

Regards

Victor Romero


El 25/07/19 a las 7:40, Udayan Tandon escribió:
Hi guys,


I have been trying to setup Fineract CN on top of docker swarm. I have largely been successful in doing so and the microservices are up and running with a tenant and a bunch of users created. A constant problem that I have been facing is that after a while a microservice has been up say such as the office microservice it gives an error when it tries to fetch data from MariaDB instance running in another container. This is the error log that it gives:


applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:55:51.125 [main] INFO  o.a.f.cn.office.OfficeApplication - Started OfficeApplication in 14.768 seconds (JVM running for 15.77)
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC username not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC password not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:00:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:05:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:10:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:15:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.795 [qtp1962266146-17] INFO  o.a.f.c.l.c.ServiceExceptionFilter - Unexpected exception caught org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.798 [qtp1962266146-17] WARN  o.e.jetty.servlet.ServletHandler -
applications_office-ms.1.o15rgpqagwmv@Fineract    | org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:225)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.ServiceExceptionFilter.doFilterInternal(ServiceExceptionFilter.java:44)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.anubis.filter.UserContextFilter.doFilterInternal(UserContextFilter.java:51)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:121)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.TenantHeaderFilter.doFilterInternal(TenantHeaderFilter.java:53)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.Server.handle(Server.java:524)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.Thread.run(Thread.java:748)
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.internal.service.OfficeService$$EnhancerBySpringCGLIB$$d825377c.fetchOffices(<generated>)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.rest.controller.OfficeRestController.fetchOffices(OfficeRestController.java:132)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.reflect.Method.invoke(Method.java:498)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 87 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 108 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 110 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 119 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.net.SocketException: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:210)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:166)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 121 common frames omitted


On the MariaDB side of the things it just says “Got an error reading communication packets” and closes a bunch of connections. I have tried changing the “wait_timeout” as mentioned in this issue: https://github.com/docker-library/mariadb/issues/113 and have also increased the max_communication_packet size to no avail. My BoneCP pool settings are the default as follows:


bonecp:
  idleMaxAgeInMinutes: 240
  idleConnectionTestPeriodInMinutes: 60
  maxConnectionsPerPartition: 10
  minConnectionsPerPartition: 1
  partitionCount: 2
  acquireIncrement: 5
  statementsCacheSize: 100


If anyone has any insight on the matter or has experienced the same set of problems before any help would be much appreciated.


Thanks and Regards
Udayan Tandon
https://github.com/udayan12167
https://www.threewheelsunited.com/

RE: MariaDB connection getting reset in microservices

Posted by VIctor Romero <vi...@fintecheando.mx>.

I think it is more related to the MariaDb settings or TCP settings at OS level. 

Could you please share your maria db configuration file? 

Regards

⁣Obtener BlueMail para Android 

En 29 jul 2019 2:05 p. m., en 2:05 p. m., Udayan Tandon <ud...@threewheelsunited.com> escribió:
>
>Hi Victor,
>
>
>1.  I’m using Java version 1.8. To build every docker image. Basically
>using the Dockerfiles provided in each repo of each Fineract CN
>microservice.
>  2.  MariaDB 10.4
>3.  MariaDB is running in a container inside a docker swarm on the same
>host as the microservices which are running in their own separate
>containers.
>4.  I created a docker network and all the containers run on them and
>thus MariaDB is connected to other services through this network. The
>DB connection does happen initially and as I provision services for the
>tenant it even runs Flyway migrations perfectly. But then starts
>erroring out and refuses to reconnect unless I restart and re-provision
>the service.
>5.  Yes it is docker CE v18.09.7 running on an Azure Ubuntu VM. The
>MariaDB side shows the following logs:
>Aborted connection #SNo. to db: 'playground' user: 'root' host:
>'172.16.238.69' (Got an error reading communication packets)
>Where the host IP address is actually the IP of the internal docker
>network. As I mentioned before I have tried to tinker and corrected all
>obvious properties on the MariaDB side that can be the cause of that
>error.
>
>Let me know if you have more questions.
>
>Thanks and Regards
>Udayan Tandon
>
>________________________________
>From: Victor Manuel Romero Rodriguez <vi...@hotmail.com>
>Sent: Monday, July 29, 2019 10:56:10 PM
>To: dev@fineract.apache.org <de...@fineract.apache.org>
>Subject: Re: MariaDB connection getting reset in microservices
>
>Hi Udayan,
>
>I have questions to try to help further:
>
>1. Which Java version are you using to run the services?
>2. Which is the MariaDB version used?
>3. MariaDB is running in the same host or in another? in VM, Container
>or in a bare bone server?
>4. Which kind of networking have you set in Docker?
>5. It is Docker CE? any event or log is shown in the MariaDB side?
>
>Please let me know.
>
>About our experience have been growing in Fineract-CN and let me
>explain the way we did it
>
>1st. We have improved the DB connection replacing BoneCP with HikariCP.
>
>2nd. We added a HA Proxy in Front of a Galera Cluster for MariaDB. (We
>have timeouts but it is expected because the pooling and the timeout
>set in the HA Proxy, but connection is recovered and it is only a
>warning, by the way we use the least connection algorithm for
>balancing).
>
>3rd. Some customers has requested to use the MariaDB services in the
>cloud (AWS, GCP and Azure).
>
>4th. For high volume/rate of transactions we had had to create
>connectors and some other stuff in the microservices to make it work
>with Oracle and TiDB. Which is more complex but they are different use
>cases.
>
>Regards
>
>Victor Romero
>
>
>El 25/07/19 a las 7:40, Udayan Tandon escribió:
>Hi guys,
>
>I have been trying to setup Fineract CN on top of docker swarm. I have
>largely been successful in doing so and the microservices are up and
>running with a tenant and a bunch of users created. A constant problem
>that I have been facing is that after a while a microservice has been
>up say such as the office microservice it gives an error when it tries
>to fetch data from MariaDB instance running in another container. This
>is the error log that it gives:
>
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:55:51.125 [main]
>INFO  o.a.f.cn.office.OfficeApplication - Started OfficeApplication in
>14.768 seconds (JVM running for 15.77)
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411
>[qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC username
>not set in driver properties, copying it from pool config
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411
>[qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC password
>not set in driver properties, copying it from pool config
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:00:50.711
>[AsyncResolver-bootstrap-executor-0] INFO 
>c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
>configuration
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:05:50.711
>[AsyncResolver-bootstrap-executor-0] INFO 
>c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
>configuration
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:10:50.712
>[AsyncResolver-bootstrap-executor-0] INFO 
>c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
>configuration
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:15:50.712
>[AsyncResolver-bootstrap-executor-0] INFO 
>c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
>configuration
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.795
>[qtp1962266146-17] INFO  o.a.f.c.l.c.ServiceExceptionFilter -
>Unexpected exception caught
>org.springframework.web.util.NestedServletException: Request processing
>failed; nested exception is
>org.springframework.transaction.CannotCreateTransactionException: Could
>not open JPA EntityManager for transaction; nested exception is
>javax.persistence.PersistenceException:
>java.sql.SQLNonTransientConnectionException: Could not read resultset:
>Connection reset
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set
>autocommit=0
>applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.798
>[qtp1962266146-17] WARN  o.e.jetty.servlet.ServletHandler -
>applications_office-ms.1.o15rgpqagwmv@Fineract    |
>org.springframework.web.util.NestedServletException: Request processing
>failed; nested exception is
>org.springframework.transaction.CannotCreateTransactionException: Could
>not open JPA EntityManager for transaction; nested exception is
>javax.persistence.PersistenceException:
>java.sql.SQLNonTransientConnectionException: Could not read resultset:
>Connection reset
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set
>autocommit=0
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:225)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.apache.fineract.cn.lang.config.ServiceExceptionFilter.doFilterInternal(ServiceExceptionFilter.java:44)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.apache.fineract.cn.anubis.filter.UserContextFilter.doFilterInternal(UserContextFilter.java:51)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:121)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.apache.fineract.cn.lang.config.TenantHeaderFilter.doFilterInternal(TenantHeaderFilter.java:53)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.Server.handle(Server.java:524)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>java.lang.Thread.run(Thread.java:748)
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by:
>org.springframework.transaction.CannotCreateTransactionException: Could
>not open JPA EntityManager for transaction; nested exception is
>javax.persistence.PersistenceException:
>java.sql.SQLNonTransientConnectionException: Could not read resultset:
>Connection reset
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set
>autocommit=0
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.apache.fineract.cn.office.internal.service.OfficeService$$EnhancerBySpringCGLIB$$d825377c.fetchOffices(<generated>)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.apache.fineract.cn.office.rest.controller.OfficeRestController.fetchOffices(OfficeRestController.java:132)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>java.lang.reflect.Method.invoke(Method.java:498)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 87 common
>frames omitted
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by:
>javax.persistence.PersistenceException:
>java.sql.SQLNonTransientConnectionException: Could not read resultset:
>Connection reset
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 108 common
>frames omitted
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by:
>java.sql.SQLNonTransientConnectionException: Could not read resultset:
>Connection reset
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set
>autocommit=0
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 110 common
>frames omitted
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by:
>org.mariadb.jdbc.internal.util.dao.QueryException: Could not read
>resultset: Connection reset
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set
>autocommit=0
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 119 common
>frames omitted
>applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by:
>java.net.SocketException: Connection reset
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>java.net.SocketInputStream.read(SocketInputStream.java:210)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>java.net.SocketInputStream.read(SocketInputStream.java:141)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:166)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     at
>org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
>applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 121 common
>frames omitted
>
>On the MariaDB side of the things it just says “Got an error reading
>communication packets” and closes a bunch of connections. I have tried
>changing the “wait_timeout” as mentioned in this issue:
>https://github.com/docker-library/mariadb/issues/113 and have also
>increased the max_communication_packet size to no avail. My BoneCP pool
>settings are the default as follows:
>
>bonecp:
>  idleMaxAgeInMinutes: 240
>  idleConnectionTestPeriodInMinutes: 60
>  maxConnectionsPerPartition: 10
>  minConnectionsPerPartition: 1
>  partitionCount: 2
>  acquireIncrement: 5
>  statementsCacheSize: 100
>
>If anyone has any insight on the matter or has experienced the same set
>of problems before any help would be much appreciated.
>
>Thanks and Regards
>Udayan Tandon
>https://github.com/udayan12167
>https://www.threewheelsunited.com/

Re: MariaDB connection getting reset in microservices

Posted by Victor Romero <vi...@fintecheando.mx>.

Hi Udayan,


Please check if you Maria DB environment is not having this issue:


https://github.com/docker-library/mariadb/issues/113


If so, please change the wait_time as explained there.

Regards

Victor

El 29/07/19 a las 14:05, Udayan Tandon escribió:
>
> Hi Victor,
>
>  1. I’m using Java version 1.8. To build every docker image. Basically
>     using the Dockerfiles provided in each repo of each Fineract CN
>     microservice.
>  2. MariaDB 10.4
>  3. MariaDB is running in a container inside a docker swarm on the
>     same host as the microservices which are running in their own
>     separate containers.
>  4. I created a docker network and all the containers run on them and
>     thus MariaDB is connected to other services through this network.
>     The DB connection does happen initially and as I provision
>     services for the tenant it even runs Flyway migrations perfectly.
>     But then starts erroring out and refuses to reconnect unless I
>     restart and re-provision the service.
>  5. Yes it is docker CE v18.09.7 running on an Azure Ubuntu VM. The
>     MariaDB side shows the following logs:
>     Aborted connection #SNo. to db: 'playground' user: 'root' host:
>     '172.16.238.69' (Got an error reading communication packets)
>     Where the host IP address is actually the IP of the internal
>     docker network. As I mentioned before I have tried to tinker and
>     corrected all obvious properties on the MariaDB side that can be
>     the cause of that error.
>
> Let me know if you have more questions.
>
> Thanks and Regards
>
> Udayan Tandon
>
> ------------------------------------------------------------------------
> *From:* Victor Manuel Romero Rodriguez <vi...@hotmail.com>
> *Sent:* Monday, July 29, 2019 10:56:10 PM
> *To:* dev@fineract.apache.org <de...@fineract.apache.org>
> *Subject:* Re: MariaDB connection getting reset in microservices
> Hi Udayan,
>
> I have questions to try to help further:
>
> 1. Which Java version are you using to run the services?
> 2. Which is the MariaDB version used?
> 3. MariaDB is running in the same host or in another? in VM, Container 
> or in a bare bone server?
> 4. Which kind of networking have you set in Docker?
> 5. It is Docker CE? any event or log is shown in the MariaDB side?
>
> Please let me know.
>
> About our experience have been growing in Fineract-CN and let me 
> explain the way we did it
>
> 1st. We have improved the DB connection replacing BoneCP with HikariCP.
>
> 2nd. We added a HA Proxy in Front of a Galera Cluster for MariaDB. (We 
> have timeouts but it is expected because the pooling and the timeout 
> set in the HA Proxy, but connection is recovered and it is only a 
> warning, by the way we use the least connection algorithm for balancing).
>
> 3rd. Some customers has requested to use the MariaDB services in the 
> cloud (AWS, GCP and Azure).
>
> 4th. For high volume/rate of transactions we had had to create 
> connectors and some other stuff in the microservices to make it work 
> with Oracle and TiDB. Which is more complex but they are different use 
> cases.
>
> Regards
>
> Victor Romero
>
>
> El 25/07/19 a las 7:40, Udayan Tandon escribió:
>>
>> Hi guys,
>>
>> I have been trying to setup Fineract CN on top of docker swarm. I 
>> have largely been successful in doing so and the microservices are up 
>> and running with a tenant and a bunch of users created. A constant 
>> problem that I have been facing is that after a while a microservice 
>> has been up say such as the office microservice it gives an error 
>> when it tries to fetch data from MariaDB instance running in another 
>> container. This is the error log that it gives:
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 08:55:51.125[main] 
>> INFOo.a.f.cn.office.OfficeApplication- Started OfficeApplication in 
>> 14.768seconds (JVM running for 15.77)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> 08:56:03.411[qtp1962266146-20] WARNcom.jolbox.bonecp.BoneCPConfig- 
>> JDBC username not set in driver properties, copying it from pool config
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> 08:56:03.411[qtp1962266146-20] WARNcom.jolbox.bonecp.BoneCPConfig- 
>> JDBC password not set in driver properties, copying it from pool config
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> 09:00:50.711[AsyncResolver-bootstrap-executor-0] 
>> INFOc.n.d.s.r.aws.ConfigClusterResolver- Resolving eureka endpoints 
>> via configuration
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> 09:05:50.711[AsyncResolver-bootstrap-executor-0] 
>> INFOc.n.d.s.r.aws.ConfigClusterResolver- Resolving eureka endpoints 
>> via configuration
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> 09:10:50.712[AsyncResolver-bootstrap-executor-0] 
>> INFOc.n.d.s.r.aws.ConfigClusterResolver- Resolving eureka endpoints 
>> via configuration
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> 09:15:50.712[AsyncResolver-bootstrap-executor-0] 
>> INFOc.n.d.s.r.aws.ConfigClusterResolver- Resolving eureka endpoints 
>> via configuration
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> 09:16:54.795[qtp1962266146-17] 
>> INFOo.a.f.c.l.c.ServiceExceptionFilter- Unexpected exception caught 
>> /org.springframework.web.util.NestedServletException/: Request 
>> processing failed; nested exception is 
>> /org.springframework.transaction.CannotCreateTransactionException/: 
>> Could not open JPA EntityManager for transaction; nested exception is 
>> /javax.persistence.PersistenceException/: 
>> /java.sql.SQLNonTransientConnectionException/: Could not read 
>> resultset: Connection reset
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Query is : set 
>> autocommit=0
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> 09:16:54.798[qtp1962266146-17] WARNo.e.jetty.servlet.ServletHandler-
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | 
>> /org.springframework.web.util.NestedServletException/: Request 
>> processing failed; nested exception is 
>> /org.springframework.transaction.CannotCreateTransactionException/: 
>> Could not open JPA EntityManager for transaction; nested exception is 
>> /javax.persistence.PersistenceException/: 
>> /java.sql.SQLNonTransientConnectionException/: Could not read 
>> resultset: Connection reset
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Query is : set 
>> autocommit=0
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:225)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.apache.fineract.cn.lang.config.ServiceExceptionFilter.doFilterInternal(ServiceExceptionFilter.java:44)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.apache.fineract.cn.anubis.filter.UserContextFilter.doFilterInternal(UserContextFilter.java:51)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract   |     at 
>> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:121)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.apache.fineract.cn.lang.config.TenantHeaderFilter.doFilterInternal(TenantHeaderFilter.java:53)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.Server.handle(Server.java:524)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> java.lang.Thread.run(Thread.java:748)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Caused by: 
>> /org.springframework.transaction.CannotCreateTransactionException/: 
>> Could not open JPA EntityManager for transaction; nested exception is 
>> /javax.persistence.PersistenceException/: 
>> /java.sql.SQLNonTransientConnectionException/: Could not read 
>> resultset: Connection reset
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Query is : set 
>> autocommit=0
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.apache.fineract.cn.office.internal.service.OfficeService$$EnhancerBySpringCGLIB$$d825377c.fetchOffices(<generated>)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.apache.fineract.cn.office.rest.controller.OfficeRestController.fetchOffices(OfficeRestController.java:132)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> java.lang.reflect.Method.invoke(Method.java:498)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     ... 87common 
>> frames omitted
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Caused by: 
>> /javax.persistence.PersistenceException/: 
>> /java.sql.SQLNonTransientConnectionException/: Could not read 
>> resultset: Connection reset
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> /org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException/(AbstractEntityManagerImpl.java:1700)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     ... 108common 
>> frames omitted
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Caused by: 
>> /java.sql.SQLNonTransientConnectionException/: Could not read 
>> resultset: Connection reset
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Query is : set 
>> autocommit=0
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> /org.mariadb.jdbc.internal.util.ExceptionMapper.throwException/(ExceptionMapper.java:69)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     ... 110common 
>> frames omitted
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Caused by: 
>> /org.mariadb.jdbc.internal.util.dao.QueryException/: Could not read 
>> resultset: Connection reset
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Query is : set 
>> autocommit=0
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     ... 119common 
>> frames omitted
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract | Caused by: 
>> /java.net.SocketException/: Connection reset
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> java.net.SocketInputStream.read(SocketInputStream.java:210)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> java.net.SocketInputStream.read(SocketInputStream.java:141)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:166)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     at 
>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
>>
>> applications_office-ms.1.o15rgpqagwmv@Fineract |     ... 121common 
>> frames omitted
>>
>> On the MariaDB side of the things it just says “Got an error reading 
>> communication packets” and closes a bunch of connections. I have 
>> tried changing the “wait_timeout” as mentioned in this issue: 
>> https://github.com/docker-library/mariadb/issues/113 
>> <https://github.com/docker-library/mariadb/issues/113> and have also 
>> increased the max_communication_packet size to no avail. My BoneCP 
>> pool settings are the default as follows:
>>
>> bonecp:
>>
>>   idleMaxAgeInMinutes: 240
>>
>>   idleConnectionTestPeriodInMinutes: 60
>>
>>   maxConnectionsPerPartition: 10
>>
>>   minConnectionsPerPartition: 1
>>
>>   partitionCount: 2
>>
>>   acquireIncrement: 5
>>
>>   statementsCacheSize: 100
>>
>> If anyone has any insight on the matter or has experienced the same 
>> set of problems before any help would be much appreciated.
>>
>> Thanks and Regards
>>
>> Udayan Tandon
>>
>> https://github.com/udayan12167
>>
>> https://www.threewheelsunited.com/
>>

RE: MariaDB connection getting reset in microservices

Posted by Udayan Tandon <ud...@threewheelsunited.com>.

Hi Victor,


  1.  I’m using Java version 1.8. To build every docker image. Basically using the Dockerfiles provided in each repo of each Fineract CN microservice.
  2.  MariaDB 10.4
  3.  MariaDB is running in a container inside a docker swarm on the same host as the microservices which are running in their own separate containers.
  4.  I created a docker network and all the containers run on them and thus MariaDB is connected to other services through this network. The DB connection does happen initially and as I provision services for the tenant it even runs Flyway migrations perfectly. But then starts erroring out and refuses to reconnect unless I restart and re-provision the service.
  5.  Yes it is docker CE v18.09.7 running on an Azure Ubuntu VM. The MariaDB side shows the following logs:
Aborted connection #SNo. to db: 'playground' user: 'root' host: '172.16.238.69' (Got an error reading communication packets)
Where the host IP address is actually the IP of the internal docker network. As I mentioned before I have tried to tinker and corrected all obvious properties on the MariaDB side that can be the cause of that error.

Let me know if you have more questions.

Thanks and Regards
Udayan Tandon

________________________________
From: Victor Manuel Romero Rodriguez <vi...@hotmail.com>
Sent: Monday, July 29, 2019 10:56:10 PM
To: dev@fineract.apache.org <de...@fineract.apache.org>
Subject: Re: MariaDB connection getting reset in microservices

Hi Udayan,

I have questions to try to help further:

1. Which Java version are you using to run the services?
2. Which is the MariaDB version used?
3. MariaDB is running in the same host or in another? in VM, Container or in a bare bone server?
4. Which kind of networking have you set in Docker?
5. It is Docker CE? any event or log is shown in the MariaDB side?

Please let me know.

About our experience have been growing in Fineract-CN and let me explain the way we did it

1st. We have improved the DB connection replacing BoneCP with HikariCP.

2nd. We added a HA Proxy in Front of a Galera Cluster for MariaDB. (We have timeouts but it is expected because the pooling and the timeout set in the HA Proxy, but connection is recovered and it is only a warning, by the way we use the least connection algorithm for balancing).

3rd. Some customers has requested to use the MariaDB services in the cloud (AWS, GCP and Azure).

4th. For high volume/rate of transactions we had had to create connectors and some other stuff in the microservices to make it work with Oracle and TiDB. Which is more complex but they are different use cases.

Regards

Victor Romero


El 25/07/19 a las 7:40, Udayan Tandon escribió:
Hi guys,

I have been trying to setup Fineract CN on top of docker swarm. I have largely been successful in doing so and the microservices are up and running with a tenant and a bunch of users created. A constant problem that I have been facing is that after a while a microservice has been up say such as the office microservice it gives an error when it tries to fetch data from MariaDB instance running in another container. This is the error log that it gives:

applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:55:51.125 [main] INFO  o.a.f.cn.office.OfficeApplication - Started OfficeApplication in 14.768 seconds (JVM running for 15.77)
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC username not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC password not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:00:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:05:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:10:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:15:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.795 [qtp1962266146-17] INFO  o.a.f.c.l.c.ServiceExceptionFilter - Unexpected exception caught org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.798 [qtp1962266146-17] WARN  o.e.jetty.servlet.ServletHandler -
applications_office-ms.1.o15rgpqagwmv@Fineract    | org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:225)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.ServiceExceptionFilter.doFilterInternal(ServiceExceptionFilter.java:44)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.anubis.filter.UserContextFilter.doFilterInternal(UserContextFilter.java:51)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:121)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.TenantHeaderFilter.doFilterInternal(TenantHeaderFilter.java:53)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.Server.handle(Server.java:524)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.Thread.run(Thread.java:748)
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.internal.service.OfficeService$$EnhancerBySpringCGLIB$$d825377c.fetchOffices(<generated>)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.rest.controller.OfficeRestController.fetchOffices(OfficeRestController.java:132)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.reflect.Method.invoke(Method.java:498)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 87 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 108 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 110 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 119 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.net.SocketException: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:210)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:166)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 121 common frames omitted

On the MariaDB side of the things it just says “Got an error reading communication packets” and closes a bunch of connections. I have tried changing the “wait_timeout” as mentioned in this issue: https://github.com/docker-library/mariadb/issues/113 and have also increased the max_communication_packet size to no avail. My BoneCP pool settings are the default as follows:

bonecp:
  idleMaxAgeInMinutes: 240
  idleConnectionTestPeriodInMinutes: 60
  maxConnectionsPerPartition: 10
  minConnectionsPerPartition: 1
  partitionCount: 2
  acquireIncrement: 5
  statementsCacheSize: 100

If anyone has any insight on the matter or has experienced the same set of problems before any help would be much appreciated.

Thanks and Regards
Udayan Tandon
https://github.com/udayan12167
https://www.threewheelsunited.com/

Re: MariaDB connection getting reset in microservices

Posted by Victor Manuel Romero Rodriguez <vi...@hotmail.com>.

Hi Udayan,

I have questions to try to help further:

1. Which Java version are you using to run the services?
2. Which is the MariaDB version used?
3. MariaDB is running in the same host or in another? in VM, Container or in a bare bone server?
4. Which kind of networking have you set in Docker?
5. It is Docker CE? any event or log is shown in the MariaDB side?

Please let me know.

About our experience have been growing in Fineract-CN and let me explain the way we did it

1st. We have improved the DB connection replacing BoneCP with HikariCP.

2nd. We added a HA Proxy in Front of a Galera Cluster for MariaDB. (We have timeouts but it is expected because the pooling and the timeout set in the HA Proxy, but connection is recovered and it is only a warning, by the way we use the least connection algorithm for balancing).

3rd. Some customers has requested to use the MariaDB services in the cloud (AWS, GCP and Azure).

4th. For high volume/rate of transactions we had had to create connectors and some other stuff in the microservices to make it work with Oracle and TiDB. Which is more complex but they are different use cases.

Regards

Victor Romero


El 25/07/19 a las 7:40, Udayan Tandon escribió:
Hi guys,

I have been trying to setup Fineract CN on top of docker swarm. I have largely been successful in doing so and the microservices are up and running with a tenant and a bunch of users created. A constant problem that I have been facing is that after a while a microservice has been up say such as the office microservice it gives an error when it tries to fetch data from MariaDB instance running in another container. This is the error log that it gives:

applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:55:51.125 [main] INFO  o.a.f.cn.office.OfficeApplication - Started OfficeApplication in 14.768 seconds (JVM running for 15.77)
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC username not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 08:56:03.411 [qtp1962266146-20] WARN  com.jolbox.bonecp.BoneCPConfig - JDBC password not set in driver properties, copying it from pool config
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:00:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:05:50.711 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:10:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:15:50.712 [AsyncResolver-bootstrap-executor-0] INFO  c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via configuration
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.795 [qtp1962266146-17] INFO  o.a.f.c.l.c.ServiceExceptionFilter - Unexpected exception caught org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    | 09:16:54.798 [qtp1962266146-17] WARN  o.e.jetty.servlet.ServletHandler -
applications_office-ms.1.o15rgpqagwmv@Fineract    | org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:225)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.ServiceExceptionFilter.doFilterInternal(ServiceExceptionFilter.java:44)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.anubis.filter.UserContextFilter.doFilterInternal(UserContextFilter.java:51)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:121)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.lang.config.TenantHeaderFilter.doFilterInternal(TenantHeaderFilter.java:53)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.Server.handle(Server.java:524)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.Thread.run(Thread.java:748)
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.internal.service.OfficeService$$EnhancerBySpringCGLIB$$d825377c.fetchOffices(<generated>)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.apache.fineract.cn.office.rest.controller.OfficeRestController.fetchOffices(OfficeRestController.java:132)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.lang.reflect.Method.invoke(Method.java:498)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 87 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 108 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 110 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not read resultset: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    | Query is : set autocommit=0
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 119 common frames omitted
applications_office-ms.1.o15rgpqagwmv@Fineract    | Caused by: java.net.SocketException: Connection reset
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:210)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.net.SocketInputStream.read(SocketInputStream.java:141)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:166)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
applications_office-ms.1.o15rgpqagwmv@Fineract    |     ... 121 common frames omitted

On the MariaDB side of the things it just says “Got an error reading communication packets” and closes a bunch of connections. I have tried changing the “wait_timeout” as mentioned in this issue: https://github.com/docker-library/mariadb/issues/113 and have also increased the max_communication_packet size to no avail. My BoneCP pool settings are the default as follows:

bonecp:
  idleMaxAgeInMinutes: 240
  idleConnectionTestPeriodInMinutes: 60
  maxConnectionsPerPartition: 10
  minConnectionsPerPartition: 1
  partitionCount: 2
  acquireIncrement: 5
  statementsCacheSize: 100

If anyone has any insight on the matter or has experienced the same set of problems before any help would be much appreciated.

Thanks and Regards
Udayan Tandon
https://github.com/udayan12167
https://www.threewheelsunited.com/