You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2019/09/27 01:32:01 UTC
[GitHub] [incubator-shardingsphere] maxiaoxian opened a new issue #3127: CPU
LOAD more than 300%
maxiaoxian opened a new issue #3127: CPU LOAD more than 300%
URL: https://github.com/apache/incubator-shardingsphere/issues/3127
The original version of version 3.1.0, frequent yongGC. After upgrading version 4.0.0 RC2, the following problems occur:
top 3 thread stack：
`"globalThreadPoolSlave-166" Id=530 cpuUsage=9% RUNNABLE
at org.apache.commons.collections4.map.AbstractHashedMap.getEntry(AbstractHashedMap.java:461)
at org.apache.commons.collections4.map.AbstractReferenceMap.getEntry(AbstractReferenceMap.java:427)
at org.apache.commons.collections4.map.AbstractReferenceMap.get(AbstractReferenceMap.java:244)
at org.apache.shardingsphere.core.parse.cache.SQLParseResultCache.getSQLStatement(SQLParseResultCache.java:54)
at org.apache.shardingsphere.core.parse.SQLParseEngine.parse0(SQLParseEngine.java:66)
at org.apache.shardingsphere.core.parse.SQLParseEngine.parse(SQLParseEngine.java:53)
at org.apache.shardingsphere.core.route.router.sharding.ParsingSQLRouter.parse(ParsingSQLRouter.java:66)
at org.apache.shardingsphere.core.route.PreparedStatementRoutingEngine.route(PreparedStatementRoutingEngine.java:64)
at org.apache.shardingsphere.core.PreparedQueryShardingEngine.route(PreparedQueryShardingEngine.java:60)
at org.apache.shardingsphere.core.BaseShardingEngine.executeRoute(BaseShardingEngine.java:86)
at org.apache.shardingsphere.core.BaseShardingEngine.shard(BaseShardingEngine.java:70)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.shard(ShardingPreparedStatement.java:224)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.execute(ShardingPreparedStatement.java:170)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:63)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
at com.sun.proxy.$Proxy147.query(Unknown Source)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:326)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:136)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141)
at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
at com.sun.proxy.$Proxy96.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
at com.baomidou.mybatisplus.core.override.PageMapperMethod.executeForMany(PageMapperMethod.java:173)
at com.baomidou.mybatisplus.core.override.PageMapperMethod.execute(PageMapperMethod.java:86)
at com.baomidou.mybatisplus.core.override.PageMapperProxy.invoke(PageMapperProxy.java:64)
at com.sun.proxy.$Proxy106.commonQuery(Unknown Source)
at sun.reflect.GeneratedMethodAccessor544.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy107.commonQuery(Unknown Source)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl.commonQuery(DtSyncRuleServiceImpl.java:109)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl.oneQuery(DtSyncRuleServiceImpl.java:146)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl$$FastClassBySpringCGLIB$$e914ca73.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl$$EnhancerBySpringCGLIB$$83a80106.oneQuery(<generated>)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl.lambda$syncMainTable$1(DataToEsServiceImpl.java:184)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl$$Lambda$420/539399554.executeInvokeLogic(Unknown Source)
at com.midea.ec.fc.datacenter.common.utils.RedisLockUtil.executeSynchOperate(RedisLockUtil.java:54)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl.syncMainTable(DataToEsServiceImpl.java:170)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$2.mainExec(DataToEsConsumer.java:100)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$Data2EsProcess.exec(DataToEsConsumer.java:135)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer.exec(DataToEsConsumer.java:107)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer.lambda$null$0(DataToEsConsumer.java:64)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$$Lambda$322/1023403346.run(Unknown Source)
at org.apache.skywalking.apm.toolkit.trace.RunnableWrapper.run(RunnableWrapper.java:37)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@20a40fc8
"ConsumeMessageThread_5" Id=98 cpuUsage=8% RUNNABLE
at org.apache.commons.collections4.map.AbstractReferenceMap.purge(AbstractReferenceMap.java:400)
at org.apache.commons.collections4.map.AbstractReferenceMap.purge(AbstractReferenceMap.java:381)
at org.apache.commons.collections4.map.AbstractReferenceMap.purgeBeforeRead(AbstractReferenceMap.java:358)
at org.apache.commons.collections4.map.AbstractReferenceMap.get(AbstractReferenceMap.java:243)
at org.apache.shardingsphere.core.parse.cache.SQLParseResultCache.getSQLStatement(SQLParseResultCache.java:54)
at org.apache.shardingsphere.core.parse.SQLParseEngine.parse0(SQLParseEngine.java:66)
at org.apache.shardingsphere.core.parse.SQLParseEngine.parse(SQLParseEngine.java:53)
at org.apache.shardingsphere.core.route.router.sharding.ParsingSQLRouter.parse(ParsingSQLRouter.java:66)
at org.apache.shardingsphere.core.route.PreparedStatementRoutingEngine.route(PreparedStatementRoutingEngine.java:64)
at org.apache.shardingsphere.core.PreparedQueryShardingEngine.route(PreparedQueryShardingEngine.java:60)
at org.apache.shardingsphere.core.BaseShardingEngine.executeRoute(BaseShardingEngine.java:86)
at org.apache.shardingsphere.core.BaseShardingEngine.shard(BaseShardingEngine.java:70)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.shard(ShardingPreparedStatement.java:224)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.execute(ShardingPreparedStatement.java:170)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:63)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
at com.sun.proxy.$Proxy147.query(Unknown Source)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:326)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:136)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141)
at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
at com.sun.proxy.$Proxy96.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
at com.baomidou.mybatisplus.core.override.PageMapperMethod.executeForMany(PageMapperMethod.java:173)
at com.baomidou.mybatisplus.core.override.PageMapperMethod.execute(PageMapperMethod.java:86)
at com.baomidou.mybatisplus.core.override.PageMapperProxy.invoke(PageMapperProxy.java:64)
at com.sun.proxy.$Proxy106.commonQuery(Unknown Source)
at sun.reflect.GeneratedMethodAccessor544.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy107.commonQuery(Unknown Source)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl.commonQuery(DtSyncRuleServiceImpl.java:109)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl.oneQuery(DtSyncRuleServiceImpl.java:146)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl$$FastClassBySpringCGLIB$$e914ca73.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl$$EnhancerBySpringCGLIB$$83a80106.oneQuery(<generated>)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl.lambda$syncMainTable$1(DataToEsServiceImpl.java:184)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl$$Lambda$420/539399554.executeInvokeLogic(Unknown Source)
at com.midea.ec.fc.datacenter.common.utils.RedisLockUtil.executeSynchOperate(RedisLockUtil.java:54)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl.syncMainTable(DataToEsServiceImpl.java:170)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$2.mainExec(DataToEsConsumer.java:100)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$Data2EsProcess.exec(DataToEsConsumer.java:135)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer.exec(DataToEsConsumer.java:107)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer.lambda$null$0(DataToEsConsumer.java:64)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$$Lambda$322/1023403346.run(Unknown Source)
at org.apache.skywalking.apm.toolkit.trace.RunnableWrapper.run(RunnableWrapper.java:37)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:322)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer.lambda$consume$1(DataToEsConsumer.java:64)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$$Lambda$320/956448191.consumeMessage(Unknown Source)
at com.alibaba.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:155)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@7b38e392
"ConsumeMessageThread_1" Id=85 cpuUsage=6% RUNNABLE
at java.lang.Class.getDeclaringClass0(Native Method)
at java.lang.Class.getDeclaringClass(Class.java:1235)
at java.lang.Class.getEnclosingClass(Class.java:1277)
at java.lang.Class.getSimpleBinaryName(Class.java:1443)
at java.lang.Class.getSimpleName(Class.java:1309)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.isMatchedNode(ExtractorUtils.java:138)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:128)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:132)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:132)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:132)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:132)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:132)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:132)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:132)
at org.apache.shardingsphere.core.parse.core.extractor.util.ExtractorUtils.getAllDescendantNodes(ExtractorUtils.java:132)
at org.apache.shardingsphere.core.parse.core.extractor.impl.dml.select.SubqueryPredicateExtractor.extract(SubqueryPredicateExtractor.java:44)
at org.apache.shardingsphere.core.parse.core.extractor.SQLSegmentsExtractorEngine.extract(SQLSegmentsExtractorEngine.java:52)
at org.apache.shardingsphere.core.parse.core.SQLParseKernel.parse(SQLParseKernel.java:62)
at org.apache.shardingsphere.core.parse.SQLParseEngine.parse0(SQLParseEngine.java:71)
at org.apache.shardingsphere.core.parse.SQLParseEngine.parse(SQLParseEngine.java:53)
at org.apache.shardingsphere.core.route.router.sharding.ParsingSQLRouter.parse(ParsingSQLRouter.java:66)
at org.apache.shardingsphere.core.route.PreparedStatementRoutingEngine.route(PreparedStatementRoutingEngine.java:64)
at org.apache.shardingsphere.core.PreparedQueryShardingEngine.route(PreparedQueryShardingEngine.java:60)
at org.apache.shardingsphere.core.BaseShardingEngine.executeRoute(BaseShardingEngine.java:86)
at org.apache.shardingsphere.core.BaseShardingEngine.shard(BaseShardingEngine.java:70)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.shard(ShardingPreparedStatement.java:224)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.execute(ShardingPreparedStatement.java:170)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:63)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
at com.sun.proxy.$Proxy147.query(Unknown Source)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:326)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:136)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141)
at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
at com.sun.proxy.$Proxy96.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
at com.baomidou.mybatisplus.core.override.PageMapperMethod.executeForMany(PageMapperMethod.java:173)
at com.baomidou.mybatisplus.core.override.PageMapperMethod.execute(PageMapperMethod.java:86)
at com.baomidou.mybatisplus.core.override.PageMapperProxy.invoke(PageMapperProxy.java:64)
at com.sun.proxy.$Proxy106.commonQuery(Unknown Source)
at sun.reflect.GeneratedMethodAccessor544.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy107.commonQuery(Unknown Source)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl.commonQuery(DtSyncRuleServiceImpl.java:109)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl.oneQuery(DtSyncRuleServiceImpl.java:146)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl$$FastClassBySpringCGLIB$$e914ca73.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
at com.midea.ec.fc.datacenter.fc.service.impl.DtSyncRuleServiceImpl$$EnhancerBySpringCGLIB$$83a80106.oneQuery(<generated>)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl.lambda$syncMainTable$1(DataToEsServiceImpl.java:184)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl$$Lambda$420/539399554.executeInvokeLogic(Unknown Source)
at com.midea.ec.fc.datacenter.common.utils.RedisLockUtil.executeSynchOperate(RedisLockUtil.java:54)
at com.midea.ec.fc.datacenter.fastindex.service.impl.DataToEsServiceImpl.syncMainTable(DataToEsServiceImpl.java:170)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$2.mainExec(DataToEsConsumer.java:100)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$Data2EsProcess.exec(DataToEsConsumer.java:135)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer.exec(DataToEsConsumer.java:107)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer.lambda$null$0(DataToEsConsumer.java:64)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$$Lambda$322/1023403346.run(Unknown Source)
at org.apache.skywalking.apm.toolkit.trace.RunnableWrapper.run(RunnableWrapper.java:37)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:322)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer.lambda$consume$1(DataToEsConsumer.java:64)
at com.midea.ec.fc.datacenter.mq.receiver.fastindex.DataToEsConsumer$$Lambda$320/956448191.consumeMessage(Unknown Source)
at com.alibaba.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:155)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@2783726b
Affect(row-cnt:0) cost in 1911 ms.
`
top 10 threads order by CPU：
` 08:16:09 - PID: 1330 JVM: 1.8.0_191 USER: pub UPTIME: 12h16m
PROCESS: 352.97% cpu(88.24% of 4 core), 390 thread
MEMORY: 9034m rss, 9046m peak, 0m swap | DISK: 0B read, 54kB write
THREAD: 367 live, 274 daemon, 367 peak, 0 new | CLASS: 17468 loaded, 123 unloaded, 5 new
HEAP: 2952m/5072m/NaN eden, 88m/88m/NaN sur, 1363m/3032m/8192m old
NON-HEAP: 104m/109m/NaN metaspace, 63m/65m/240m codeCache, 11m/12m/1024m ccs
OFF-HEAP: 0m/0m direct(max=NaN), 0m/0m map(count=0), 367m threadStack
GC: 0/0ms/0ms ygc, 0/0ms fgc | SAFE-POINT: 2 count, 30ms time, 24ms syncTime
TID NAME STATE CPU SYSCPU TOTAL TOLSYS
98 ConsumeMessageThread_5 RUNNABLE 12.75% 0.29% 2.75% 0.01%
544 globalThreadPoolSlave-179 RUNNABLE 11.62% 0.00% 1.86% 0.00%
514 globalThreadPoolSlave-150 RUNNABLE 11.13% 0.00% 4.00% 0.00%
504 globalThreadPoolSlave-136 RUNNABLE 11.13% 0.00% 4.21% 0.00%
588 globalThreadPoolSlave-217 RUNNABLE 10.86% 0.09% 0.11% 0.00%
524 globalThreadPoolSlave-160 RUNNABLE 10.48% 0.00% 1.91% 0.00%
498 globalThreadPoolSlave-122 RUNNABLE 10.34% 0.14% 1.96% 0.00%
99 ConsumeMessageThread_4 RUNNABLE 10.28% 0.08% 3.45% 0.01%
553 globalThreadPoolSlave-188 RUNNABLE 10.25% 0.05% 1.85% 0.00%
557 globalThreadPoolSlave-185 RUNNABLE 10.15% 0.00% 1.85% 0.00%
Total : 240.51% cpu(user=237.11%, sys=3.40%) by 36 active java threads, 0.06% by others
Setting: top 10 threads order by CPU, flush every 1s
`
jvm collector :
`-server -Xms8g -Xmx8g -XX:+UseG1GC`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services