You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "Gaofei Cao (Jira)" <ji...@apache.org> on 2022/11/02 07:42:00 UTC
[jira] [Assigned] (IOTDB-4703) 3rep/3C3D , stop 1 datanode ,delete storage group root.** failed
[ https://issues.apache.org/jira/browse/IOTDB-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gaofei Cao reassigned IOTDB-4703:
---------------------------------
Assignee: Yongzao Dan (was: Gaofei Cao)
> 3rep/3C3D , stop 1 datanode ,delete storage group root.** failed
> ----------------------------------------------------------------
>
> Key: IOTDB-4703
> URL: https://issues.apache.org/jira/browse/IOTDB-4703
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
> Affects Versions: 0.14.0-SNAPSHOT
> Reporter: 刘珍
> Assignee: Yongzao Dan
> Priority: Major
>
> m_1019_f2ffb49
> 3rep ,3C3D
> stop ip73 datanode , ip72 cli execute "delete storage group root.**"
> ip72 datanode error log：
> 2022-10-21 09:16:09,286 [pool-26-IoTDB-ClientRPC-Processor-987$20221021_011534_06996_3] ERROR o.a.i.d.m.p.e.c.e.ClusterConfigTaskExecutor:248 - Failed to execute delete storage group [root.**] in config node, status is TSStatus(code:506, subStatus:[TSStatus(code:400, {color:#DE350B}message:Invalidate cache failed), TSStatus(code:400, message:Invalidate cache failed), TSStatus(code:400, message:Invalidate cache failed)]).{color}
> 2022-10-21 09:16:09,288 [pool-23-IoTDB-MPPCoordinator-17] ERROR o.a.i.d.m.p.e.c.ConfigExecution:124 - Failures happened during running ConfigExecution.
> org.apache.iotdb.commons.exception.IoTDBException: null
> at org.apache.iotdb.db.mpp.plan.execution.config.executor.ClusterConfigTaskExecutor.deleteStorageGroup(ClusterConfigTaskExecutor.java:252)
> at org.apache.iotdb.db.mpp.plan.execution.config.metadata.DeleteStorageGroupTask.execute(DeleteStorageGroupTask.java:41)
> at org.apache.iotdb.db.mpp.plan.execution.config.ConfigExecution.start(ConfigExecution.java:99)
> at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:146)
> at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatement(ClientRPCServiceImpl.java:601)
> at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatement.getResult(IClientRPCService.java:3037)
> at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatement.getResult(IClientRPCService.java:3017)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> ip72 confignode error log：
> 2022-10-21 09:22:03,847 [ProcExecWorker-2] ERROR o.a.i.c.c.s.SyncDataNodeClientPool:113 -{color:#DE350B}* INVALIDATE_SCHEMA_CACHE failed on DataNode TEndPoint(ip:192.168.10.73, port:9003)*{color}
> java.io.IOException: Borrow client from pool for node TEndPoint(ip:192.168.10.73, port:9003) failed.
> at org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:61)
> at org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool.sendSyncRequestToDataNodeWithRetry(SyncDataNodeClientPool.java:68)
> at org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv.invalidateCache(ConfigNodeProcedureEnv.java:147)
> at org.apache.iotdb.confignode.procedure.impl.statemachine.DeleteStorageGroupProcedure.executeFromState(DeleteStorageGroupProcedure.java:90)
> at org.apache.iotdb.confignode.procedure.impl.statemachine.DeleteStorageGroupProcedure.executeFromState(DeleteStorageGroupProcedure.java:47)
> at org.apache.iotdb.confignode.procedure.impl.statemachine.StateMachineProcedure.execute(StateMachineProcedure.java:186)
> at org.apache.iotdb.confignode.procedure.Procedure.doExecute(Procedure.java:365)
> at org.apache.iotdb.confignode.procedure.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:414)
> at org.apache.iotdb.confignode.procedure.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:373)
> at org.apache.iotdb.confignode.procedure.ProcedureExecutor.access$300(ProcedureExecutor.java:50)
> at org.apache.iotdb.confignode.procedure.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:741)
> Caused by: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.transport.TTransportException-->java.net.ConnectException: Connection refused (Connection refused)
> at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:235)
> at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:220)
> at net.sf.cglib.proxy.Enhancer.createUsingReflection(Enhancer.java:639)
> at net.sf.cglib.proxy.Enhancer.firstInstance(Enhancer.java:538)
> at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:231)
> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
> at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
> at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.newErrorHandler(SyncThriftClientWithErrorHandler.java:48)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:127)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:105)
> at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:780)
> at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:439)
> at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
> at org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:50)
> ... 10 common frames omitted
> Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:243)
> at org.apache.iotdb.rpc.TElasticFramedTransport.open(TElasticFramedTransport.java:91)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient.<init>(SyncDataNodeInternalServiceClient.java:63)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$5e1ad4cd.<init>(<generated>)
> at jdk.internal.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
> at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
> at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:228)
> ... 23 common frames omitted
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
> at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
> at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
> at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.base/java.net.Socket.connect(Socket.java:608)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:238)
> ... 30 common frames omitted
> 2022-10-21 09:22:10,230 [ProcExecWorker-14] ERROR o.a.i.c.p.e.ConfigNodeProcedureEnv:158 -{color:#DE350B}* Invalidate cache failed, invalidate partition cache status is TSStatus*{color}(code:709, message:All retry failed due to: Borrow client from pool for node TEndPoint(ip:192.168.10.73, port:9003) failed.), invalidate schema cache status is TSStatus(code:709, message:All retry failed due to: Borrow client from pool for node TEndPoint(ip:192.168.10.73, port:9003) failed.)
> 2022-10-21 09:22:10,235 [ProcExecWorker-1] ERROR o.a.i.c.c.s.SyncDataNodeClientPool:113 - INVALIDATE_PARTITION_CACHE failed on DataNode TEndPoint(ip:192.168.10.73, port:9003)
> java.io.IOException: Borrow client from pool for node TEndPoint(ip:192.168.10.73, port:9003) failed.
> at org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:61)
> at org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool.sendSyncRequestToDataNodeWithRetry(SyncDataNodeClientPool.java:68)
> at org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv.invalidateCache(ConfigNodeProcedureEnv.java:153)
> at org.apache.iotdb.confignode.procedure.impl.statemachine.DeleteStorageGroupProcedure.executeFromState(DeleteStorageGroupProcedure.java:90)
> at org.apache.iotdb.confignode.procedure.impl.statemachine.DeleteStorageGroupProcedure.executeFromState(DeleteStorageGroupProcedure.java:47)
> at org.apache.iotdb.confignode.procedure.impl.statemachine.StateMachineProcedure.execute(StateMachineProcedure.java:186)
> at org.apache.iotdb.confignode.procedure.Procedure.doExecute(Procedure.java:365)
> at org.apache.iotdb.confignode.procedure.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:414)
> at org.apache.iotdb.confignode.procedure.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:373)
> at org.apache.iotdb.confignode.procedure.ProcedureExecutor.access$300(ProcedureExecutor.java:50)
> at org.apache.iotdb.confignode.procedure.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:741)
> Caused by: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.transport.TTransportException-->java.net.ConnectException: Connection refused (Connection refused)
> at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:235)
> at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:220)
> at net.sf.cglib.proxy.Enhancer.createUsingReflection(Enhancer.java:639)
> at net.sf.cglib.proxy.Enhancer.firstInstance(Enhancer.java:538)
> at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:231)
> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
> at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
> at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.newErrorHandler(SyncThriftClientWithErrorHandler.java:48)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:127)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:105)
> at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:780)
> at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:439)
> at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
> at org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:50)
> ... 10 common frames omitted
> Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:243)
> at org.apache.iotdb.rpc.TElasticFramedTransport.open(TElasticFramedTransport.java:91)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient.<init>(SyncDataNodeInternalServiceClient.java:63)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$5e1ad4cd.<init>(<generated>)
> at jdk.internal.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
> at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
> at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:228)
> ... 23 common frames omitted
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
> at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
> at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
> at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.base/java.net.Socket.connect(Socket.java:608)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:238)
> ... 30 common frames omitted
> 2022-10-21 09:22:10,236 [ProcExecWorker-1] ERROR o.a.i.c.p.e.ConfigNodeProcedureEnv:158 - Invalidate cache failed, invalidate partition cache status is TSStatus(code:709, message:All retry failed due to: Borrow client from pool for node TEndPoint(ip:192.168.10.73, port:9003) failed.), invalidate schema cache status is TSStatus(code:709, message:All retry failed due to: Borrow client from pool for node TEndPoint(ip:192.168.10.73, port:9003) failed.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)