You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "ZhangZhengming (Jira)" <ji...@apache.org> on 2022/06/28 02:20:00 UTC

[jira] [Created] (IOTDB-3672) [cluster]Kill the first confignode,cluster exception

|  ![](cid:jira-generated-image-avatar-f5e8bd56-f203-4114-bb7e-3c7c4841c19e) |
[ZhangZhengming](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=zhangzhengming)
**created** an issue  
---|---  
|  
---  
|  [Apache IoTDB](https://issues.apache.org/jira/browse/IOTDB) /
[![Bug](cid:jira-generated-image-
avatar-9581a98f-ee71-4e14-9563-492da8423652)](https://issues.apache.org/jira/browse/IOTDB-3672)
[IOTDB-3672](https://issues.apache.org/jira/browse/IOTDB-3672)  
---  
[[cluster]Kill the first confignode,cluster
exception](https://issues.apache.org/jira/browse/IOTDB-3672)  
| Issue Type: |  ![Bug](cid:jira-generated-image-
avatar-9581a98f-ee71-4e14-9563-492da8423652) Bug  
---|---  
Affects Versions: |  0.14.0-SNAPSHOT  
Assignee: |  [Jinrui
Zhang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=xingtanzjr)  
Components: |  Core/Cluster  
Created: |  28/Jun/22 02:19  
Priority: |  ![Major](cid:jira-generated-image-static-
major-b86a4e40-7571-42dd-aef1-5332efc9d911) Major  
Reporter: |
[ZhangZhengming](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=zhangzhengming)  
|

commit 56d6940b01f1e7d85e6a72b50078e8148c6a679e  
Author: Haonan <hh...@outlook.com>  
Date:   Sat Jun 25 13:35:10 2022 +0800

    Change default partition cache size to 0 (#6434)

Reproduce steps:

  1. Setup a cluster with 3C3D(MultiLeaderConsensus)
  2. Using 3BMs to insert data

BM    \-----  》  IoTDB Node

172.20.70.7    \-----  》 172.20.70.22

172.20.70.8    \-----  》 172.20.70.23

172.20.70.9    \-----  》 172.20.70.24

Got error from benchmark log like this:

2022-06-25 18:18:19,784 ERROR
cn.edu.tsinghua.iotdb.benchmark.tsdb.DBWrapper:131 - Insert batch failed
because  
org.apache.iotdb.rpc.StatementExecutionException: 411: Fragment cannot be
dispatched  
        at org.apache.iotdb.rpc.RpcUtils.verifySuccess(RpcUtils.java:94)  
        at org.apache.iotdb.rpc.RpcUtils.verifySuccessWithRedirection(RpcUtils.java:115)  
        at org.apache.iotdb.session.SessionConnection.insertTablet(SessionConnection.java:589)  
        at org.apache.iotdb.session.Session.insertTablet(Session.java:1573)  
        at org.apache.iotdb.session.Session.insertTablet(Session.java:1560)  
        at cn.edu.tsinghua.iotdb.benchmark.iotdb013.IoTDBSession.insertOneBatchByTablet(IoTDBSession.java:165)  
        at cn.edu.tsinghua.iotdb.benchmark.iotdb013.IoTDBSessionBase.insertOneBatch(IoTDBSessionBase.java:156)  
        at cn.edu.tsinghua.iotdb.benchmark.tsdb.DBWrapper.insertOneBatch(DBWrapper.java:83)  
        at cn.edu.tsinghua.iotdb.benchmark.client.generate.GenerateDataMixClient.ingestionOperation(GenerateDataMixClient.java:135)  
        at cn.edu.tsinghua.iotdb.benchmark.client.generate.GenerateDataMixClient.doTest(GenerateDataMixClient.java:50)  
        at cn.edu.tsinghua.iotdb.benchmark.client.DataClient.run(DataClient.java:144)  
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
        at java.lang.Thread.run(Thread.java:748)

![noimage.png](cid:jira-generated-image-static-noimage-
bdc7f0a4-a802-4703-9fd0-c7ed3181539e)

IoTDB error msg:

2022-06-25 18:19:45,759 [20220625_101925_02042_3-300] ERROR
o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:151 - cannot dispatch FI for
write operation  
org.apache.iotdb.db.exception.mpp.FragmentInstanceDispatchException:
org.apache.thrift.TException: Error in calling method sendPlanNode  
        at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:195)  
        at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:163)  
        at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:144)  
        at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:93)  
        at org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:95)  
        at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:215)  
        at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:169)  
        at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:131)  
        at org.apache.iotdb.db.service.thrift.impl.DataNodeTSIServiceImpl.insertTablet(DataNodeTSIServiceImpl.java:911)  
        at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablet.getResult(TSIService.java:3328)  
        at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablet.getResult(TSIService.java:3308)  
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)  
        at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)  
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)  
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
        at java.lang.Thread.run(Thread.java:748)  
Caused by: org.apache.thrift.TException: Error in calling method sendPlanNode  
        at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94)  
        at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$ae445e1.sendPlanNode(<generated>)  
        at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:190)  
        ... 16 common frames omitted  
Caused by: org.apache.thrift.TException: Error in calling method
recv_sendPlanNode  
        at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94)  
        at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$ae445e1.recv_sendPlanNode(<generated>)  
        at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.sendPlanNode(InternalService.java:223)  
        at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$ae445e1.CGLIB$sendPlanNode$41(<generated>)  
        at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$ae445e1$$FastClassByCGLIB$$9ee0aca5.invoke(<generated>)  
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)  
        at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)  
        ... 18 common frames omitted  
Caused by: org.apache.thrift.TException: Error in calling method receiveBase  
        at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94)  
        at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$ae445e1.receiveBase(<generated>)  
        at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_sendPlanNode(InternalService.java:236)  
        at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$ae445e1.CGLIB$recv_sendPlanNode$11(<generated>)  
        at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$ae445e1$$FastClassByCGLIB$$9ee0aca5.invoke(<generated>)  
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)  
        at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)  
        ... 24 common frames omitted

![noimage.png](cid:jira-generated-image-static-noimage-
bdc7f0a4-a802-4703-9fd0-c7ed3181539e)  
  
---  
|  |  [ ![Add Comment](cid:jira-generated-image-static-comment-
icon-e85aa502-0db6-49ed-a671-5463482dcac6)
](https://issues.apache.org/jira/browse/IOTDB-3672#add-comment "Add Comment")
|  [Add Comment](https://issues.apache.org/jira/browse/IOTDB-3672#add-comment
"Add Comment")  
---|---  
  
|  This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9) |  |
![Atlassian logo](https://issues.apache.org/jira/images/mail/atlassian-email-
logo.png)  
---