You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "刘珍 (Jira)" <ji...@apache.org> on 2022/06/14 07:37:00 UTC
[jira] [Assigned] (IOTDB-3483) Internal error processing onEndOfDataBlockEvent / count devices returns "Empty set"
[ https://issues.apache.org/jira/browse/IOTDB-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
刘珍 reassigned IOTDB-3483:
-------------------------
Assignee: Jialin Qiao
> Internal error processing onEndOfDataBlockEvent / count devices returns "Empty set"
> ------------------------------------------------------------------------------------
>
> Key: IOTDB-3483
> URL: https://issues.apache.org/jira/browse/IOTDB-3483
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
> Affects Versions: 0.14.0-SNAPSHOT
> Reporter: 刘珍
> Assignee: Jialin Qiao
> Priority: Major
> Attachments: IOTDB-3483.zip, config.properties, image-2022-06-14-15-20-52-189.png, iotdb-confignode.properties, iotdb-engine.properties
>
>
> master_0613_38f3332
> 3节点3副本
> DEVICE_NUMBER=1000
> SENSOR_NUMBER=6000
> CLIENT_NUMBER=100
> GROUP_NUMBER=1
> BATCH_SIZE_PER_WRITE=10
> LOOP=10000
> OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0
> 3小时20分钟后，datanode error：
> 2022-06-14 14:17:30,613 [20220614_061627_19086_2.1.0-38] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TApplicationException, message Internal error processing onEndOfDataBlockEvent
> 2022-06-14 14:17:30,614 [20220614_061657_19097_2.1.0-39] ERROR o.a.i.d.m.e.driver.Driver:195 - Failed to execute fragment instance 20220614_061657_19097_2.1.0
> java.lang.IllegalStateException: Sink handle is aborted.
> at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
> at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
> at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
> 2022-06-14 14:17:30,614 [20220614_061627_19086_2.1.0-38] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal error processing onEndOfDataBlockEvent, LocalizedMessage Internal error processing onEndOfDataBlockEvent,
> org.apache.thrift.TApplicationException: Internal error processing onEndOfDataBlockEvent
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$receiveBase$20(<generated>)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
> at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.receiveBase(<generated>)
> at org.apache.iotdb.mpp.rpc.thrift.DataBlockService$Client.recv_onEndOfDataBlockEvent(DataBlockService.java:136)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$recv_onEndOfDataBlockEvent$13(<generated>)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
> at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.recv_onEndOfDataBlockEvent(<generated>)
> at org.apache.iotdb.mpp.rpc.thrift.DataBlockService$Client.onEndOfDataBlockEvent(DataBlockService.java:123)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$onEndOfDataBlockEvent$16(<generated>)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
> at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.onEndOfDataBlockEvent(<generated>)
> at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.sendEndOfDataBlockEvent(SinkHandle.java:174)
> at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.setNoMoreTsBlocks(SinkHandle.java:198)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.closeAndDestroyOperators(Driver.java:338)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.destroyIfNecessary(Driver.java:312)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:277)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
> at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
> at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
> 集群时间节点的细节信息：
> 2022-06-14 10:50:55 启动benchmark
> 2022-06-14 10:58:30,935 bm创建元数据成功
> 查看3个节点的mlog.bin , ip62的元数据先落盘了一部分 (其余2个节点完全落盘）
> !image-2022-06-14-15-20-52-189.png!
> 50多分钟后ip62的元数据刷盘：
> liuzhen@fit-62:/data/liuzhen_test/master_0613_38f3332/datanode$ find ./data -name mlog.bin|xargs ls -lort
> -rw-rw-r-- 1 liuzhen 430230000 Jun 14 11:45 ./data/system/schema/root.bm68_dn62.g_0/0/mlog.bin
> count devices;
> count timeseries ； 结果正确。
> 出现异常后：
> {color:red}count devices 返回Empty set.{color}
> 还有其它异常：
> 2022-06-14 14:17:30,614 [20220614_061657_19097_2.1.0-39] ERROR o.a.i.d.m.e.driver.Driver:195 - Failed to execute fragment instance 20220614_061657_19097_2.1.0
> java.lang.IllegalStateException: Sink handle is aborted.
> at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
> at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
> at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
> 2022-06-14 14:17:30,617 [Worker-Thread-3] ERROR o.a.i.d.m.e.s.AbstractDriverThread:61 - Executor Worker-Thread-3 processes failed
> java.lang.RuntimeException: Driver was interrupted
> at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:207)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
> at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
> at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
> Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
> at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
> ... 5 common frames omitted
> Caused by: java.lang.Exception: Interrupted By
> at java.lang.Thread.getStackTrace(Thread.java:1559)
> at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:444)
> at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:150)
> at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:114)
> at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
> at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 复现流程：
> 1. 192.168.10.62/66/68 72C256G
> 2. bm 连ip62 （配置见附件）
> 3. 各节点datanode log见附件。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)