You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "刘珍 (Jira)" <ji...@apache.org> on 2022/06/14 07:28:00 UTC
[jira] [Created] (IOTDB-3483) Internal error processing onEndOfDataBlockEvent / count devices returns "Empty set"
刘珍 created IOTDB-3483:
-------------------------
Summary: Internal error processing onEndOfDataBlockEvent / count devices returns "Empty set"
Key: IOTDB-3483
URL: https://issues.apache.org/jira/browse/IOTDB-3483
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Attachments: image-2022-06-14-15-20-52-189.png
master_0613_38f3332
3节点3副本
DEVICE_NUMBER=1000
SENSOR_NUMBER=6000
CLIENT_NUMBER=100
GROUP_NUMBER=1
BATCH_SIZE_PER_WRITE=10
LOOP=10000
OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0
3小时20分钟后，datanode error：
2022-06-14 14:17:30,613 [20220614_061627_19086_2.1.0-38] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TApplicationException, message Internal error processing onEndOfDataBlockEvent
2022-06-14 14:17:30,614 [20220614_061657_19097_2.1.0-39] ERROR o.a.i.d.m.e.driver.Driver:195 - Failed to execute fragment instance 20220614_061657_19097_2.1.0
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
2022-06-14 14:17:30,614 [20220614_061627_19086_2.1.0-38] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal error processing onEndOfDataBlockEvent, LocalizedMessage Internal error processing onEndOfDataBlockEvent,
org.apache.thrift.TApplicationException: Internal error processing onEndOfDataBlockEvent
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$receiveBase$20(<generated>)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.receiveBase(<generated>)
at org.apache.iotdb.mpp.rpc.thrift.DataBlockService$Client.recv_onEndOfDataBlockEvent(DataBlockService.java:136)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$recv_onEndOfDataBlockEvent$13(<generated>)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.recv_onEndOfDataBlockEvent(<generated>)
at org.apache.iotdb.mpp.rpc.thrift.DataBlockService$Client.onEndOfDataBlockEvent(DataBlockService.java:123)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.CGLIB$onEndOfDataBlockEvent$16(<generated>)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921$$FastClassByCGLIB$$5fd16e39.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
at org.apache.iotdb.commons.client.sync.SyncDataNodeDataBlockServiceClient$$EnhancerByCGLIB$$51ce1921.onEndOfDataBlockEvent(<generated>)
at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.sendEndOfDataBlockEvent(SinkHandle.java:174)
at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.setNoMoreTsBlocks(SinkHandle.java:198)
at org.apache.iotdb.db.mpp.execution.driver.Driver.closeAndDestroyOperators(Driver.java:338)
at org.apache.iotdb.db.mpp.execution.driver.Driver.destroyIfNecessary(Driver.java:312)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:277)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
集群时间节点的细节信息：
2022-06-14 10:50:55 启动benchmark
2022-06-14 10:58:30,935 bm创建元数据成功
查看3个节点的mlog.bin , ip62的元数据先落盘了一部分 (其余2个节点完全落盘）
!image-2022-06-14-15-20-52-189.png!
50多分钟后ip62的元数据刷盘：
liuzhen@fit-62:/data/liuzhen_test/master_0613_38f3332/datanode$ find ./data -name mlog.bin|xargs ls -lort
-rw-rw-r-- 1 liuzhen 430230000 Jun 14 11:45 ./data/system/schema/root.bm68_dn62.g_0/0/mlog.bin
count devices;
count timeseries ； 结果正确。
出现异常后：
{color:red}count devices 返回Empty set.{color}
还有其它异常：
2022-06-14 14:17:30,614 [20220614_061657_19097_2.1.0-39] ERROR o.a.i.d.m.e.driver.Driver:195 - Failed to execute fragment instance 20220614_061657_19097_2.1.0
java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
2022-06-14 14:17:30,617 [Worker-Thread-3] ERROR o.a.i.d.m.e.s.AbstractDriverThread:61 - Executor Worker-Thread-3 processes failed
java.lang.RuntimeException: Driver was interrupted
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:207)
at org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
at org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:60)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:57)
Suppressed: java.lang.IllegalStateException: Sink handle is aborted.
at org.apache.iotdb.db.mpp.execution.datatransfer.SinkHandle.send(SinkHandle.java:124)
at org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:190)
... 5 common frames omitted
Caused by: java.lang.Exception: Interrupted By
at java.lang.Thread.getStackTrace(Thread.java:1559)
at org.apache.iotdb.db.mpp.execution.driver.Driver$DriverLock.interruptCurrentOwner(Driver.java:444)
at org.apache.iotdb.db.mpp.execution.driver.Driver.close(Driver.java:150)
at org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceExecution.lambda$initialize$0(FragmentInstanceExecution.java:114)
at org.apache.iotdb.db.mpp.execution.StateMachine.fireStateChangedListener(StateMachine.java:241)
at org.apache.iotdb.db.mpp.execution.StateMachine.lambda$fireStateChanged$0(StateMachine.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
复现流程：
1. 192.168.10.62/66/68 72C256G
2. bm 连ip62 （配置见附件）
3. 各节点datanode log见附件。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)