You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "changxue (Jira)" <ji...@apache.org> on 2023/02/17 06:42:00 UTC
[jira] [Commented] (IOTDB-5550) [memory]OOM occurs when insert less than 1M data with benchmark and default configurations of IoTDB
[ https://issues.apache.org/jira/browse/IOTDB-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690175#comment-17690175 ]
changxue commented on IOTDB-5550:
---------------------------------
datanode日志中有很多OOM，写入数据量并不大：10设备400sensor 1客户端 1次写入100行，落磁盘不到1M。
metric监控显示memory 并未使用多少，gc统计为0，OOM的thread并未结束（监控的数量未变）写入时间十几分钟，然后使用cli连接不上。
> [memory]OOM occurs when insert less than 1M data with benchmark and default configurations of IoTDB
> ---------------------------------------------------------------------------------------------------
>
> Key: IOTDB-5550
> URL: https://issues.apache.org/jira/browse/IOTDB-5550
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Engine
> Reporter: changxue
> Assignee: Jinrui Zhang
> Priority: Major
> Attachments: IOTDB-5550.tar.gz, image-2023-02-17-14-28-57-874.png
>
>
> [memory]OOM occurs when insert less than 1M data with benchmark and default configurations of IoTDB
> build:
> master 1a69b40 1.1.0-SNAPSHOT
> reproduction:
> 1. start 1C1D with the configurations of attachment
> 2. run benchmarks to insert data into IoTDB
> datanode log:
> {code}
> 2023-02-17 10:56:32,311 [pool-24-IoTDB-DataNodeInternalRPC-Processor-1] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2023-02-17 10:56:21,422 [pool-19-IoTDB-ratis-bg-disk-guardian-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:191 - Run thread failed
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2023-02-17 10:56:50,734 [pool-23-IoTDB-MPPCoordinatorScheduled-9] WARN o.a.i.d.m.p.s.SimpleQueryTerminator:108 - cancel query 20230217_025125_00644_1 on node TEndPoint(ip:iotdb-44, port:10730) failed.
> org.apache.thrift.TException: Error in calling method cancelQuery, because: GC overhead limit exceeded
> at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:57)
> at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$c055cf81.cancelQuery(<generated>)
> at org.apache.iotdb.db.mpp.plan.scheduler.SimpleQueryTerminator.syncTerminate(SimpleQueryTerminator.java:102)
> at org.apache.iotdb.commons.concurrent.WrappedCallable$1.callMayThrow(WrappedCallable.java:46)
> at org.apache.iotdb.commons.concurrent.WrappedCallable.call(WrappedCallable.java:31)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> {code}
> disk taken:
> {code}
> 4.1M ./datanode/consensus
> 28K ./datanode/data
> 40M ./datanode/wal
> 64K ./datanode/system
> 45M ./datanode
> 4.1M ./confignode/consensus
> 24K ./confignode/system
> 4.1M ./confignode
> 49M .
> {code}
> metric monitor:
> !image-2023-02-17-14-28-57-874.png|thumbnail|width=800!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)