You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2020/01/08 08:20:33 UTC

[GitHub] [incubator-iotdb] jutleo opened a new issue #716: IoTDB system load is too large to add timeseries

jutleo opened a new issue #716: IoTDB system load is too large to add timeseries
URL: https://github.com/apache/incubator-iotdb/issues/716
 
 
   15万条数据，分散在不同的设备上，
   root.20200108.1812022495 -> root.20200108.[设备ID] 
   
   出现如下异常
   
   2020-01-08 16:13:21,166 [pool-3-IoTDB-JDBC-Client-thread-2] DEBUG org.apache.iotdb.db.conf.adapter.IoTDBConfigDynamicAdapter:134 - memtableSizeInByte 73717977 is smaller than memTableSizeFloorThreshold 73736192
   2020-01-08 16:13:21,167 [pool-3-IoTDB-JDBC-Client-thread-2] ERROR org.apache.iotdb.db.service.TSServiceImpl:1276 - IoTDB: error occurs when executing statements
   org.apache.iotdb.db.exception.query.QueryProcessException: IoTDB system load is too large to add timeseries
           at org.apache.iotdb.db.qp.executor.QueryProcessExecutor.insertBatch(QueryProcessExecutor.java:301)
           at org.apache.iotdb.db.service.TSServiceImpl.insertBatch(TSServiceImpl.java:1257)
           at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertBatch.getResult(TSIService.java:1705)
           at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertBatch.getResult(TSIService.java:1690)
           at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
           at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
           at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   2020-01-08 16:13:21,167 [pool-3-IoTDB-JDBC-Client-thread-2] DEBUG org.apache.iotdb.db.conf.adapter.IoTDBConfigDynamicAdapter:134 - memtableSizeInByte 73717977 is smaller than memTableSizeFloorThreshold 73736192
   2020-01-08 16:13:21,168 [pool-3-IoTDB-JDBC-Client-thread-2] ERROR org.apache.iotdb.db.service.TSServiceImpl:1276 - IoTDB: error occurs when executing statements

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] qiaojialin commented on issue #716: IoTDB system load is too large to add timeseries

Posted by GitBox <gi...@apache.org>.
qiaojialin commented on issue #716: IoTDB system load is too large to add timeseries
URL: https://github.com/apache/incubator-iotdb/issues/716#issuecomment-571990818
 
 
   这个是内存不足导致的，为了控制内存，IoTDB 现在采用了比较悲观的内存估计策略，大概1万时间序列需要1G内存。如果内存不足会拒绝新的时间序列的创建。如果不想用这个自动检查，可以把 iotdb-engine.properties 里的 enable_parameter_adapter=true 改成 false。
   
   如果只有顺序数据，那一个存储组需要 2倍的 memtable_size_threshold 的内存大小。如果有乱序，需要4倍的。可以大概估计一下内存大小，此外还有其他的一些元数据在占用内存，可以再多给几个G。

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [iotdb] HTHou closed issue #716: IoTDB system load is too large to add timeseries

Posted by GitBox <gi...@apache.org>.
HTHou closed issue #716:
URL: https://github.com/apache/iotdb/issues/716


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] HTHou closed issue #716: IoTDB system load is too large to add timeseries

Posted by GitBox <gi...@apache.org>.
HTHou closed issue #716:
URL: https://github.com/apache/iotdb/issues/716


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] jixuan1989 commented on issue #716: IoTDB system load is too large to add timeseries

Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on issue #716: IoTDB system load is too large to add timeseries
URL: https://github.com/apache/incubator-iotdb/issues/716#issuecomment-572550347
 
 
   1. 按天设计存储组不好。只有8G内存的话，可以考虑尝试10~50个存储组。 
   2. 50万设备，如果这些设备都是5s来一条数据的话，并且有乱序数据，基本上每个设备不到4KB就会形成一个磁盘上的数据块。感觉内存稍微小了些。如果有性能结果，可以分享下。

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-iotdb] jutleo commented on issue #716: IoTDB system load is too large to add timeseries

Posted by GitBox <gi...@apache.org>.
jutleo commented on issue #716: IoTDB system load is too large to add timeseries
URL: https://github.com/apache/incubator-iotdb/issues/716#issuecomment-572320667
 
 
   @qiaojialin 您好，我的实际情况是这样的，50万个设备，每5秒一条数据，不完全是时间序列（有可能有补报的数据上来），存储组按天设计root.20200108， 序列为root.20200108.设备ID， 每天约400GB的数据，分散在50万个序列上，也就是为了后期使用方便，我的序列默认就是一个设备一个序列。
   
   对于我的需求场景为典型的key- list数据结构，因为初次接触IotDB，不知道这样的结构设计是不是最优，达到的效果为8GB内存+机械硬盘 单节点满足需求。
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services