You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/07/28 14:19:54 UTC
[GitHub] [iotdb] mnote opened a new issue, #6827: iotdb 0.13.0 batch write hang
mnote opened a new issue, #6827:
URL: https://github.com/apache/iotdb/issues/6827
**Describe the bug**
A clear and concise description of what the bug is.
iotdb client batch write 1 TimeSeries with about 50000 datapoint over one neight at 740 Query Per Min，but iotdb server hang at 09:xx and Client cannot return.
Client stack trace：
"TaskScheduler_Worker-1" #25 prio=3 os_prio=-1 tid=0x0000000020ebb800 nid=0x4434 waiting on condition [0x0000000024a7d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c97390c8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
at org.apache.iotdb.session.Session.insertByGroup(Session.java:2430)
at org.apache.iotdb.session.Session.insertRecordsWithLeaderCache(Session.java:1513)
at org.apache.iotdb.session.Session.insertRecords(Session.java:1135)
at org.apache.iotdb.session.pool.SessionPool.insertRecords(SessionPool.java:584)
Then I open iotdb cli and execute "flush" command at about 09:40 AM, with server reporting errors:
2022-07-28 09:40:07,446 [pool-4-IoTDB-Flush-3] INFO o.a.i.d.e.s.TsFileProcessor:1202 - Storage group root.sg\0 close the file D:\iotdb\data\data\sequence\root.sg\0\0\1658928352042-20284-0-0.tsfile, TsFile size is 221787816, time consumption of flushing metadata is 154ms
2022-07-28 09:40:07,443 [pool-13-IoTDB-RPC-Client-1351] WARN o.a.i.d.u.ErrorHandlingUtils:146 - Exception occurred while processing non-query.
org.apache.iotdb.db.exception.BatchProcessException: Batch process failed:[TSStatus(code:413, message:org.apache.iotdb.db.exception.StorageEngineException: org.apache.iotdb.db.exception.WriteProcessRejectException: System rejected over 10004ms), TSStatus(code:413, message:org.apache.iotdb.db.exception.StorageEngineException: org.apache.iotdb.db.exception.WriteProcessRejectException: System rejected over 10004ms), ......, TSStatus(code:413, message:org.apache.iotdb.db.exception.StorageEngineException: org.apache.iotdb.db.exception.WriteProcessRejectException: System rejected over 10009ms), TSStatus(code:200), TSStatus(code:200), ......]
at org.apache.iotdb.db.qp.executor.PlanExecutor.insert(PlanExecutor.java:1629)
at org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:282)
at org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:53)
at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:2062)
at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.insertRecords(TSServiceImpl.java:1222)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertRecords.getResult(TSIService.java:3378)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertRecords.getResult(TSIService.java:3358)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
[log_all.txt.zip](https://github.com/apache/iotdb/files/9210479/log_all.txt.zip)
**Desktop (please complete the following information):**
- OS: Windows 2012, JDK 8 x64
[iotdb serve thread dump before execte flush.txt](https://github.com/apache/iotdb/files/9210490/iotdb.serve.thread.dump.before.execte.flush.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin closed issue #6827: iotdb 0.13.0 batch write hang
Posted by GitBox <gi...@apache.org>.
qiaojialin closed issue #6827: iotdb 0.13.0 batch write hang
URL: https://github.com/apache/iotdb/issues/6827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] github-actions[bot] commented on issue #6827: iotdb 0.13.0 batch write hang
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6827:
URL: https://github.com/apache/iotdb/issues/6827#issuecomment-1198347073
Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on issue #6827: iotdb 0.13.0 batch write hang
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on issue #6827:
URL: https://github.com/apache/iotdb/issues/6827#issuecomment-1198796705
Hi, "System rejected over xxx" is due to the system cannot support the write throughput, so it limits the write.
The log shows that the system flushes TsFile frequently, we could increase the avg_series_point_number_threshold to 100000 in iotdb-engine.properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org