You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "Jinrui Zhang (Jira)" <ji...@apache.org> on 2022/11/24 05:16:00 UTC
[jira] [Commented] (IOTDB-5015) [write]when writing for about 6 hours to only 1 sensor, the writing stopped with error: too many requests need to process
[ https://issues.apache.org/jira/browse/IOTDB-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638098#comment-17638098 ]
Jinrui Zhang commented on IOTDB-5015:
-------------------------------------
This issue may caused by the same reason in this issue https://issues.apache.org/jira/browse/IOTDB-5019.
> [write]when writing for about 6 hours to only 1 sensor, the writing stopped with error: too many requests need to process
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: IOTDB-5015
> URL: https://issues.apache.org/jira/browse/IOTDB-5015
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Server
> Affects Versions: 0.14.0-SNAPSHOT
> Reporter: changxue
> Assignee: Jinrui Zhang
> Priority: Major
> Attachments: allnodes-logs.tar.gz, config.properties, screenshot-1.png
>
>
> [write]when writing for about 6 hours to only 1 sensor, the writing stopped with error: too many requests need to process
> environment:
> 3C3D cluster, 2 replicas
> |RegionId|Type|Status|Database|SeriesSlotId|TimeSlotId|DataNodeId|Host|RpcPort|Role|
> |10|SchemaRegion|Running|root.aggr.g_0|1|0|1|172.20.70.44|6667|Follower|
> |10|SchemaRegion|Running|root.aggr.g_0|1|0|5|172.20.70.46|6667|Leader|
> |11|DataRegion|Running|root.aggr.g_0|1|10|1|172.20.70.44|6667|Follower|
> |11|DataRegion|Running|root.aggr.g_0|1|10|5|172.20.70.46|6667|Leader|
> reproduction:
> 1. start the cluster successfully
> 2. start the 0.13 benchmark at about 20:23 Nov.21, the benchmark configuration see the attachment of config.properties
> 3. errors occurred at about 3:20 Nov.22 and writing can't be continued.
> 4. 6:30 Nov.22, I start benchmark again, but can't write to iotdb successfully
> 5. run stop-datanode.sh and start-datanode.sh at the bad node of 46
> 6. start benchmark again, now it can write successfully
> 172.20.70.46 datanode:
> {code:sh}
> 2022-11-22 03:20:23,586 [pool-8-IoTDB-WAL-Delete-1] INFO o.a.i.d.w.n.WALNode$DeleteOutdatedFileTask:367 - WAL node-root.aggr.g_0-11 flushes memTable-4510 to TsFile /data/iotdb/apache-iotdb-0.14.0-SNAPSHOT-all-bin/data/datanode/data/sequence/root.aggr.g_0/11/52/1669036247165-4504-0-0.tsfile, memTable size is 1531600.
> 2022-11-22 03:20:42,915 [pool-25-IoTDB-ClientRPC-Processor-2$20221121_192013_20413_5.1.0] ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:234 - write locally failed. TSStatus: TSStatus(code:606, message:Reject write because there are too many requests need to process), message: Reject write because there are too many requests need to process
> 2022-11-22 03:20:42,978 [pool-25-IoTDB-ClientRPC-Processor-2$20221121_192043_20414_5.1.0] INFO o.a.i.c.m.MultiLeaderServerImpl:178 - [Throttle Down] index:380448, safeIndex:380448
> 2022-11-22 03:20:43,594 [pool-8-IoTDB-WAL-Delete-1] INFO o.a.i.d.w.n.WALNode$DeleteOutdatedFileTask:242 - Effective information ratio 1.8484028968067935E-4 (active memTables cost is 13563200, flushed memTables cost is 73364378500) of wal node-root.aggr.g_0-11 is below wal min effective info ratio 0.1, some memTables will be snapshot or flushed.
> {code}
> 3:20 monitor:
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)