You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "Steve Yurong Su (Jira)" <ji...@apache.org> on 2023/07/24 03:57:00 UTC
[jira] [Created] (IOTDB-6081) Pipe: use HybridExtractor instead of LogExtractor when realtime mode is set to log to avoid OOM under heavy insertion load
Steve Yurong Su created IOTDB-6081:
--------------------------------------
Summary: Pipe: use HybridExtractor instead of LogExtractor when realtime mode is set to log to avoid OOM under heavy insertion load
Key: IOTDB-6081
URL: https://issues.apache.org/jira/browse/IOTDB-6081
Project: Apache IoTDB
Issue Type: Bug
Reporter: Steve Yurong Su
Attachments: image-2023-07-24-11-54-50-433.png
Scenarios:
1. source iotdb: 3C3D 2 replica, target iotdb: 1C1D
2. create and start 10 realtime pipes with log mode enabled
{code:java}
create pipe pipe_db_1 with extractor ('extractor'='iotdb-extractor', 'extractor.pattern'='root.test.g_0','extractor.realtime.enable'='true','extractor.realtime.mode'='log') with connector ('connector'='iotdb-thrift-connector', 'connector.ip'='172.20.70.15', 'connector.port'='6667');
{code}
4. start benchmark: device: 100 sensor: 10 database: 1 aligned timeseries
!image-2023-07-24-11-54-50-433.png|width=863,height=415!
Fixes introcuded:
* When user specifies 'extractor.realtime.mode'='log', we use hybrid extractors which can automatically degrade to file mode when load is too heavy.
* Add a parameter: 'extractor.realtime.mode'='forced-log', the pipe will always use log mode and will never degrade to file mode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)