You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by ha...@apache.org on 2024/11/11 02:31:31 UTC

(iotdb) branch rc/1.3.3 updated (461b55ee48e -> eb66c590735)

This is an automated email from the ASF dual-hosted git repository.

haonan pushed a change to branch rc/1.3.3
in repository https://gitbox.apache.org/repos/asf/iotdb.git


    omit 461b55ee48e [To rc/1.3.3] Print Dataset is closed error msg if user continues to fetch from it
    omit 670424ac1e4 [to rc/1.3.3] Fix DeleteDatabaseProcedure (#13927) #13973
    omit d218f1edffe Pipe: Optimize lock in PipeTsFileResourceManager using segment lock strategy (#13915) (#13994)
    omit 55c193115cd Pipe: Add fsync option for file receiver to avoid data region unexpected behavior after system restart (#13984) (#13993)
    omit 81a26d2a2a6 [To rc/1.3.3] Combine CSV SQL TSfile data import/export script (#13930)
    omit bd6e6751e8c Pipe: Avoid the file reading when the memory is not enough for tablet parsing (#13954) (#13988)
    omit d88c8ee8026 AINode: Fix built-model inference error (#13986)
    omit 1a52cff1a32 [To rc/1.3.3] Correct the memory estimation for SeriesAggregationScanOperator
    omit 7d6bb733d89 Subscription: intro memory control when constructing file payload to avoid OOM (#13937) (#13977)
    omit e7ad615a115 Subscription: implement resource-aware and request-aware dynamic prefetching logic (#13956) (#13976)
    omit 784fdbe2ce6 Subscription: avoid endless loop when fetching subsequent responses by poll timer (#13968) (#13975)
    omit 1fc0d65c41a [to rc/1.3.3] Fix negative mods file metric (#13970)
    omit 2668c9854cc Pipe: Disable sloppyPattern in PipeHistoricalDataRegionTsFileAndDeletionExtractor (#13934) (#13957)
    omit 2d2be99dcfe [To rc/1.3.3]Fix IoTV2 metric NPE due to concurrency #13952
    omit ce4dea1a75f [to rc/1.3.3] Reduce deserialized file num in compaction validation (#13944)
    omit a19fda5010f Pipe: validate all pipe parameters before alter (#13932) (#13958)
    omit 6406f238047 [to rc/1.3.3] Modify message of unsupported DataNode->ConfigNode RPC (CP 13945) #13948
    omit 6ea5ec2af59 [region migration] Retry WRITE_PROCESS_REJECT status code #13921 (#13924)
    omit 679198151d8 fix negative file metric items (#13943)
    omit 742b98a43bd Fix the case that there is nonexist measurement in having clause
    omit ba6f77d2dd9 [To 1.3] Batch update inserted point number metric (#13939)
    omit fd0940ab670 Pipe: Support specifing receiver's username and password when syncing data between clusters (#13933)
    omit f8adc22119d Subscription: implement subscription event optimistic transmission strategy to reduce peak memory usage (#13763) (#13928)
    omit dcbb2d3532d Subscription: improve subscription meta management with sub-procedures strong exception handling semantics (#13698) (#13923)
    omit ec640661f6a [to rc/1.3.3] Split CnToDnRequestType to sync and async & Add check for adding new request type (see #13660) (#13784)
    omit 62276a551f3 Pipe/Subscription: avoid executing `rollbackFromValidateTask` or `rollbackFromValidate` multiple times in retry with rollback scenarios (#13825) (#13916)
    omit ad1e7e1bc55 [to rc/1.3.3] Fix compaction cached time chunk flip twice (#13904)
    omit 64bf6d82a10 [region migration] Enable write retry for ratis local write (#13901) (#13913)
    omit cc2764f7c71 Active Load: Refactoring the code for handling exception information after loading failure (#13880) (#13897)
    omit d937c997879 Pipe: Avoid constructing unnecessary pipe subtasks if db and pattern dismatch (#13873) (#13892)
    omit 7837d4988ac fix DirectoryNotEmpty (#13710)
    omit 0f63b4674e5 Fix TimePartitionUtils Overflow (#13881) (#13885)
    omit cb0d2a51650 Pipe: fix calculations of PipeDataNodeRemainingEventAndTimeOperator (#13876) (#13889)
     new d0d283305e2 [to rc/1.3.3] Fix compaction cached time chunk flip twice (#13904)
     new 3966c6ac05b Pipe: fix calculations of PipeDataNodeRemainingEventAndTimeOperator (#13876) (#13889)
     new 4375409c5e1 Pipe: Fixed the bug that the PipeDataNodeRemainingEventAndTimeMetrics may generate NullPointerException (#14015) (#14024)
     new bc9ef048d74 Pipe: GC device2Measurements in PipeRealtimeEvent in time to avoid OOM when insertion is in high load (#14018) (#14023)
     new 7984b196603 [To dev/1.3] Fix query resource clear async thread stuck bug
     new eb66c590735 Change typo in warn log for disk_space_warning_threshold (#14042)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (461b55ee48e)
            \
             N -- N -- N   refs/heads/rc/1.3.3 (eb66c590735)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/iotdb/ainode/it/AINodeBasicIT.java  |   13 -
 .../db/it/alignbydevice/IoTDBAlignByDeviceIT.java  |   45 -
 .../pipe/it/manual/IoTDBPipePermissionIT.java      |  164 ---
 .../apache/iotdb/tools/it/ExportDataTestIT.java    |   12 +-
 .../apache/iotdb/tools/it/ImportDataTestIT.java    |   16 +-
 .../api/customizer/parameter/PipeParameters.java   |    1 -
 .../src/assembly/resources/tools/export-data.bat   |    6 +-
 .../src/assembly/resources/tools/export-data.sh    |    6 +-
 .../src/assembly/resources/tools/import-data.bat   |    6 +-
 .../src/assembly/resources/tools/import-data.sh    |    6 +-
 .../apache/iotdb/tool/data/AbstractDataTool.java   | 1106 +-------------------
 .../apache/iotdb/tool/data/AsyncImportData.java    |  160 ---
 .../org/apache/iotdb/tool/data/ExportData.java     |  397 ++-----
 .../org/apache/iotdb/tool/data/ImportData.java     | 1100 ++++++++++++-------
 .../apache/iotdb/tool/data/ImportDataScanTool.java |   63 --
 .../iotdb/tool/tsfile/AbstractTsFileTool.java      |    3 +-
 .../org/apache/iotdb/tool/tsfile/ExportTsFile.java |  105 +-
 .../org/apache/iotdb/tool/tsfile/ImportTsFile.java |   70 +-
 .../apache/iotdb/tool/tsfile/ImportTsFileBase.java |   10 +-
 .../iotdb/tool/tsfile/ImportTsFileRemotely.java    |   25 +-
 .../java/org/apache/iotdb/rpc/IoTDBRpcDataSet.java |    3 -
 .../iotdb/rpc/subscription/config/TopicConfig.java |    7 +-
 .../payload/poll/SubscriptionPollResponse.java     |   20 +-
 .../consumer/SubscriptionConsumer.java             |   56 +-
 .../iotdb/ainode/model/built_in_model_factory.py   |    2 +-
 ...syncRequestType.java => CnToDnRequestType.java} |   24 +-
 .../CnToDnInternalServiceAsyncRequestManager.java  |  144 ++-
 .../handlers/DataNodeAsyncRequestContext.java      |   12 +-
 .../rpc/CheckTimeSeriesExistenceRPCHandler.java    |    4 +-
 .../rpc/CountPathsUsingTemplateRPCHandler.java     |    4 +-
 .../rpc/DataNodeAsyncRequestRPCHandler.java        |   10 +-
 .../handlers/rpc/DataNodeTSStatusRPCHandler.java   |    4 +-
 .../rpc/FetchSchemaBlackListRPCHandler.java        |    4 +-
 .../handlers/rpc/PipeHeartbeatRPCHandler.java      |    4 +-
 .../async/handlers/rpc/PipePushMetaRPCHandler.java |    4 +-
 .../async/handlers/rpc/SchemaUpdateRPCHandler.java |    4 +-
 .../rpc/SubmitTestConnectionTaskRPCHandler.java    |    4 +-
 .../handlers/rpc/TransferLeaderRPCHandler.java     |    4 +-
 .../CheckSchemaRegionUsingTemplateRPCHandler.java  |    4 +-
 .../ConsumerGroupPushMetaRPCHandler.java           |    4 +-
 .../rpc/subscription/TopicPushMetaRPCHandler.java  |    4 +-
 .../client/sync/CnToDnSyncRequestType.java         |   49 -
 .../client/sync/SyncDataNodeClientPool.java        |  136 +--
 .../confignode/conf/ConfigNodeStartupCheck.java    |    9 -
 .../iotdb/confignode/manager/ClusterManager.java   |    9 +-
 .../confignode/manager/ClusterQuotaManager.java    |    6 +-
 .../iotdb/confignode/manager/TriggerManager.java   |    4 +-
 .../iotdb/confignode/manager/UDFManager.java       |    6 +-
 .../manager/load/balancer/RouteBalancer.java       |    8 +-
 .../iotdb/confignode/manager/node/NodeManager.java |   29 +-
 .../manager/partition/PartitionManager.java        |    8 +-
 .../client/IoTDBConfigNodeSyncClientManager.java   |    4 -
 .../protocol/IoTDBConfigRegionAirGapConnector.java |    2 -
 .../protocol/IoTDBConfigRegionConnector.java       |    4 -
 .../runtime/heartbeat/PipeHeartbeatScheduler.java  |    4 +-
 .../metric/PipeConfigNodeRemainingTimeMetrics.java |    9 +-
 .../PipeConfigNodeRemainingTimeOperator.java       |    5 +-
 .../manager/schema/ClusterSchemaManager.java       |    4 +-
 .../subscription/SubscriptionMetaSyncer.java       |   38 +-
 .../persistence/subscription/SubscriptionInfo.java |   15 +-
 .../procedure/env/ConfigNodeProcedureEnv.java      |   57 +-
 .../procedure/env/RegionMaintainHandler.java       |   22 +-
 .../impl/pipe/AbstractOperatePipeProcedureV2.java  |   18 +-
 .../impl/pipe/runtime/PipeMetaSyncProcedure.java   |    3 +-
 .../impl/pipe/task/AlterPipeProcedureV2.java       |    2 +-
 .../impl/schema/AlterLogicalViewProcedure.java     |    8 +-
 .../impl/schema/DataNodeRegionTaskExecutor.java    |    8 +-
 .../impl/schema/DeactivateTemplateProcedure.java   |   16 +-
 .../impl/schema/DeleteDatabaseProcedure.java       |   28 +-
 .../impl/schema/DeleteLogicalViewProcedure.java    |   12 +-
 .../impl/schema/DeleteTimeSeriesProcedure.java     |   16 +-
 .../procedure/impl/schema/SchemaUtils.java         |    6 +-
 .../procedure/impl/schema/SetTTLProcedure.java     |    4 +-
 .../impl/schema/SetTemplateProcedure.java          |   12 +-
 .../impl/schema/UnsetTemplateProcedure.java        |   10 +-
 .../AbstractOperateSubscriptionProcedure.java      |   42 +-
 .../consumer/AlterConsumerGroupProcedure.java      |   60 +-
 .../runtime/ConsumerGroupMetaSyncProcedure.java    |    6 +-
 .../subscription/CreateSubscriptionProcedure.java  |  260 +++--
 .../subscription/DropSubscriptionProcedure.java    |  147 ++-
 .../subscription/topic/AlterTopicProcedure.java    |   44 +-
 .../subscription/topic/CreateTopicProcedure.java   |   33 +-
 .../subscription/topic/DropTopicProcedure.java     |    5 +-
 .../topic/runtime/TopicMetaSyncProcedure.java      |    6 +-
 .../impl/sync/AuthOperationProcedure.java          |    4 +-
 .../CreateSubscriptionProcedureTest.java           |    9 +
 .../DropSubscriptionProcedureTest.java             |    8 +
 .../iotdb/consensus/ratis/RatisConsensus.java      |    4 -
 .../db/pipe/agent/task/PipeDataNodeTaskAgent.java  |    8 +-
 .../agent/task/builder/PipeDataNodeBuilder.java    |    6 +-
 .../client/IoTDBDataNodeAsyncClientManager.java    |   28 +-
 .../client/IoTDBDataNodeSyncClientManager.java     |    4 -
 .../airgap/IoTDBDataNodeAirGapConnector.java       |    2 -
 .../protocol/legacy/IoTDBLegacyPipeConnector.java  |    8 +-
 .../connector/protocol/opcua/OpcUaConnector.java   |    8 +-
 .../async/IoTDBDataRegionAsyncConnector.java       |    2 -
 .../thrift/sync/IoTDBDataNodeSyncConnector.java    |    4 -
 .../event/common/heartbeat/PipeHeartbeatEvent.java |    4 +-
 .../tablet/PipeInsertNodeTabletInsertionEvent.java |    4 +-
 .../common/tablet/PipeRawTabletInsertionEvent.java |    4 +-
 .../common/tsfile/PipeTsFileInsertionEvent.java    |   34 +-
 .../db/pipe/event/realtime/PipeRealtimeEvent.java  |    4 +-
 .../dataregion/DataRegionListeningFilter.java      |   23 +-
 .../PipeHistoricalDataRegionTsFileExtractor.java   |    4 +
 .../PipeDataNodeRemainingEventAndTimeMetrics.java  |   45 +-
 .../PipeDataNodeRemainingEventAndTimeOperator.java |    5 +-
 .../protocol/airgap/IoTDBAirGapReceiver.java       |    8 -
 .../pipeconsensus/PipeConsensusReceiver.java       |   22 +-
 .../protocol/thrift/IoTDBDataNodeReceiver.java     |   50 +-
 .../pipe/resource/memory/PipeFixedMemoryBlock.java |   64 --
 .../pipe/resource/memory/PipeMemoryBlockType.java  |   26 -
 .../db/pipe/resource/memory/PipeMemoryManager.java |  118 +--
 .../resource/memory/PipeTabletMemoryBlock.java     |   39 +-
 .../resource/memory/PipeTsFileMemoryBlock.java     |   27 -
 .../resource/tsfile/PipeTsFileResourceManager.java |  123 +--
 .../tsfile/PipeTsFileResourceSegmentLock.java      |  109 --
 .../iotdb/db/protocol/client/ConfigNodeClient.java |   15 +-
 .../impl/DataNodeInternalRPCServiceImpl.java       |    2 +-
 .../db/queryengine/execution/driver/Driver.java    |   31 +-
 .../AbstractSeriesAggregationScanOperator.java     |    5 +-
 .../AlignedSeriesAggregationScanOperator.java      |    5 -
 .../source/SeriesAggregationScanOperator.java      |    3 -
 .../queryengine/plan/analyze/AnalyzeVisitor.java   |    9 +-
 .../plan/analyze/ExpressionAnalyzer.java           |   13 -
 .../config/executor/ClusterConfigTaskExecutor.java |  110 +-
 .../visitor/ColumnTransformerVisitor.java          |   15 +-
 .../visitor/ExistUnknownTypeInExpression.java      |   60 --
 .../ConcatDeviceAndBindSchemaForHavingVisitor.java |   77 --
 .../db/storageengine/dataregion/DataRegion.java    |   16 -
 .../execute/task/AbstractCompactionTask.java       |  108 +-
 .../execute/task/CrossSpaceCompactionTask.java     |   11 +-
 .../execute/task/InnerSpaceCompactionTask.java     |   10 +-
 .../task/InsertionCrossSpaceCompactionTask.java    |    6 +-
 .../compaction/execute/utils/CompactionUtils.java  |   26 +-
 .../dataregion/memtable/AbstractMemTable.java      |  140 ++-
 .../dataregion/memtable/IMemTable.java             |   11 +-
 .../dataregion/memtable/TsFileProcessor.java       |   20 +-
 .../wal/recover/file/TsFilePlanRedoer.java         |   22 +-
 .../active/ActiveLoadFailedMessageHandler.java     |  116 --
 .../load/active/ActiveLoadTsFileLoader.java        |   27 +-
 .../agent/SubscriptionBrokerAgent.java             |   90 +-
 .../agent/SubscriptionConsumerAgent.java           |   27 +-
 .../db/subscription/broker/SubscriptionBroker.java |   44 +-
 .../broker/SubscriptionPrefetchingQueue.java       |  173 ++-
 .../broker/SubscriptionPrefetchingQueueStates.java |  128 ---
 .../broker/SubscriptionPrefetchingTabletQueue.java |    7 +-
 .../broker/SubscriptionPrefetchingTsFileQueue.java |    6 +-
 .../db/subscription/event/SubscriptionEvent.java   |  297 ++++--
 ...ache.java => SubscriptionEventBinaryCache.java} |   43 +-
 .../event/batch/SubscriptionPipeEventBatch.java    |   84 +-
 .../batch/SubscriptionPipeTabletEventBatch.java    |  157 ++-
 .../batch/SubscriptionPipeTsFileEventBatch.java    |   81 +-
 .../cache/CachedSubscriptionPollResponse.java      |   93 --
 .../event/pipe/SubscriptionPipeEmptyEvent.java     |    7 +
 .../event/pipe/SubscriptionPipeEvents.java         |    7 +
 .../pipe/SubscriptionPipeTabletBatchEvents.java    |    7 +
 .../pipe/SubscriptionPipeTsFileBatchEvents.java    |   14 +-
 .../pipe/SubscriptionPipeTsFilePlainEvent.java     |    7 +
 .../SubscriptionEventExtendableResponse.java       |  151 ---
 .../event/response/SubscriptionEventResponse.java  |   52 -
 .../response/SubscriptionEventSingleResponse.java  |  122 ---
 .../response/SubscriptionEventTabletResponse.java  |  166 ---
 .../response/SubscriptionEventTsFileResponse.java  |  188 ----
 .../receiver/SubscriptionReceiverV1.java           |   14 +-
 .../task/subtask/SubscriptionConnectorSubtask.java |    4 +-
 .../execution/operator/OperatorMemoryTest.java     |   10 +-
 .../cross/InsertionCrossSpaceCompactionTest.java   |    8 +-
 .../db/subscription/SubscriptionStatesTest.java    |    5 +
 .../src/test/resources/iotdb-system.properties     |    5 -
 .../client/request/AsyncRequestManager.java        |    6 -
 .../apache/iotdb/commons/conf/CommonConfig.java    |   53 -
 .../iotdb/commons/conf/CommonDescriptor.java       |   28 -
 .../commons/exception/IoTDBRuntimeException.java   |   66 --
 .../exception/UncheckedStartupException.java       |   41 -
 .../iotdb/commons/partition/SchemaPartition.java   |    2 -
 .../iotdb/commons/pipe/config/PipeConfig.java      |   19 -
 .../config/constant/PipeConnectorConstant.java     |    2 -
 .../pipe/connector/client/IoTDBClientManager.java  |   18 +-
 .../connector/client/IoTDBSyncClientManager.java   |   19 +-
 .../common/PipeTransferHandshakeConstant.java      |    2 -
 .../pipe/connector/protocol/IoTDBConnector.java    |   24 -
 .../connector/protocol/IoTDBSslSyncConnector.java  |    4 -
 .../datastructure/pattern/IoTDBPipePattern.java    |    9 -
 .../pipe/datastructure/pattern/PipePattern.java    |    7 -
 .../datastructure/pattern/PrefixPipePattern.java   |    9 -
 .../commons/pipe/metric/PipeRemainingOperator.java |   12 +-
 .../commons/pipe/receiver/IoTDBFileReceiver.java   |   42 +-
 .../subscription/config/SubscriptionConfig.java    |   22 +-
 .../meta/consumer/ConsumerGroupMeta.java           |    4 -
 .../meta/consumer/ConsumerGroupMetaKeeper.java     |   25 -
 .../commons/subscription/meta/topic/TopicMeta.java |   19 +-
 .../apache/iotdb/commons/utils/StatusUtils.java    |    1 -
 .../iotdb/commons/utils/TimePartitionUtils.java    |  104 +-
 .../commons/utils/TimePartitionUtilsTest.java      |   16 -
 .../apache/iotdb/library/dprofile/UDAFMedian.java  |   14 +-
 195 files changed, 2788 insertions(+), 6257 deletions(-)
 delete mode 100644 integration-test/src/test/java/org/apache/iotdb/pipe/it/manual/IoTDBPipePermissionIT.java
 delete mode 100644 iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/AsyncImportData.java
 delete mode 100644 iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/ImportDataScanTool.java
 rename iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/client/{async/CnToDnAsyncRequestType.java => CnToDnRequestType.java} (87%)
 delete mode 100644 iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/client/sync/CnToDnSyncRequestType.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeFixedMemoryBlock.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeMemoryBlockType.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeTsFileMemoryBlock.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceSegmentLock.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/expression/visitor/ExistUnknownTypeInExpression.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/expression/visitor/cartesian/ConcatDeviceAndBindSchemaForHavingVisitor.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/active/ActiveLoadFailedMessageHandler.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/SubscriptionPrefetchingQueueStates.java
 rename iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/{cache/SubscriptionPollResponseCache.java => SubscriptionEventBinaryCache.java} (78%)
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/cache/CachedSubscriptionPollResponse.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/response/SubscriptionEventExtendableResponse.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/response/SubscriptionEventResponse.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/response/SubscriptionEventSingleResponse.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/response/SubscriptionEventTabletResponse.java
 delete mode 100644 iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/response/SubscriptionEventTsFileResponse.java
 delete mode 100644 iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/exception/IoTDBRuntimeException.java
 delete mode 100644 iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/exception/UncheckedStartupException.java


(iotdb) 02/06: Pipe: fix calculations of PipeDataNodeRemainingEventAndTimeOperator (#13876) (#13889)

Posted by ha...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

haonan pushed a commit to branch rc/1.3.3
in repository https://gitbox.apache.org/repos/asf/iotdb.git

commit 3966c6ac05b5fa4ff2d999cbe70f71416b7e0036
Author: V_Galaxy <vg...@apache.org>
AuthorDate: Wed Oct 23 19:50:20 2024 +0800

    Pipe: fix calculations of PipeDataNodeRemainingEventAndTimeOperator (#13876) (#13889)
---
 .../common/tablet/PipeInsertNodeTabletInsertionEvent.java   |  4 ++--
 .../event/common/tablet/PipeRawTabletInsertionEvent.java    |  4 ++--
 .../metric/PipeDataNodeRemainingEventAndTimeMetrics.java    |  8 ++++----
 .../metric/PipeDataNodeRemainingEventAndTimeOperator.java   | 13 +++++++------
 4 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeInsertNodeTabletInsertionEvent.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeInsertNodeTabletInsertionEvent.java
index bd01f26b655..5932c6b544b 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeInsertNodeTabletInsertionEvent.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeInsertNodeTabletInsertionEvent.java
@@ -136,7 +136,7 @@ public class PipeInsertNodeTabletInsertionEvent extends EnrichedEvent
       PipeDataNodeResourceManager.wal().pin(walEntryHandler);
       if (Objects.nonNull(pipeName)) {
         PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-            .increaseInsertionEventCount(pipeName + "_" + creationTime);
+            .increaseTabletEventCount(pipeName + "_" + creationTime);
       }
       return true;
     } catch (final Exception e) {
@@ -169,7 +169,7 @@ public class PipeInsertNodeTabletInsertionEvent extends EnrichedEvent
     } finally {
       if (Objects.nonNull(pipeName)) {
         PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-            .decreaseInsertionEventCount(pipeName + "_" + creationTime);
+            .decreaseTabletEventCount(pipeName + "_" + creationTime);
       }
     }
   }
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeRawTabletInsertionEvent.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeRawTabletInsertionEvent.java
index a5795ede49c..b89c11b1f39 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeRawTabletInsertionEvent.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeRawTabletInsertionEvent.java
@@ -118,7 +118,7 @@ public class PipeRawTabletInsertionEvent extends EnrichedEvent implements Tablet
                 PipeMemoryWeightUtil.calculateTabletSizeInBytes(tablet));
     if (Objects.nonNull(pipeName)) {
       PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-          .increaseInsertionEventCount(pipeName + "_" + creationTime);
+          .increaseTabletEventCount(pipeName + "_" + creationTime);
     }
     return true;
   }
@@ -127,7 +127,7 @@ public class PipeRawTabletInsertionEvent extends EnrichedEvent implements Tablet
   public boolean internallyDecreaseResourceReferenceCount(final String holderMessage) {
     if (Objects.nonNull(pipeName)) {
       PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-          .decreaseInsertionEventCount(pipeName + "_" + creationTime);
+          .decreaseTabletEventCount(pipeName + "_" + creationTime);
     }
     allocatedMemoryBlock.close();
 
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeMetrics.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeMetrics.java
index a8765b3b61d..25c2ede2407 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeMetrics.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeMetrics.java
@@ -129,16 +129,16 @@ public class PipeDataNodeRemainingEventAndTimeMetrics implements IMetricSet {
     }
   }
 
-  public void increaseInsertionEventCount(final String pipeID) {
+  public void increaseTabletEventCount(final String pipeID) {
     remainingEventAndTimeOperatorMap
         .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
-        .increaseInsertionEventCount();
+        .increaseTabletEventCount();
   }
 
-  public void decreaseInsertionEventCount(final String pipeID) {
+  public void decreaseTabletEventCount(final String pipeID) {
     remainingEventAndTimeOperatorMap
         .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
-        .decreaseInsertionEventCount();
+        .decreaseTabletEventCount();
   }
 
   public void increaseTsFileEventCount(final String pipeID) {
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeOperator.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeOperator.java
index c0072215146..bee0e6975b4 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeOperator.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeOperator.java
@@ -44,7 +44,7 @@ class PipeDataNodeRemainingEventAndTimeOperator extends PipeRemainingOperator {
   private final Set<IoTDBSchemaRegionExtractor> schemaRegionExtractors =
       Collections.newSetFromMap(new ConcurrentHashMap<>());
 
-  private final AtomicInteger insertionEventCount = new AtomicInteger(0);
+  private final AtomicInteger tabletEventCount = new AtomicInteger(0);
   private final AtomicInteger tsfileEventCount = new AtomicInteger(0);
   private final AtomicInteger heartbeatEventCount = new AtomicInteger(0);
 
@@ -58,12 +58,12 @@ class PipeDataNodeRemainingEventAndTimeOperator extends PipeRemainingOperator {
 
   //////////////////////////// Remaining event & time calculation ////////////////////////////
 
-  void increaseInsertionEventCount() {
-    tsfileEventCount.incrementAndGet();
+  void increaseTabletEventCount() {
+    tabletEventCount.incrementAndGet();
   }
 
-  void decreaseInsertionEventCount() {
-    tsfileEventCount.decrementAndGet();
+  void decreaseTabletEventCount() {
+    tabletEventCount.decrementAndGet();
   }
 
   void increaseTsFileEventCount() {
@@ -84,6 +84,7 @@ class PipeDataNodeRemainingEventAndTimeOperator extends PipeRemainingOperator {
 
   long getRemainingEvents() {
     return tsfileEventCount.get()
+        + tabletEventCount.get()
         + heartbeatEventCount.get()
         + schemaRegionExtractors.stream()
             .map(IoTDBSchemaRegionExtractor::getUnTransferredEventCount)
@@ -105,7 +106,7 @@ class PipeDataNodeRemainingEventAndTimeOperator extends PipeRemainingOperator {
     final double invocationValue = collectInvocationHistogram.getMean();
     // Do not take heartbeat event into account
     final double totalDataRegionWriteEventCount =
-        tsfileEventCount.get() * Math.max(invocationValue, 1) + insertionEventCount.get();
+        tsfileEventCount.get() * Math.max(invocationValue, 1) + tabletEventCount.get();
 
     dataRegionCommitMeter.updateAndGet(
         meter -> {


(iotdb) 01/06: [to rc/1.3.3] Fix compaction cached time chunk flip twice (#13904)

Posted by ha...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

haonan pushed a commit to branch rc/1.3.3
in repository https://gitbox.apache.org/repos/asf/iotdb.git

commit d0d283305e292b67001d0131a862d022612f0076
Author: shuwenwei <55...@users.noreply.github.com>
AuthorDate: Fri Oct 25 13:19:13 2024 +0800

    [to rc/1.3.3] Fix compaction cached time chunk flip twice (#13904)
    
    * fix cached time chunk flip twice
    
    * add ut
    
    * rewind
---
 .../executor/batch/utils/BatchCompactionPlan.java  | 17 ++++++++--
 .../estimator/AbstractCompactionEstimator.java     |  2 +-
 ...atchedAlignedSeriesFastInnerCompactionTest.java | 37 ++++++++++++++++++++++
 3 files changed, 52 insertions(+), 4 deletions(-)

diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/executor/batch/utils/BatchCompactionPlan.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/executor/batch/utils/BatchCompactionPlan.java
index bc3785852de..06473bb5cd9 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/executor/batch/utils/BatchCompactionPlan.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/executor/batch/utils/BatchCompactionPlan.java
@@ -19,6 +19,7 @@
 
 package org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.executor.batch.utils;
 
+import org.apache.iotdb.commons.utils.TestOnly;
 import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.executor.ModifiedStatus;
 
 import org.apache.tsfile.file.metadata.ChunkMetadata;
@@ -35,7 +36,7 @@ import java.util.List;
 import java.util.Map;
 
 public class BatchCompactionPlan {
-  public static final long MAX_CACHED_TIME_CHUNKS_SIZE = 2 * 1024 * 1024;
+  public static long maxCachedTimeChunksSize = 2 * 1024 * 1024;
   private final List<CompactChunkPlan> compactChunkPlans = new ArrayList<>();
   private final Map<String, Map<TimeRange, ModifiedStatus>> alignedPageModifiedStatusCache =
       new HashMap<>();
@@ -51,12 +52,12 @@ public class BatchCompactionPlan {
     if (chunk == null) {
       chunk = reader.readMemChunk(chunkMetadata);
     }
-    chunk.getData().flip();
+    chunk.getData().rewind();
     return chunk;
   }
 
   public void addTimeChunkToCache(String file, long offset, Chunk chunk) {
-    if (cachedTimeChunkSize >= MAX_CACHED_TIME_CHUNKS_SIZE) {
+    if (cachedTimeChunkSize >= maxCachedTimeChunksSize) {
       return;
     }
     cachedTimeChunks.put(
@@ -96,6 +97,16 @@ public class BatchCompactionPlan {
     return compactChunkPlans.isEmpty();
   }
 
+  @TestOnly
+  public static void setMaxCachedTimeChunksSize(long size) {
+    maxCachedTimeChunksSize = size;
+  }
+
+  @TestOnly
+  public static long getMaxCachedTimeChunksSize() {
+    return maxCachedTimeChunksSize;
+  }
+
   @Override
   public String toString() {
     return compactChunkPlans.toString();
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/estimator/AbstractCompactionEstimator.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/estimator/AbstractCompactionEstimator.java
index 4a5554d91da..e07af3ca1f6 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/estimator/AbstractCompactionEstimator.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/estimator/AbstractCompactionEstimator.java
@@ -68,7 +68,7 @@ public abstract class AbstractCompactionEstimator {
               ((double) SystemInfo.getInstance().getMemorySizeForCompaction()
                   / IoTDBDescriptor.getInstance().getConfig().getCompactionThreadCount()
                   * IoTDBDescriptor.getInstance().getConfig().getChunkMetadataSizeProportion())
-          + BatchCompactionPlan.MAX_CACHED_TIME_CHUNKS_SIZE;
+          + BatchCompactionPlan.maxCachedTimeChunksSize;
 
   protected abstract long calculatingMetadataMemoryCost(CompactionTaskInfo taskInfo);
 
diff --git a/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/inner/BatchedAlignedSeriesFastInnerCompactionTest.java b/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/inner/BatchedAlignedSeriesFastInnerCompactionTest.java
index 7f4b1f83bcf..f07b5de8079 100644
--- a/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/inner/BatchedAlignedSeriesFastInnerCompactionTest.java
+++ b/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/inner/BatchedAlignedSeriesFastInnerCompactionTest.java
@@ -29,6 +29,7 @@ import org.apache.iotdb.db.storageengine.dataregion.compaction.constant.Compacti
 import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.impl.FastCompactionPerformer;
 import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.subtask.FastCompactionTaskSummary;
 import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.executor.batch.utils.BatchCompactionPlan;
 import org.apache.iotdb.db.storageengine.dataregion.compaction.utils.CompactionCheckerUtils;
 import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion;
 import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
@@ -318,6 +319,42 @@ public class BatchedAlignedSeriesFastInnerCompactionTest extends AbstractCompact
     validate(targetResource);
   }
 
+  @Test
+  public void testCompactionByDeserializeWithLargeTimeChunk() throws Exception {
+    long defaultMaxCachedTimeChunkSize = BatchCompactionPlan.getMaxCachedTimeChunksSize();
+    try {
+      BatchCompactionPlan.setMaxCachedTimeChunksSize(1);
+      TsFileResource unseqResource1 =
+          generateSingleAlignedSeriesFile(
+              "d0",
+              Arrays.asList("s0", "s1", "s2"),
+              new TimeRange[][] {
+                new TimeRange[] {new TimeRange(100, 200), new TimeRange(500, 600)}
+              },
+              TSEncoding.PLAIN,
+              CompressionType.LZ4,
+              Arrays.asList(false, false, false),
+              false);
+      unseqResources.add(unseqResource1);
+
+      TsFileResource unseqResource2 =
+          generateSingleAlignedSeriesFile(
+              "d0",
+              Arrays.asList("s0", "s1", "s2"),
+              new TimeRange[] {new TimeRange(150, 550)},
+              TSEncoding.PLAIN,
+              CompressionType.LZ4,
+              Arrays.asList(false, false, false),
+              false);
+      unseqResources.add(unseqResource2);
+
+      TsFileResource targetResource = performCompaction();
+      validate(targetResource);
+    } finally {
+      BatchCompactionPlan.setMaxCachedTimeChunksSize(defaultMaxCachedTimeChunkSize);
+    }
+  }
+
   @Test
   public void testCompactionByDeserializeWithEmptyColumn() throws Exception {
     TsFileResource unseqResource1 =


(iotdb) 05/06: [To dev/1.3] Fix query resource clear async thread stuck bug

Posted by ha...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

haonan pushed a commit to branch rc/1.3.3
in repository https://gitbox.apache.org/repos/asf/iotdb.git

commit 7984b196603739469330d1989bab871e9d22c270
Author: Jackie Tien <ja...@gmail.com>
AuthorDate: Sat Nov 9 21:02:58 2024 +0800

    [To dev/1.3] Fix query resource clear async thread stuck bug
---
 .../db/queryengine/execution/driver/Driver.java    | 31 ++++++++++++++++------
 .../apache/iotdb/library/dprofile/UDAFMedian.java  | 14 +++++++---
 2 files changed, 33 insertions(+), 12 deletions(-)

diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/driver/Driver.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/driver/Driver.java
index 812c84298fc..ac353e32213 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/driver/Driver.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/driver/Driver.java
@@ -51,6 +51,7 @@ import java.util.function.Supplier;
 import static com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Throwables.throwIfUnchecked;
+import static com.google.common.base.Verify.verify;
 import static com.google.common.util.concurrent.MoreExecutors.directExecutor;
 import static java.lang.Boolean.TRUE;
 import static org.apache.iotdb.db.queryengine.execution.operator.Operator.NOT_BLOCKED;
@@ -321,15 +322,16 @@ public abstract class Driver implements IDriver {
       return Optional.empty();
     }
 
-    Optional<T> result;
+    T result = null;
+    Throwable failure = null;
     try {
-      result = Optional.of(task.get());
+      result = task.get();
+      // opportunistic check to avoid unnecessary lock reacquisition
+      destroyIfNecessary();
+    } catch (Throwable t) {
+      failure = t;
     } finally {
-      try {
-        destroyIfNecessary();
-      } finally {
-        exclusiveLock.unlock();
-      }
+      exclusiveLock.unlock();
     }
 
     // We need to recheck whether the state is NEED_DESTRUCTION, if so, destroy the driver.
@@ -342,12 +344,25 @@ public abstract class Driver implements IDriver {
     if (state.get() == State.NEED_DESTRUCTION && exclusiveLock.tryLock(interruptOnClose)) {
       try {
         destroyIfNecessary();
+      } catch (Throwable t) {
+        if (failure == null) {
+          failure = t;
+        } else if (failure != t) {
+          failure.addSuppressed(t);
+        }
       } finally {
         exclusiveLock.unlock();
       }
     }
 
-    return result;
+    if (failure != null) {
+      throwIfUnchecked(failure);
+      // should never happen
+      throw new AssertionError(failure);
+    }
+
+    verify(result != null, "result is null");
+    return Optional.of(result);
   }
 
   @SuppressWarnings({"squid:S1181", "squid:S112"})
diff --git a/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFMedian.java b/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFMedian.java
index 89f43fdccc7..9c78bd316e5 100644
--- a/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFMedian.java
+++ b/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFMedian.java
@@ -31,6 +31,8 @@ import org.apache.iotdb.udf.api.customizer.parameter.UDFParameters;
 import org.apache.iotdb.udf.api.customizer.strategy.RowByRowAccessStrategy;
 import org.apache.iotdb.udf.api.type.Type;
 
+import java.util.NoSuchElementException;
+
 /** calculate the exact or approximate median. */
 public class UDAFMedian implements UDTF {
 
@@ -73,10 +75,14 @@ public class UDAFMedian implements UDTF {
 
   @Override
   public void terminate(PointCollector collector) throws Exception {
-    if (exact) {
-      collector.putDouble(0, statistics.getMedian());
-    } else {
-      collector.putDouble(0, sketch.query(0.5));
+    try {
+      if (exact) {
+        collector.putDouble(0, statistics.getMedian());
+      } else {
+        collector.putDouble(0, sketch.query(0.5));
+      }
+    } catch (NoSuchElementException e) {
+      // just ignore it
     }
   }
 }


(iotdb) 03/06: Pipe: Fixed the bug that the PipeDataNodeRemainingEventAndTimeMetrics may generate NullPointerException (#14015) (#14024)

Posted by ha...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

haonan pushed a commit to branch rc/1.3.3
in repository https://gitbox.apache.org/repos/asf/iotdb.git

commit 4375409c5e10ba0cfde9fbda56e3e6af31e6e6f2
Author: Caideyipi <87...@users.noreply.github.com>
AuthorDate: Fri Nov 8 15:36:21 2024 +0800

    Pipe: Fixed the bug that the PipeDataNodeRemainingEventAndTimeMetrics may generate NullPointerException (#14015) (#14024)
---
 .../metric/PipeConfigNodeRemainingTimeMetrics.java |  9 ++++-
 .../PipeConfigNodeRemainingTimeOperator.java       |  5 ++-
 .../event/common/heartbeat/PipeHeartbeatEvent.java |  4 +-
 .../tablet/PipeInsertNodeTabletInsertionEvent.java |  4 +-
 .../common/tablet/PipeRawTabletInsertionEvent.java |  4 +-
 .../common/tsfile/PipeTsFileInsertionEvent.java    |  4 +-
 .../PipeDataNodeRemainingEventAndTimeMetrics.java  | 45 +++++++++++++++-------
 .../PipeDataNodeRemainingEventAndTimeOperator.java |  5 ++-
 .../commons/pipe/metric/PipeRemainingOperator.java | 12 +++---
 9 files changed, 59 insertions(+), 33 deletions(-)

diff --git a/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/metric/PipeConfigNodeRemainingTimeMetrics.java b/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/metric/PipeConfigNodeRemainingTimeMetrics.java
index ab5440eab2d..23c43461c78 100644
--- a/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/metric/PipeConfigNodeRemainingTimeMetrics.java
+++ b/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/metric/PipeConfigNodeRemainingTimeMetrics.java
@@ -103,7 +103,11 @@ public class PipeConfigNodeRemainingTimeMetrics implements IMetricSet {
     // The metric is global thus the regionId is omitted
     final String pipeID = extractor.getPipeName() + "_" + extractor.getCreationTime();
     remainingTimeOperatorMap
-        .computeIfAbsent(pipeID, k -> new PipeConfigNodeRemainingTimeOperator())
+        .computeIfAbsent(
+            pipeID,
+            k ->
+                new PipeConfigNodeRemainingTimeOperator(
+                    extractor.getPipeName(), extractor.getCreationTime()))
         .register(extractor);
     if (Objects.nonNull(metricService)) {
       createMetrics(pipeID);
@@ -157,7 +161,8 @@ public class PipeConfigNodeRemainingTimeMetrics implements IMetricSet {
   public double getRemainingTime(final String pipeName, final long creationTime) {
     return remainingTimeOperatorMap
         .computeIfAbsent(
-            pipeName + "_" + creationTime, k -> new PipeConfigNodeRemainingTimeOperator())
+            pipeName + "_" + creationTime,
+            k -> new PipeConfigNodeRemainingTimeOperator(pipeName, creationTime))
         .getRemainingTime();
   }
 
diff --git a/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/metric/PipeConfigNodeRemainingTimeOperator.java b/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/metric/PipeConfigNodeRemainingTimeOperator.java
index e53a4bb1cb2..298bc521b74 100644
--- a/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/metric/PipeConfigNodeRemainingTimeOperator.java
+++ b/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/metric/PipeConfigNodeRemainingTimeOperator.java
@@ -43,6 +43,10 @@ class PipeConfigNodeRemainingTimeOperator extends PipeRemainingOperator {
 
   private double lastConfigRegionCommitSmoothingValue = Long.MAX_VALUE;
 
+  PipeConfigNodeRemainingTimeOperator(String pipeName, long creationTime) {
+    super(pipeName, creationTime);
+  }
+
   //////////////////////////// Remaining time calculation ////////////////////////////
 
   /**
@@ -91,7 +95,6 @@ class PipeConfigNodeRemainingTimeOperator extends PipeRemainingOperator {
   //////////////////////////// Register & deregister (pipe integration) ////////////////////////////
 
   void register(final IoTDBConfigRegionExtractor extractor) {
-    setNameAndCreationTime(extractor.getPipeName(), extractor.getCreationTime());
     configRegionExtractors.add(extractor);
   }
 
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/heartbeat/PipeHeartbeatEvent.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/heartbeat/PipeHeartbeatEvent.java
index e64c3e12cd6..29f770686ea 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/heartbeat/PipeHeartbeatEvent.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/heartbeat/PipeHeartbeatEvent.java
@@ -84,7 +84,7 @@ public class PipeHeartbeatEvent extends EnrichedEvent {
   public boolean internallyIncreaseResourceReferenceCount(final String holderMessage) {
     if (Objects.nonNull(pipeName)) {
       PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-          .increaseHeartbeatEventCount(pipeName + "_" + creationTime);
+          .increaseHeartbeatEventCount(pipeName, creationTime);
     }
     return true;
   }
@@ -95,7 +95,7 @@ public class PipeHeartbeatEvent extends EnrichedEvent {
     // not the event copied and passed to the extractor
     if (Objects.nonNull(pipeName)) {
       PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-          .decreaseHeartbeatEventCount(pipeName + "_" + creationTime);
+          .decreaseHeartbeatEventCount(pipeName, creationTime);
       if (shouldPrintMessage && LOGGER.isDebugEnabled()) {
         LOGGER.debug(this.toString());
       }
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeInsertNodeTabletInsertionEvent.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeInsertNodeTabletInsertionEvent.java
index 5932c6b544b..4979a3d2777 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeInsertNodeTabletInsertionEvent.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeInsertNodeTabletInsertionEvent.java
@@ -136,7 +136,7 @@ public class PipeInsertNodeTabletInsertionEvent extends EnrichedEvent
       PipeDataNodeResourceManager.wal().pin(walEntryHandler);
       if (Objects.nonNull(pipeName)) {
         PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-            .increaseTabletEventCount(pipeName + "_" + creationTime);
+            .increaseTabletEventCount(pipeName, creationTime);
       }
       return true;
     } catch (final Exception e) {
@@ -169,7 +169,7 @@ public class PipeInsertNodeTabletInsertionEvent extends EnrichedEvent
     } finally {
       if (Objects.nonNull(pipeName)) {
         PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-            .decreaseTabletEventCount(pipeName + "_" + creationTime);
+            .decreaseTabletEventCount(pipeName, creationTime);
       }
     }
   }
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeRawTabletInsertionEvent.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeRawTabletInsertionEvent.java
index b89c11b1f39..24e3c7cd1d4 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeRawTabletInsertionEvent.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/PipeRawTabletInsertionEvent.java
@@ -118,7 +118,7 @@ public class PipeRawTabletInsertionEvent extends EnrichedEvent implements Tablet
                 PipeMemoryWeightUtil.calculateTabletSizeInBytes(tablet));
     if (Objects.nonNull(pipeName)) {
       PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-          .increaseTabletEventCount(pipeName + "_" + creationTime);
+          .increaseTabletEventCount(pipeName, creationTime);
     }
     return true;
   }
@@ -127,7 +127,7 @@ public class PipeRawTabletInsertionEvent extends EnrichedEvent implements Tablet
   public boolean internallyDecreaseResourceReferenceCount(final String holderMessage) {
     if (Objects.nonNull(pipeName)) {
       PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-          .decreaseTabletEventCount(pipeName + "_" + creationTime);
+          .decreaseTabletEventCount(pipeName, creationTime);
     }
     allocatedMemoryBlock.close();
 
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/PipeTsFileInsertionEvent.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/PipeTsFileInsertionEvent.java
index 9e7a8d421e3..49f097b5f02 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/PipeTsFileInsertionEvent.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/PipeTsFileInsertionEvent.java
@@ -247,7 +247,7 @@ public class PipeTsFileInsertionEvent extends EnrichedEvent implements TsFileIns
       }
       if (Objects.nonNull(pipeName)) {
         PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-            .increaseTsFileEventCount(pipeName + "_" + creationTime);
+            .increaseTsFileEventCount(pipeName, creationTime);
       }
       return true;
     } catch (final Exception e) {
@@ -278,7 +278,7 @@ public class PipeTsFileInsertionEvent extends EnrichedEvent implements TsFileIns
     } finally {
       if (Objects.nonNull(pipeName)) {
         PipeDataNodeRemainingEventAndTimeMetrics.getInstance()
-            .decreaseTsFileEventCount(pipeName + "_" + creationTime);
+            .decreaseTsFileEventCount(pipeName, creationTime);
       }
     }
   }
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeMetrics.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeMetrics.java
index 25c2ede2407..85be65eb9f5 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeMetrics.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeMetrics.java
@@ -122,46 +122,62 @@ public class PipeDataNodeRemainingEventAndTimeMetrics implements IMetricSet {
     // The metric is global thus the regionId is omitted
     final String pipeID = extractor.getPipeName() + "_" + extractor.getCreationTime();
     remainingEventAndTimeOperatorMap
-        .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
+        .computeIfAbsent(
+            pipeID,
+            k ->
+                new PipeDataNodeRemainingEventAndTimeOperator(
+                    extractor.getPipeName(), extractor.getCreationTime()))
         .register(extractor);
     if (Objects.nonNull(metricService)) {
       createMetrics(pipeID);
     }
   }
 
-  public void increaseTabletEventCount(final String pipeID) {
+  public void increaseTabletEventCount(final String pipeName, final long creationTime) {
     remainingEventAndTimeOperatorMap
-        .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
+        .computeIfAbsent(
+            pipeName + "_" + creationTime,
+            k -> new PipeDataNodeRemainingEventAndTimeOperator(pipeName, creationTime))
         .increaseTabletEventCount();
   }
 
-  public void decreaseTabletEventCount(final String pipeID) {
+  public void decreaseTabletEventCount(final String pipeName, final long creationTime) {
     remainingEventAndTimeOperatorMap
-        .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
+        .computeIfAbsent(
+            pipeName + "_" + creationTime,
+            k -> new PipeDataNodeRemainingEventAndTimeOperator(pipeName, creationTime))
         .decreaseTabletEventCount();
   }
 
-  public void increaseTsFileEventCount(final String pipeID) {
+  public void increaseTsFileEventCount(final String pipeName, final long creationTime) {
     remainingEventAndTimeOperatorMap
-        .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
+        .computeIfAbsent(
+            pipeName + "_" + creationTime,
+            k -> new PipeDataNodeRemainingEventAndTimeOperator(pipeName, creationTime))
         .increaseTsFileEventCount();
   }
 
-  public void decreaseTsFileEventCount(final String pipeID) {
+  public void decreaseTsFileEventCount(final String pipeName, final long creationTime) {
     remainingEventAndTimeOperatorMap
-        .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
+        .computeIfAbsent(
+            pipeName + "_" + creationTime,
+            k -> new PipeDataNodeRemainingEventAndTimeOperator(pipeName, creationTime))
         .decreaseTsFileEventCount();
   }
 
-  public void increaseHeartbeatEventCount(final String pipeID) {
+  public void increaseHeartbeatEventCount(final String pipeName, final long creationTime) {
     remainingEventAndTimeOperatorMap
-        .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
+        .computeIfAbsent(
+            pipeName + "_" + creationTime,
+            k -> new PipeDataNodeRemainingEventAndTimeOperator(pipeName, creationTime))
         .increaseHeartbeatEventCount();
   }
 
-  public void decreaseHeartbeatEventCount(final String pipeID) {
+  public void decreaseHeartbeatEventCount(final String pipeName, final long creationTime) {
     remainingEventAndTimeOperatorMap
-        .computeIfAbsent(pipeID, k -> new PipeDataNodeRemainingEventAndTimeOperator())
+        .computeIfAbsent(
+            pipeName + "_" + creationTime,
+            k -> new PipeDataNodeRemainingEventAndTimeOperator(pipeName, creationTime))
         .decreaseHeartbeatEventCount();
   }
 
@@ -237,7 +253,8 @@ public class PipeDataNodeRemainingEventAndTimeMetrics implements IMetricSet {
       final String pipeName, final long creationTime) {
     final PipeDataNodeRemainingEventAndTimeOperator operator =
         remainingEventAndTimeOperatorMap.computeIfAbsent(
-            pipeName + "_" + creationTime, k -> new PipeDataNodeRemainingEventAndTimeOperator());
+            pipeName + "_" + creationTime,
+            k -> new PipeDataNodeRemainingEventAndTimeOperator(pipeName, creationTime));
     return new Pair<>(operator.getRemainingEvents(), operator.getRemainingTime());
   }
 
diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeOperator.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeOperator.java
index bee0e6975b4..4194acc9a03 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeOperator.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/PipeDataNodeRemainingEventAndTimeOperator.java
@@ -56,6 +56,10 @@ class PipeDataNodeRemainingEventAndTimeOperator extends PipeRemainingOperator {
   private double lastDataRegionCommitSmoothingValue = Long.MAX_VALUE;
   private double lastSchemaRegionCommitSmoothingValue = Long.MAX_VALUE;
 
+  PipeDataNodeRemainingEventAndTimeOperator(final String pipeName, final long creationTime) {
+    super(pipeName, creationTime);
+  }
+
   //////////////////////////// Remaining event & time calculation ////////////////////////////
 
   void increaseTabletEventCount() {
@@ -163,7 +167,6 @@ class PipeDataNodeRemainingEventAndTimeOperator extends PipeRemainingOperator {
   //////////////////////////// Register & deregister (pipe integration) ////////////////////////////
 
   void register(final IoTDBSchemaRegionExtractor extractor) {
-    setNameAndCreationTime(extractor.getPipeName(), extractor.getCreationTime());
     schemaRegionExtractors.add(extractor);
   }
 
diff --git a/iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/metric/PipeRemainingOperator.java b/iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/metric/PipeRemainingOperator.java
index 97f80e5d985..443b6e42d58 100644
--- a/iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/metric/PipeRemainingOperator.java
+++ b/iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/metric/PipeRemainingOperator.java
@@ -32,6 +32,11 @@ public abstract class PipeRemainingOperator {
   private long lastNonEmptyTimeStamp = System.currentTimeMillis();
   protected boolean isStopped = true;
 
+  protected PipeRemainingOperator(final String pipeName, final long creationTime) {
+    this.pipeName = pipeName;
+    this.creationTime = creationTime;
+  }
+
   //////////////////////////// Tags ////////////////////////////
 
   public String getPipeName() {
@@ -42,13 +47,6 @@ public abstract class PipeRemainingOperator {
     return creationTime;
   }
 
-  //////////////////////////// Register & deregister (pipe integration) ////////////////////////////
-
-  protected void setNameAndCreationTime(final String pipeName, final long creationTime) {
-    this.pipeName = pipeName;
-    this.creationTime = creationTime;
-  }
-
   //////////////////////////// Switch ////////////////////////////
 
   protected void notifyNonEmpty() {


(iotdb) 04/06: Pipe: GC device2Measurements in PipeRealtimeEvent in time to avoid OOM when insertion is in high load (#14018) (#14023)

Posted by ha...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

haonan pushed a commit to branch rc/1.3.3
in repository https://gitbox.apache.org/repos/asf/iotdb.git

commit bc9ef048d74058585a1a1cd1e8933a7b9d651d89
Author: Steve Yurong Su <ro...@apache.org>
AuthorDate: Fri Nov 8 15:37:00 2024 +0800

    Pipe: GC device2Measurements in PipeRealtimeEvent in time to avoid OOM when insertion is in high load (#14018) (#14023)
    
    (cherry picked from commit caac607464359cb194b07bd1fa6c985bb745826e)
---
 .../org/apache/iotdb/db/pipe/event/realtime/PipeRealtimeEvent.java    | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/realtime/PipeRealtimeEvent.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/realtime/PipeRealtimeEvent.java
index 25f6c401779..3bf32f00ea9 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/realtime/PipeRealtimeEvent.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/realtime/PipeRealtimeEvent.java
@@ -171,7 +171,9 @@ public class PipeRealtimeEvent extends EnrichedEvent {
         event.shallowCopySelfAndBindPipeTaskMetaForProgressReport(
             pipeName, creationTime, pipeTaskMeta, pattern, startTime, endTime),
         this.tsFileEpoch,
-        this.device2Measurements,
+        // device2Measurements is not used anymore, so it is not copied.
+        // If null is not passed, the field will not be GCed and may cause OOM.
+        null,
         pipeTaskMeta,
         pattern,
         startTime,


(iotdb) 06/06: Change typo in warn log for disk_space_warning_threshold (#14042)

Posted by ha...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

haonan pushed a commit to branch rc/1.3.3
in repository https://gitbox.apache.org/repos/asf/iotdb.git

commit eb66c590735230062ce1a95bc37e2206c338597d
Author: Jackie Tien <ja...@gmail.com>
AuthorDate: Sun Nov 10 18:14:50 2024 +0800

    Change typo in warn log for disk_space_warning_threshold (#14042)
---
 .../iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java
index 40ac005458e..dcb2b852c36 100644
--- a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java
+++ b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java
@@ -1763,7 +1763,7 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
             "The available disk space is : {}, "
                 + "the total disk space is : {}, "
                 + "and the remaining disk usage ratio: {} is "
-                + "less than disk_spec_warning_threshold: {}, set system to readonly!",
+                + "less than disk_space_warning_threshold: {}, set system to readonly!",
             RamUsageEstimator.humanReadableUnits((long) availableDisk),
             RamUsageEstimator.humanReadableUnits((long) totalDisk),
             freeDiskRatio,