You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by xi...@apache.org on 2022/08/09 07:50:54 UTC
[iotdb] branch ml_0808_test_exp1_parallel updated (6d3a295aeb -> 8b31ab5705)
This is an automated email from the ASF dual-hosted git repository.
xingtanzjr pushed a change to branch ml_0808_test_exp1_parallel
in repository https://gitbox.apache.org/repos/asf/iotdb.git
from 6d3a295aeb previous fix is not correct I think, let's use this one
new 3a743f41ae clear log
add 4c8edc5661 [IOTDB-3985] Retry removePeer for region bug (#6829)
add 54f9452888 Load ConfigNode after registering the internal server (#6832)
add 903a9d6ac0 Extract ISchemaTree interface (#6830)
add 6e7e7e22e3 [IOTDB-3961] Fix bug of Tsfile compact (#6770)
add 4789a5e368 [IOTDB-3999] Avoid using PlanNodeType.ordinal() (#6835)
add e52b8c23c6 [IOTDB-3986] Unified ConfigNode heartbeat request
add def138b590 [IOTDB-3971] Improve the process of writing chunks in compaction for aligned series (#6786)
add 991cbe7e1e [IOTDB-3940] Using bitmap in PageHeader to optimize ValuePageReader from row-base scanning to columnar-style scanning (#6806)
add 22af8d479f remove useless log in cross space compaction (#6837)
add 8ddb8da5f5 Optimize CI workflow (#6841)
add d7627c8751 [IOTDB-4003] del duplicated dependency defination in pom.xml (#6843)
add 55b5b17af2 [IOTDB-3983] Recover wal file's metadata (#6838)
add 2de0dd8731 fix template api in SessionPool for master (#6810)
add 5f118301f6 fix NullPointerException (#6852)
add 8e39a52e23 [IOTDB-3844] Clear up Non-Seed-ConfigNode register process (#6831)
add 1140cf2da6 Influxdb service adapts to distributed MPP framework and fixes some bugs in InfluxFunction and TagInfoRecord (#6828)
add df9c629453 Improve the performance of Raw Query Without ValueFilter for nonAligned (#6853)
add 47d4b06db3 Add the CPU load status, memory status, disk status in DashBoard (#6834)
add ef0e514ca6 [IOTDB-3955] Improved node information metrics to push more detailed node information, online and offline total and survival information of each node (#6801)
add c7b5eeb8cf [IOTDB-3814] Fix insert rows by sql with wrong data type NPE (#6859)
add 21c530abf7 [IOTDB-4007]Support multiple paths in privilege control (#6854)
add 73d5b80f34 [IOTDB-3847] Clear up interfaces in NodeManager(#6862)
add 32fefe24bd [IOTDB-3847] Clear up interfaces in NodeManager (#6862)
add ceae065482 Fix ConfigNode startup bug (#6867)
add e84b1e3185 [IOTDB-3280] Can not set ttl to all sg in cli with one sql (#6842)
add 0bc694c320 [IOTDB-3758] support merge in new cluster (#6866)
add bdc79a6316 [IOTDB-4032] fix show all ttl display is empty in stand (#6877)
add ec589a5f98 Bump testcontainers from 1.15.2 to 1.15.3 (#6740)
add f0cbaf7865 [IOTDB-4026] unknown PhysicalPlan type: 51 (#6873)
add 5010712706 [IOTDB-2952] IOTDB Grafana Plugin: Build on Windows (#6752)
add bcc8913814 [IOTDB-4024] Implement ColumnTransformerVisitor for Expression (#6872)
add c941a41e6f Revert "[IOTDB-2952] IOTDB Grafana Plugin: Build on Windows (#6752)" (#6883)
add f18922b4a6 Fix typo (#6885)
add 8f12321e75 [IOTDB-3973] Implement a new UDF named 'deDup' to only keep distinct … (#6874)
add 4251d49630 [IOTDB-4037] reduce thread count for new standalone server (#6888)
add 8c1b616f9d Change default query execute threads to cpu cores (#6889)
add 5ef5936d56 [IOTDB-4017] Fix Error happened while calling onAcknowledgeDataBlockEvent (#6869)
add 3e4405e465 Fix Calculating size for same TsBlock is not consistent between SinkHandle and SourceHandle (#6882)
add 1bdf3f9598 [IOTDB-4028] Npe in updateNodeLoadStatistic of LoadManager when cluster first starts. (#6886)
add f900a34295 [IOTDB-3982] Shielding heartbeat error log (#6860)
add 23f0d63297 [IOTDB-3820] Build a integration test environment for ConfigNode IT (#6753)
add 786c698613 [IOTDB-4037] Do not start DataNodeInternalRPCService in Standalone version (#6895)
add 3a67962016 change state tracker from RPC to method call locally (#6898)
add 10cab5e83a update website about release-0.13 (#6897)
add ec2b68e9c9 Relacing `select * from root` with `select * from root.**` in documents (#6900)
add d037f1b739 [IOTDB-4023]C++ interface execute SQL query statement and the returned result set records error (#6870)
add ffd346c236 Support TimePartiton in mpp query (#6901)
add ade087b330 [IOTDB-4029] Latent routing policy for MultiLeader protocol (#6880)
add 399340e6fd [IOTDB-3845] Add annotations in confignode.thrift (#6899)
add 8a97792086 [IOTDB-4030] Simplify Privilege Operations SQL (#6875)
add 742cc64c85 [IOTDB-4050] Failed to wait region migrate finished (#6908)
add 39db994138 [IOTDB-4051] fix nodeid duplicated problem (#6909)
add c04767fb60 update administration user-guide (#6913)
add a8f757d6e4 disable the log in state tracker for previous debugging (#6917)
add a2a690b3d8 [IOTDB-2952] IOTDB Grafana Plugin: Build on Windows (#6907)
add b7c4062f28 [IOTDB-4056] Fix the problem that could not find or load main class in shell `remove-confignode.sh` (#6916)
add 87d200e54b Move clientPool factory from ConfigNode to node-commons (#6910)
add e514211827 [IOTDB-4065] support negative timestamp (before 1970 year.) (#6918)
add 51cf191fe1 Refine writeValueColumn() to TsBlockBuilder for common use (#6912)
add a04a1eb62b [IOTDB-3996] Forbidden CQ in REST query API (#6922)
new 8b31ab5705 Merge branch 'master' into ml_0808_test_exp1_parallel
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/client-cpp.yml | 4 +-
.github/workflows/influxdb-protocol.yml | 2 +-
.../org/apache/iotdb/db/qp/sql/IoTDBSqlParser.g4 | 45 +-
checkstyle.xml | 7 +
client-cpp/src/main/Session.cpp | 12 +
client-cpp/src/test/cpp/sessionIT.cpp | 28 ++
.../assembly/resources/sbin/remove-confignode.sh | 40 +-
.../assembly/resources/sbin/start-confignode.sh | 36 +-
.../src/assembly/resources/sbin/stop-confignode.sh | 11 +-
.../confignode/client/DataNodeRequestType.java | 4 +-
...ava => AsyncConfigNodeHeartbeatClientPool.java} | 42 +-
.../async/datanode/AsyncDataNodeClientPool.java | 40 +-
.../datanode/AsyncDataNodeHeartbeatClientPool.java | 72 +++
.../async/handlers/ConfigNodeHeartbeatHandler.java | 16 +-
.../async/handlers/DataNodeHeartbeatHandler.java | 10 +-
.../client/async/handlers/MergeHandler.java | 83 ++++
.../sync/confignode/SyncConfigNodeClientPool.java | 22 +-
.../sync/datanode/SyncDataNodeClientPool.java | 4 +-
.../iotdb/confignode/conf/ConfigNodeConfig.java | 41 --
.../confignode/conf/ConfigNodeDescriptor.java | 17 -
.../consensus/request/ConfigPhysicalPlan.java | 4 +
.../consensus/request/ConfigPhysicalPlanType.java | 2 +
.../consensus/request/auth/AuthorPlan.java | 25 +-
.../consensus/request/write/SetTTLPlan.java | 29 +-
.../exception/AddConsensusGroupException.java | 11 +-
.../confignode/manager/ClusterSchemaManager.java | 83 ++--
.../iotdb/confignode/manager/ConfigManager.java | 101 ++--
.../apache/iotdb/confignode/manager/IManager.java | 10 +-
.../iotdb/confignode/manager/NodeManager.java | 135 ++----
.../iotdb/confignode/manager/PartitionManager.java | 21 +-
.../iotdb/confignode/manager/ProcedureManager.java | 29 +-
.../iotdb/confignode/manager/load/LoadManager.java | 350 ++++++++++++--
.../manager/load/balancer/RouteBalancer.java | 60 ++-
.../load/balancer/router/LazyGreedyRouter.java | 154 +++++++
.../load/heartbeat/ConfigNodeHeartbeatCache.java | 2 +-
.../manager/load/heartbeat/IRegionGroupCache.java | 9 +
.../manager/load/heartbeat/RegionGroupCache.java | 7 +-
.../iotdb/confignode/persistence/AuthorInfo.java | 167 ++++---
.../iotdb/confignode/persistence/NodeInfo.java | 48 +-
.../persistence/executor/ConfigPlanExecutor.java | 10 +-
.../persistence/schema/ClusterSchemaInfo.java | 34 +-
.../procedure/env/ConfigNodeProcedureEnv.java | 20 +-
.../procedure/env/DataNodeRemoveHandler.java | 23 +-
.../procedure/impl/RegionMigrateProcedure.java | 12 +-
.../procedure/state/RegionTransitionState.java | 1 -
.../iotdb/confignode/service/ConfigNode.java | 26 +-
.../service/thrift/ConfigNodeRPCService.java | 15 +-
.../thrift/ConfigNodeRPCServiceProcessor.java | 41 +-
.../request/ConfigPhysicalPlanSerDeTest.java | 123 ++++-
.../load/balancer/router/LazyGreedyRouterTest.java | 166 +++++++
.../load/balancer/router/LeaderRouterTest.java | 2 +-
.../confignode/persistence/AuthorInfoTest.java | 344 ++++++++++++--
.../thrift/ConfigNodeRPCServiceProcessorTest.java | 84 ++--
.../common/request/IndexedConsensusRequest.java | 13 +-
.../multileader/MultiLeaderServerImpl.java | 60 +--
.../multileader/client/DispatchLogHandler.java | 3 -
.../multileader/logdispatcher/LogDispatcher.java | 206 ++++-----
.../service/MultiLeaderRPCServiceProcessor.java | 4 -
docker/src/main/Dockerfile-single-influxdb | 4 +-
docs/Download/README.md | 28 +-
.../Administration-Management/Administration.md | 97 ++--
.../UserGuide/Ecosystem Integration/Spark IoTDB.md | 2 +-
.../Maintenance-Tools/Maintenance-Command.md | 14 +-
docs/UserGuide/Maintenance-Tools/Metric-Tool.md | 10 +-
docs/UserGuide/Operate-Metadata/TTL.md | 21 +-
docs/UserGuide/Reference/SQL-Reference.md | 4 +-
docs/UserGuide/UDF-Library/Series-Processing.md | 70 +++
docs/UserGuide/Write-And-Delete-Data/CSV-Tool.md | 14 +-
docs/zh/Development/VoteRelease.md | 4 +-
docs/zh/Download/README.md | 28 +-
.../Administration-Management/Administration.md | 97 ++--
.../UserGuide/Ecosystem Integration/Spark IoTDB.md | 2 +-
.../Maintenance-Tools/Maintenance-Command.md | 16 +-
docs/zh/UserGuide/Maintenance-Tools/Metric-Tool.md | 22 +-
docs/zh/UserGuide/Operate-Metadata/TTL.md | 18 +
.../Query-Data/Performance-Tracing-Tool.md | 30 +-
docs/zh/UserGuide/Query-Data/Tracing-Tool.md | 30 +-
docs/zh/UserGuide/QuickStart/WayToGetIoTDB.md | 2 +-
docs/zh/UserGuide/Reference/SQL-Reference.md | 2 +-
docs/zh/UserGuide/UDF-Library/Series-Processing.md | 69 +++
.../zh/UserGuide/Write-And-Delete-Data/CSV-Tool.md | 14 +-
.../Apache IoTDB ConfigNode Dashboard v0.14.0.json | 275 ++++++++++-
.../Apache IoTDB DataNode Dashboard v0.14.0.json | 289 +++++++++++-
grafana-plugin/backend-compile.bat | 31 ++
grafana-plugin/go.mod | 2 +-
grafana-plugin/go.sum | 15 +-
grafana-plugin/pom.xml | 54 ++-
integration-test/import-control.xml | 8 +
.../java/org/apache/iotdb/it/env/AbstractEnv.java | 115 ++---
.../org/apache/iotdb/it/env/ConfigFactory.java | 1 +
.../org/apache/iotdb/it/env/ConfigNodeWrapper.java | 4 +
.../org/apache/iotdb/it/env/DataNodeWrapper.java | 16 +
.../java/org/apache/iotdb/it/env/EnvUtils.java | 90 ++++
.../java/org/apache/iotdb/it/env/MppConfig.java | 22 +
.../org/apache/iotdb/it/env/RemoteServerEnv.java | 28 ++
.../apache/iotdb/it/env/StandaloneOnMppEnv.java | 5 +-
.../BuiltinTimeSeriesGeneratingFunctionEnum.java | 1 +
.../org/apache/iotdb/itbase/env/BaseConfig.java | 25 +
.../java/org/apache/iotdb/itbase/env/BaseEnv.java | 15 +
.../org/apache/iotdb/db/it/IoTDBConfigNodeIT.java | 209 +++++++++
.../apache/iotdb/db/it/IoTDBFlushQueryMergeIT.java | 193 ++++++++
.../java/org/apache/iotdb/db/it/IoTDBTtlIT.java | 269 +++++++++++
.../IoTDBAlignedSeriesQueryWithDeletionIT.java | 139 ++++++
.../db/it/aligned/IoTDBInsertAlignedValuesIT.java | 12 +
.../org/apache/iotdb/db/it/env/StandaloneEnv.java | 30 ++
.../db/it/udf/IoTDBUDTFBuiltinFunctionIT.java | 89 +++-
.../iotdb/db/integration/IoTDBAuthorizationIT.java | 20 +-
.../apache/iotdb/db/integration/IoTDBTtlIT.java | 130 ++++++
library-udf/pom.xml | 6 -
.../micrometer/MicrometerMetricManagerTest.java | 11 +
.../apache/iotdb/commons/client/ClientManager.java | 13 +-
.../iotdb/commons/client/ClientPoolFactory.java | 56 ++-
.../iotdb/commons/client/IClientManager.java | 5 +
.../AsyncConfigNodeHeartbeatServiceClient.java | 140 ++++++
.../async/AsyncDataNodeHeartbeatServiceClient.java | 140 ++++++
.../apache/iotdb/commons/cluster/NodeStatus.java | 4 +
.../apache/iotdb/commons/conf/CommonConfig.java | 40 ++
.../iotdb/commons/conf/CommonDescriptor.java | 17 +
.../iotdb/commons/partition/DataPartition.java | 1 +
.../BuiltinTimeSeriesGeneratingFunction.java | 1 +
.../commons/udf/builtin/UDTFDeDuplication.java | 139 ++++++
.../commons/utils/BasicStructureSerDeUtil.java | 31 ++
pom.xml | 8 +-
.../resources/conf/iotdb-datanode.properties | 4 +-
.../iotdb/db/auth/ClusterAuthorityFetcher.java | 9 +-
.../apache/iotdb/db/client/ConfigNodeClient.java | 28 +-
.../java/org/apache/iotdb/db/conf/IoTDBConfig.java | 6 +-
.../org/apache/iotdb/db/conf/IoTDBDescriptor.java | 23 +-
.../statemachine/DataRegionStateMachine.java | 13 +-
.../apache/iotdb/db/engine/StorageEngineV2.java | 30 +-
.../db/engine/compaction/CompactionWorker.java | 34 +-
.../RewriteCrossSpaceCompactionSelector.java | 7 -
.../impl/ReadPointCompactionPerformer.java | 18 +-
.../writer/AbstractCompactionWriter.java | 7 +-
.../writer/CrossSpaceCompactionWriter.java | 19 +-
.../writer/InnerSpaceCompactionWriter.java | 15 +-
.../iotdb/db/engine/storagegroup/DataRegion.java | 3 -
.../iotdb/db/localconfignode/LocalConfigNode.java | 173 ++++---
.../db/metadata/cache/DataNodeSchemaCache.java | 9 +-
.../{SchemaTree.java => ClusterSchemaTree.java} | 22 +-
.../db/mpp/common/schematree/ISchemaTree.java | 70 +++
.../iotdb/db/mpp/execution/driver/Driver.java | 3 +-
.../db/mpp/execution/exchange/ISinkHandle.java | 2 +-
.../db/mpp/execution/exchange/LocalSinkHandle.java | 10 +-
.../db/mpp/execution/exchange/SinkHandle.java | 52 ++-
.../iotdb/db/mpp/execution/memory/MemoryPool.java | 2 +-
.../operator/process/DeviceMergeOperator.java | 2 +-
.../operator/process/FilterAndProjectOperator.java | 2 +-
.../RowBasedTimeJoinOperator.java} | 62 +--
.../process/{ => join}/TimeJoinOperator.java | 7 +-
.../{ => join}/merge/AscTimeComparator.java | 2 +-
.../process/{ => join}/merge/ColumnMerger.java | 24 +-
.../{ => join}/merge/DescTimeComparator.java | 2 +-
.../{ => join}/merge/MultiColumnMerger.java | 56 ++-
.../merge/NonOverlappedMultiColumnMerger.java | 25 +-
.../{ => join}/merge/SingleColumnMerger.java | 44 +-
.../process/{ => join}/merge/TimeComparator.java | 2 +-
.../operator/schema/SchemaFetchMergeOperator.java | 4 +-
.../operator/schema/SchemaFetchScanOperator.java | 8 +-
.../schema/TimeSeriesSchemaScanOperator.java | 25 +-
.../org/apache/iotdb/db/mpp/plan/Coordinator.java | 5 +-
.../apache/iotdb/db/mpp/plan/analyze/Analysis.java | 8 +-
.../iotdb/db/mpp/plan/analyze/AnalyzeVisitor.java | 51 +-
.../db/mpp/plan/analyze/ClusterSchemaFetcher.java | 41 +-
.../db/mpp/plan/analyze/ExpressionAnalyzer.java | 10 +-
.../db/mpp/plan/analyze/FakeSchemaFetcherImpl.java | 11 +-
.../iotdb/db/mpp/plan/analyze/ISchemaFetcher.java | 8 +-
.../iotdb/db/mpp/plan/analyze/SchemaValidator.java | 6 +-
.../mpp/plan/analyze/StandaloneSchemaFetcher.java | 33 +-
.../db/mpp/plan/execution/QueryExecution.java | 3 +-
.../mpp/plan/execution/config/ConfigExecution.java | 6 +-
.../plan/execution/config/ConfigTaskVisitor.java | 29 ++
.../config/executor/ClusterConfigTaskExecutor.java | 52 ++-
.../config/executor/IConfigTaskExecutor.java | 5 +-
.../executor/StandaloneConfigTaskExecutor.java | 36 +-
.../{ => metadata}/CountStorageGroupTask.java | 4 +-
.../config/{ => metadata}/CreateFunctionTask.java | 4 +-
.../{ => metadata}/DeleteStorageGroupTask.java | 4 +-
.../config/{ => metadata}/DropFunctionTask.java | 4 +-
.../config/{ => metadata}/SetStorageGroupTask.java | 4 +-
.../config/{ => metadata}/SetTTLTask.java | 4 +-
.../config/{ => metadata}/ShowClusterTask.java | 4 +-
.../config/{ => metadata}/ShowConfigNodesTask.java | 4 +-
.../config/{ => metadata}/ShowDataNodesTask.java | 4 +-
.../config/{ => metadata}/ShowFunctionsTask.java | 4 +-
.../config/{ => metadata}/ShowRegionTask.java | 4 +-
.../{ => metadata}/ShowStorageGroupTask.java | 4 +-
.../config/{ => metadata}/ShowTTLTask.java | 4 +-
.../config/{ => metadata}/UnSetTTLTask.java | 2 +-
.../template}/CreateSchemaTemplateTask.java | 4 +-
.../template}/SetSchemaTemplateTask.java | 4 +-
.../template}/ShowNodesInSchemaTemplateTask.java | 4 +-
.../template}/ShowPathSetTemplateTask.java | 4 +-
.../template}/ShowSchemaTemplateTask.java | 4 +-
.../execution/config/{ => sys}/AuthorizerTask.java | 4 +-
.../execution/config/{ => sys}/ClearCacheTask.java | 6 +-
.../plan/execution/config/{ => sys}/FlushTask.java | 4 +-
.../{ClearCacheTask.java => sys/MergeTask.java} | 29 +-
.../iotdb/db/mpp/plan/expression/Expression.java | 144 +++---
.../plan/expression/binary/AdditionExpression.java | 12 -
.../plan/expression/binary/BinaryExpression.java | 74 +--
.../plan/expression/binary/DivisionExpression.java | 12 -
.../plan/expression/binary/EqualToExpression.java | 11 -
.../expression/binary/GreaterEqualExpression.java | 12 -
.../expression/binary/GreaterThanExpression.java | 12 -
.../expression/binary/LessEqualExpression.java | 12 -
.../plan/expression/binary/LessThanExpression.java | 12 -
.../plan/expression/binary/LogicAndExpression.java | 11 -
.../plan/expression/binary/LogicOrExpression.java | 11 -
.../plan/expression/binary/ModuloExpression.java | 12 -
.../binary/MultiplicationExpression.java | 12 -
.../plan/expression/binary/NonEqualExpression.java | 12 -
.../expression/binary/SubtractionExpression.java | 12 -
.../mpp/plan/expression/leaf/ConstantOperand.java | 37 +-
.../plan/expression/leaf/TimeSeriesOperand.java | 36 +-
.../mpp/plan/expression/leaf/TimestampOperand.java | 35 +-
.../plan/expression/multi/FunctionExpression.java | 93 +---
.../plan/expression/ternary/BetweenExpression.java | 18 -
.../plan/expression/ternary/TernaryExpression.java | 91 +---
.../db/mpp/plan/expression/unary/InExpression.java | 9 -
.../plan/expression/unary/IsNullExpression.java | 9 -
.../mpp/plan/expression/unary/LikeExpression.java | 9 -
.../plan/expression/unary/LogicNotExpression.java | 9 -
.../plan/expression/unary/NegationExpression.java | 9 -
.../plan/expression/unary/RegularExpression.java | 9 -
.../mpp/plan/expression/unary/UnaryExpression.java | 61 +--
.../visitor/ColumnTransformerVisitor.java | 478 +++++++++++++++++++
.../plan/expression/visitor/ExpressionVisitor.java | 73 +++
.../iotdb/db/mpp/plan/parser/ASTVisitor.java | 125 ++---
.../db/mpp/plan/planner/LogicalPlanVisitor.java | 9 -
.../db/mpp/plan/planner/OperatorTreeGenerator.java | 52 ++-
.../mpp/plan/planner/plan/node/PlanNodeType.java | 4 +
.../planner/plan/node/write/DeleteDataNode.java | 6 +-
.../plan/node/write/InsertMultiTabletsNode.java | 7 +-
.../plan/planner/plan/node/write/InsertNode.java | 4 +-
.../planner/plan/node/write/InsertRowNode.java | 20 +-
.../planner/plan/node/write/InsertRowsNode.java | 7 +-
.../plan/node/write/InsertRowsOfOneDeviceNode.java | 7 +-
.../planner/plan/node/write/InsertTabletNode.java | 6 +-
.../scheduler/AbstractFragInsStateTracker.java | 28 +-
.../db/mpp/plan/scheduler/ClusterScheduler.java | 3 +-
.../scheduler/FixedRateFragInsStateTracker.java | 14 +-
.../mpp/plan/scheduler/SimpleQueryTerminator.java | 4 -
.../db/mpp/plan/scheduler/StandaloneScheduler.java | 28 +-
.../db/mpp/plan/statement/StatementVisitor.java | 10 +-
.../db/mpp/plan/statement/sys/AuthorStatement.java | 20 +-
.../MergeStatement.java} | 36 +-
.../iotdb/db/protocol/influxdb/dto/IoTDBPoint.java | 40 +-
.../function/aggregator/InfluxMeanFunction.java | 4 +-
.../function/selector/InfluxFirstFunction.java | 5 +-
.../function/selector/InfluxLastFunction.java | 5 +-
.../influxdb/handler/AbstractQueryHandler.java | 511 +++++++++++++++++++++
.../protocol/influxdb/handler/NewQueryHandler.java | 200 ++++++++
.../db/protocol/influxdb/handler/QueryHandler.java | 468 +------------------
.../influxdb/meta/AbstractInfluxDBMetaManager.java | 114 +++++
.../influxdb/meta/InfluxDBMetaManager.java | 96 +---
.../influxdb/meta/NewInfluxDBMetaManager.java | 129 ++++++
.../db/protocol/influxdb/meta/TagInfoRecords.java | 28 +-
.../influxdb/util/InfluxReqAndRespUtils.java | 55 +++
.../protocol/influxdb/util/QueryResultUtils.java | 153 ++++++
.../db/protocol/influxdb/util/StringUtils.java | 11 +
.../handler/PhysicalPlanValidationHandler.java | 4 +-
.../apache/iotdb/db/qp/executor/PlanExecutor.java | 92 +++-
.../iotdb/db/qp/logical/sys/AuthorOperator.java | 26 +-
.../iotdb/db/qp/physical/sys/AuthorPlan.java | 62 +--
.../apache/iotdb/db/qp/sql/IoTDBSqlVisitor.java | 101 ++--
.../iotdb/db/service/InfluxDBRPCService.java | 26 +-
.../java/org/apache/iotdb/db/service/NewIoTDB.java | 3 -
.../iotdb/db/service/RegionMigrateService.java | 25 +-
.../recorder/CompactionMetricsRecorder.java | 2 +-
.../handler/InfluxDBServiceThriftHandler.java | 10 +-
.../impl/DataNodeInternalRPCServiceImpl.java | 26 +-
.../thrift/impl/IInfluxDBServiceWithHandler.java | 11 +-
.../service/thrift/impl/InfluxDBServiceImpl.java | 20 +-
.../thrift/impl/NewInfluxDBServiceImpl.java | 128 ++++++
.../iotdb/db/wal/recover/WALNodeRecoverTask.java | 3 +-
.../iotdb/db/wal/recover/WALRecoverWriter.java | 26 +-
.../db/metadata/cache/DataNodeSchemaCacheTest.java | 11 +-
...emaTreeTest.java => ClusterSchemaTreeTest.java} | 28 +-
.../db/mpp/execution/ConfigExecutionTest.java | 18 +-
.../iotdb/db/mpp/execution/DataDriverTest.java | 6 +-
.../execution/exchange/LocalSinkHandleTest.java | 6 +-
.../db/mpp/execution/exchange/SinkHandleTest.java | 70 +--
.../db/mpp/execution/exchange/StubSinkHandle.java | 4 +-
.../iotdb/db/mpp/execution/exchange/Utils.java | 11 +-
.../operator/AlignedSeriesScanOperatorTest.java | 8 +-
.../operator/DeviceMergeOperatorTest.java | 2 +-
.../mpp/execution/operator/LimitOperatorTest.java | 6 +-
.../execution/operator/MultiColumnMergerTest.java | 2 +-
.../NonOverlappedMultiColumnMergerTest.java | 4 +-
.../mpp/execution/operator/OffsetOperatorTest.java | 6 +-
.../operator/RawDataAggregationOperatorTest.java | 6 +-
.../execution/operator/SingleColumnMergerTest.java | 6 +-
.../execution/operator/TimeJoinOperatorTest.java | 8 +-
.../schema/SchemaFetchScanOperatorTest.java | 5 +-
.../write/InsertMultiTabletsNodeSerdeTest.java | 2 +-
.../plan/node/write/InsertRowNodeSerdeTest.java | 8 +-
.../plan/node/write/InsertRowsNodeSerdeTest.java | 2 +-
.../write/InsertRowsOfOneDeviceNodeSerdeTest.java | 3 +-
.../plan/node/write/InsertTabletNodeSerdeTest.java | 6 +-
.../plan/scheduler/StandaloneSchedulerTest.java | 5 -
.../java/org/apache/iotdb/db/qp/PlannerTest.java | 13 +-
.../iotdb/db/qp/physical/PhysicalPlanTest.java | 2 +-
.../iotdb/db/wal/recover/WALRecoverWriterTest.java | 198 ++++++++
.../org/apache/iotdb/session/pool/SessionPool.java | 3 +
thrift-commons/src/main/thrift/common.thrift | 14 +-
.../src/main/thrift/confignode.thrift | 229 +++++++--
thrift/src/main/thrift/datanode.thrift | 10 +-
.../file/metadata/statistics/Statistics.java | 10 +
.../file/metadata/statistics/TimeStatistics.java | 8 +
.../iotdb/tsfile/read/common/block/TsBlock.java | 4 +
.../tsfile/read/common/block/TsBlockBuilder.java | 16 +
.../block/column/BinaryArrayColumnEncoder.java | 26 +-
.../read/common/block/column/BinaryColumn.java | 16 +
.../read/common/block/column/BooleanColumn.java | 16 +
.../block/column/ByteArrayColumnEncoder.java | 19 +-
.../tsfile/read/common/block/column/Column.java | 38 ++
.../read/common/block/column/DoubleColumn.java | 16 +
.../read/common/block/column/FloatColumn.java | 16 +
.../block/column/Int32ArrayColumnEncoder.java | 37 +-
.../block/column/Int64ArrayColumnEncoder.java | 64 ++-
.../tsfile/read/common/block/column/IntColumn.java | 16 +
.../read/common/block/column/LongColumn.java | 16 +
.../block/column/RunLengthEncodedColumn.java | 58 +++
.../read/common/block/column/TimeColumn.java | 11 +
.../tsfile/read/reader/page/AlignedPageReader.java | 98 +++-
.../tsfile/read/reader/page/ValuePageReader.java | 99 ++++
.../tsfile/write/chunk/AlignedChunkWriterImpl.java | 63 +++
.../iotdb/tsfile/write/chunk/TimeChunkWriter.java | 10 +-
.../iotdb/tsfile/write/chunk/ValueChunkWriter.java | 24 +-
.../iotdb/tsfile/write/page/TimePageWriter.java | 6 +-
.../iotdb/tsfile/write/page/ValuePageWriter.java | 72 ++-
.../tsfile/common/block/TsBlockSerdeTest.java | 139 ++++--
333 files changed, 9546 insertions(+), 3687 deletions(-)
rename confignode/src/main/java/org/apache/iotdb/confignode/client/async/confignode/{AsyncConfigNodeClientPool.java => AsyncConfigNodeHeartbeatClientPool.java} (55%)
create mode 100644 confignode/src/main/java/org/apache/iotdb/confignode/client/async/datanode/AsyncDataNodeHeartbeatClientPool.java
create mode 100644 confignode/src/main/java/org/apache/iotdb/confignode/client/async/handlers/MergeHandler.java
copy server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/UnSetTTLTask.java => confignode/src/main/java/org/apache/iotdb/confignode/exception/AddConsensusGroupException.java (69%)
create mode 100644 confignode/src/main/java/org/apache/iotdb/confignode/manager/load/balancer/router/LazyGreedyRouter.java
create mode 100644 confignode/src/test/java/org/apache/iotdb/confignode/manager/load/balancer/router/LazyGreedyRouterTest.java
create mode 100644 docs/UserGuide/UDF-Library/Series-Processing.md
create mode 100644 docs/zh/UserGuide/UDF-Library/Series-Processing.md
create mode 100644 grafana-plugin/backend-compile.bat
create mode 100644 integration-test/src/main/java/org/apache/iotdb/it/env/EnvUtils.java
create mode 100644 integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBConfigNodeIT.java
create mode 100644 integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBFlushQueryMergeIT.java
create mode 100644 integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBTtlIT.java
create mode 100644 integration-test/src/test/java/org/apache/iotdb/db/it/aligned/IoTDBAlignedSeriesQueryWithDeletionIT.java
rename confignode/src/main/java/org/apache/iotdb/confignode/client/ConfigNodeClientPoolFactory.java => node-commons/src/main/java/org/apache/iotdb/commons/client/ClientPoolFactory.java (55%)
create mode 100644 node-commons/src/main/java/org/apache/iotdb/commons/client/async/AsyncConfigNodeHeartbeatServiceClient.java
create mode 100644 node-commons/src/main/java/org/apache/iotdb/commons/client/async/AsyncDataNodeHeartbeatServiceClient.java
create mode 100644 node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/UDTFDeDuplication.java
rename server/src/main/java/org/apache/iotdb/db/mpp/common/schematree/{SchemaTree.java => ClusterSchemaTree.java} (95%)
create mode 100644 server/src/main/java/org/apache/iotdb/db/mpp/common/schematree/ISchemaTree.java
copy server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{TimeJoinOperator.java => join/RowBasedTimeJoinOperator.java} (85%)
rename server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{ => join}/TimeJoinOperator.java (96%)
rename server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{ => join}/merge/AscTimeComparator.java (94%)
rename server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{ => join}/merge/ColumnMerger.java (70%)
rename server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{ => join}/merge/DescTimeComparator.java (94%)
rename server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{ => join}/merge/MultiColumnMerger.java (63%)
rename server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{ => join}/merge/NonOverlappedMultiColumnMerger.java (82%)
rename server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{ => join}/merge/SingleColumnMerger.java (74%)
rename server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/process/{ => join}/merge/TimeComparator.java (94%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/CountStorageGroupTask.java (92%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/CreateFunctionTask.java (89%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/DeleteStorageGroupTask.java (89%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/DropFunctionTask.java (87%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/SetStorageGroupTask.java (93%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/SetTTLTask.java (88%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/ShowClusterTask.java (94%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/ShowConfigNodesTask.java (93%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/ShowDataNodesTask.java (94%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/ShowFunctionsTask.java (96%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/ShowRegionTask.java (95%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/ShowStorageGroupTask.java (94%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/ShowTTLTask.java (92%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata}/UnSetTTLTask.java (94%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata/template}/CreateSchemaTemplateTask.java (88%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata/template}/SetSchemaTemplateTask.java (88%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata/template}/ShowNodesInSchemaTemplateTask.java (94%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata/template}/ShowPathSetTemplateTask.java (92%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => metadata/template}/ShowSchemaTemplateTask.java (93%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => sys}/AuthorizerTask.java (90%)
copy server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => sys}/ClearCacheTask.java (88%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ => sys}/FlushTask.java (92%)
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/execution/config/{ClearCacheTask.java => sys/MergeTask.java} (61%)
create mode 100644 server/src/main/java/org/apache/iotdb/db/mpp/plan/expression/visitor/ColumnTransformerVisitor.java
create mode 100644 server/src/main/java/org/apache/iotdb/db/mpp/plan/expression/visitor/ExpressionVisitor.java
rename server/src/main/java/org/apache/iotdb/db/mpp/plan/statement/{internal/LastPointFetchStatement.java => sys/MergeStatement.java} (59%)
create mode 100644 server/src/main/java/org/apache/iotdb/db/protocol/influxdb/handler/AbstractQueryHandler.java
create mode 100644 server/src/main/java/org/apache/iotdb/db/protocol/influxdb/handler/NewQueryHandler.java
create mode 100644 server/src/main/java/org/apache/iotdb/db/protocol/influxdb/meta/AbstractInfluxDBMetaManager.java
create mode 100644 server/src/main/java/org/apache/iotdb/db/protocol/influxdb/meta/NewInfluxDBMetaManager.java
create mode 100644 server/src/main/java/org/apache/iotdb/db/protocol/influxdb/util/InfluxReqAndRespUtils.java
copy confignode/src/main/java/org/apache/iotdb/confignode/procedure/state/RegionTransitionState.java => server/src/main/java/org/apache/iotdb/db/service/thrift/impl/IInfluxDBServiceWithHandler.java (77%)
create mode 100644 server/src/main/java/org/apache/iotdb/db/service/thrift/impl/NewInfluxDBServiceImpl.java
rename server/src/test/java/org/apache/iotdb/db/mpp/common/schematree/{SchemaTreeTest.java => ClusterSchemaTreeTest.java} (95%)
create mode 100644 server/src/test/java/org/apache/iotdb/db/wal/recover/WALRecoverWriterTest.java
[iotdb] 01/02: clear log
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xingtanzjr pushed a commit to branch ml_0808_test_exp1_parallel
in repository https://gitbox.apache.org/repos/asf/iotdb.git
commit 3a743f41ae5b4f522626b85cdd4bf923f978e1d0
Author: Jinrui.Zhang <xi...@gmail.com>
AuthorDate: Tue Aug 9 15:49:49 2022 +0800
clear log
---
.../common/request/IndexedConsensusRequest.java | 13 +-
.../multileader/MultiLeaderServerImpl.java | 60 +++---
.../multileader/client/DispatchLogHandler.java | 3 -
.../multileader/logdispatcher/LogDispatcher.java | 206 ++++++++-------------
.../service/MultiLeaderRPCServiceProcessor.java | 4 -
.../statemachine/DataRegionStateMachine.java | 13 +-
.../iotdb/db/engine/storagegroup/DataRegion.java | 3 -
7 files changed, 109 insertions(+), 193 deletions(-)
diff --git a/consensus/src/main/java/org/apache/iotdb/consensus/common/request/IndexedConsensusRequest.java b/consensus/src/main/java/org/apache/iotdb/consensus/common/request/IndexedConsensusRequest.java
index 9ce706d09a..1a61cf3a09 100644
--- a/consensus/src/main/java/org/apache/iotdb/consensus/common/request/IndexedConsensusRequest.java
+++ b/consensus/src/main/java/org/apache/iotdb/consensus/common/request/IndexedConsensusRequest.java
@@ -19,8 +19,6 @@
package org.apache.iotdb.consensus.common.request;
-import org.apache.iotdb.commons.StepTracker;
-
import java.nio.ByteBuffer;
import java.util.LinkedList;
import java.util.List;
@@ -69,14 +67,9 @@ public class IndexedConsensusRequest implements IConsensusRequest {
}
public List<ByteBuffer> buildSerializedRequests() {
- long startTime = System.nanoTime();
- try {
- List<ByteBuffer> result = new LinkedList<>();
- this.requests.forEach(r -> result.add(r.serializeToByteBuffer()));
- return result;
- } finally {
- StepTracker.trace("buildSerializedRequests", startTime, System.nanoTime());
- }
+ List<ByteBuffer> result = new LinkedList<>();
+ this.requests.forEach(r -> result.add(r.serializeToByteBuffer()));
+ return result;
}
public long getSearchIndex() {
diff --git a/consensus/src/main/java/org/apache/iotdb/consensus/multileader/MultiLeaderServerImpl.java b/consensus/src/main/java/org/apache/iotdb/consensus/multileader/MultiLeaderServerImpl.java
index a2fd7c0e47..096cc160a0 100644
--- a/consensus/src/main/java/org/apache/iotdb/consensus/multileader/MultiLeaderServerImpl.java
+++ b/consensus/src/main/java/org/apache/iotdb/consensus/multileader/MultiLeaderServerImpl.java
@@ -21,7 +21,6 @@ package org.apache.iotdb.consensus.multileader;
import org.apache.iotdb.common.rpc.thrift.TEndPoint;
import org.apache.iotdb.common.rpc.thrift.TSStatus;
-import org.apache.iotdb.commons.StepTracker;
import org.apache.iotdb.commons.client.IClientManager;
import org.apache.iotdb.consensus.IStateMachine;
import org.apache.iotdb.consensus.common.DataSet;
@@ -110,44 +109,33 @@ public class MultiLeaderServerImpl {
* records the index of the log and writes locally, and then asynchronous replication is performed
*/
public TSStatus write(IConsensusRequest request) {
- long leaderWriteStartTime = System.nanoTime();
synchronized (stateMachine) {
- StepTracker.trace("LeaderWriteWaitLock", leaderWriteStartTime, System.nanoTime());
- long startTimeAfterLock = System.nanoTime();
- try {
- IndexedConsensusRequest indexedConsensusRequest =
- buildIndexedConsensusRequestForLocalRequest(request);
- if (indexedConsensusRequest.getSearchIndex() % 1000 == 0) {
- logger.info(
- "DataRegion[{}]: index after build: safeIndex:{}, searchIndex: {}",
- thisNode.getGroupId(),
- getCurrentSafelyDeletedSearchIndex(),
- indexedConsensusRequest.getSearchIndex());
- }
- // TODO wal and memtable
- TSStatus result = stateMachine.write(indexedConsensusRequest);
- // TSStatus result = new TSStatus(TSStatusCode.SUCCESS_STATUS.getStatusCode());
- StepTracker.trace("stateMachineWrite", startTimeAfterLock, System.nanoTime());
- long offerStartTime = System.nanoTime();
- if (result.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
- synchronized (index) {
- logDispatcher.offer(indexedConsensusRequest);
- index.incrementAndGet();
- }
- } else {
- logger.debug(
- "{}: write operation failed. searchIndex: {}. Code: {}",
- thisNode.getGroupId(),
- indexedConsensusRequest.getSearchIndex(),
- result.getCode());
- index.decrementAndGet();
+ IndexedConsensusRequest indexedConsensusRequest =
+ buildIndexedConsensusRequestForLocalRequest(request);
+ if (indexedConsensusRequest.getSearchIndex() % 1000 == 0) {
+ logger.info(
+ "DataRegion[{}]: index after build: safeIndex:{}, searchIndex: {}",
+ thisNode.getGroupId(),
+ getCurrentSafelyDeletedSearchIndex(),
+ indexedConsensusRequest.getSearchIndex());
+ }
+ // TODO wal and memtable
+ TSStatus result = stateMachine.write(indexedConsensusRequest);
+ long offerStartTime = System.nanoTime();
+ if (result.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ synchronized (index) {
+ logDispatcher.offer(indexedConsensusRequest);
+ index.incrementAndGet();
}
- StepTracker.trace("serializeAndOfferToQueue", offerStartTime, System.nanoTime());
- return result;
- } finally {
- StepTracker.trace("MultiLeaderWriteAfterLock", startTimeAfterLock, System.nanoTime());
- StepTracker.trace("MultiLeaderWriteWhole", leaderWriteStartTime, System.nanoTime());
+ } else {
+ logger.debug(
+ "{}: write operation failed. searchIndex: {}. Code: {}",
+ thisNode.getGroupId(),
+ indexedConsensusRequest.getSearchIndex(),
+ result.getCode());
+ index.decrementAndGet();
}
+ return result;
}
}
diff --git a/consensus/src/main/java/org/apache/iotdb/consensus/multileader/client/DispatchLogHandler.java b/consensus/src/main/java/org/apache/iotdb/consensus/multileader/client/DispatchLogHandler.java
index ab6cfe655f..b4cefed078 100644
--- a/consensus/src/main/java/org/apache/iotdb/consensus/multileader/client/DispatchLogHandler.java
+++ b/consensus/src/main/java/org/apache/iotdb/consensus/multileader/client/DispatchLogHandler.java
@@ -19,7 +19,6 @@
package org.apache.iotdb.consensus.multileader.client;
-import org.apache.iotdb.commons.StepTracker;
import org.apache.iotdb.consensus.multileader.logdispatcher.LogDispatcher.LogDispatcherThread;
import org.apache.iotdb.consensus.multileader.logdispatcher.PendingBatch;
import org.apache.iotdb.consensus.multileader.thrift.TSyncLogRes;
@@ -38,7 +37,6 @@ public class DispatchLogHandler implements AsyncMethodCallback<TSyncLogRes> {
private final LogDispatcherThread thread;
private final PendingBatch batch;
private int retryCount;
- private final long startTime = System.nanoTime();
public DispatchLogHandler(LogDispatcherThread thread, PendingBatch batch) {
this.thread = thread;
@@ -47,7 +45,6 @@ public class DispatchLogHandler implements AsyncMethodCallback<TSyncLogRes> {
@Override
public void onComplete(TSyncLogRes response) {
- StepTracker.trace("leaderSendUtilResponse", 25, startTime, System.nanoTime());
if (response.getStatus().size() == 1
&& response.getStatus().get(0).getCode()
== TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()) {
diff --git a/consensus/src/main/java/org/apache/iotdb/consensus/multileader/logdispatcher/LogDispatcher.java b/consensus/src/main/java/org/apache/iotdb/consensus/multileader/logdispatcher/LogDispatcher.java
index daa34122a5..5f29665346 100644
--- a/consensus/src/main/java/org/apache/iotdb/consensus/multileader/logdispatcher/LogDispatcher.java
+++ b/consensus/src/main/java/org/apache/iotdb/consensus/multileader/logdispatcher/LogDispatcher.java
@@ -20,7 +20,6 @@
package org.apache.iotdb.consensus.multileader.logdispatcher;
import org.apache.iotdb.common.rpc.thrift.TEndPoint;
-import org.apache.iotdb.commons.StepTracker;
import org.apache.iotdb.commons.client.IClientManager;
import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
import org.apache.iotdb.consensus.common.Peer;
@@ -116,28 +115,9 @@ public class LogDispatcher {
impl.getThisNode().getGroupId(),
thread.getPeer().getEndpoint().getIp(),
thread.getPendingRequest().size());
- // long putToQueueStartTime = System.nanoTime();
- // try {
- // while (!thread
- // .getPendingRequest()
- // .offer(new IndexedConsensusRequest(serializedRequests,
- // request.getSearchIndex()))) {
- // impl.getIndexObject().wait();
- // }
- // ;
- // } catch (InterruptedException e) {
- // e.printStackTrace();
- // } finally {
- // logger.info("{}: Push a log to the queue, done",
- // impl.getThisNode().getGroupId());
- // StepTracker.trace("putToQueueWaitingTime", putToQueueStartTime,
- // System.nanoTime());
- // }
- if (thread
+ if (!thread
.getPendingRequest()
.offer(new IndexedConsensusRequest(serializedRequests, request.getSearchIndex()))) {
- thread.countQueue(request.getSearchIndex());
- } else {
logger.info(
"{}: Log queue of {} is full, ignore the log to this node, searchIndex: {}",
impl.getThisNode().getGroupId(),
@@ -147,14 +127,6 @@ public class LogDispatcher {
});
}
- private boolean needPutIntoQueue() {
- return threads.stream()
- .anyMatch(
- t ->
- t.getPendingRequest().size()
- < t.config.getReplication().getMaxPendingRequestNumPerNode());
- }
-
public class LogDispatcherThread implements Runnable {
private static final long PENDING_REQUEST_TAKING_TIME_OUT_IN_SEC = 10;
private final MultiLeaderConfig config;
@@ -187,16 +159,6 @@ public class LogDispatcher {
this.walEntryiterator = reader.getReqIterator(iteratorIndex);
}
- public void countQueue(long searchIndex) {
- this.queueCount++;
- if (queueCount % 100 == 0) {
- logger.info(
- String.format(
- "DataRegion[%s]->%s: total request from queue: [%d], requestIndex: [%d]",
- peer.getGroupId().getId(), peer.getEndpoint().ip, queueCount, searchIndex));
- }
- }
-
public IndexController getController() {
return controller;
}
@@ -244,10 +206,7 @@ public class LogDispatcher {
}
}
// we may block here if the synchronization pipeline is full
- long getBatchSlotStartTime = System.nanoTime();
syncStatus.addNextBatch(batch);
- StepTracker.trace("batchSize", 25, 0, batch.getBatches().size() * 1000_000L);
- StepTracker.trace("getBatchSlot", 25, getBatchSlotStartTime, System.nanoTime());
// sends batch asynchronously and migrates the retry logic into the callback handler
sendBatchAsync(batch, new DispatchLogHandler(this, batch));
}
@@ -267,99 +226,92 @@ public class LogDispatcher {
}
public PendingBatch getBatch() {
- long getBatchStartTime = System.nanoTime();
- try {
- PendingBatch batch;
- List<TLogBatch> logBatches = new ArrayList<>();
- long startIndex = syncStatus.getNextSendingIndex();
- long maxIndexWhenBufferedRequestEmpty = startIndex;
- logger.debug("[GetBatch] startIndex: {}", startIndex);
- long endIndex;
- if (bufferedRequest.size() <= config.getReplication().getMaxRequestPerBatch()) {
- // Use drainTo instead of poll to reduce lock overhead
- logger.debug(
- "{} : pendingRequest Size: {}, bufferedRequest size: {}",
- impl.getThisNode().getGroupId(),
- pendingRequest.size(),
- bufferedRequest.size());
- synchronized (impl.getIndexObject()) {
- pendingRequest.drainTo(
- bufferedRequest,
- config.getReplication().getMaxRequestPerBatch() - bufferedRequest.size());
- maxIndexWhenBufferedRequestEmpty = impl.getIndex() + 1;
- // impl.getIndexObject().notifyAll();
- }
- // remove all request that searchIndex < startIndex
- Iterator<IndexedConsensusRequest> iterator = bufferedRequest.iterator();
- while (iterator.hasNext()) {
- IndexedConsensusRequest request = iterator.next();
- if (request.getSearchIndex() < startIndex) {
- iterator.remove();
- } else {
- break;
- }
+ PendingBatch batch;
+ List<TLogBatch> logBatches = new ArrayList<>();
+ long startIndex = syncStatus.getNextSendingIndex();
+ long maxIndexWhenBufferedRequestEmpty = startIndex;
+ logger.debug("[GetBatch] startIndex: {}", startIndex);
+ long endIndex;
+ if (bufferedRequest.size() <= config.getReplication().getMaxRequestPerBatch()) {
+ // Use drainTo instead of poll to reduce lock overhead
+ logger.debug(
+ "{} : pendingRequest Size: {}, bufferedRequest size: {}",
+ impl.getThisNode().getGroupId(),
+ pendingRequest.size(),
+ bufferedRequest.size());
+ synchronized (impl.getIndexObject()) {
+ pendingRequest.drainTo(
+ bufferedRequest,
+ config.getReplication().getMaxRequestPerBatch() - bufferedRequest.size());
+ maxIndexWhenBufferedRequestEmpty = impl.getIndex() + 1;
+ // impl.getIndexObject().notifyAll();
+ }
+ // remove all request that searchIndex < startIndex
+ Iterator<IndexedConsensusRequest> iterator = bufferedRequest.iterator();
+ while (iterator.hasNext()) {
+ IndexedConsensusRequest request = iterator.next();
+ if (request.getSearchIndex() < startIndex) {
+ iterator.remove();
+ } else {
+ break;
}
}
- // This condition will be executed in several scenarios:
- // 1. restart
- // 2. The getBatch() is invoked immediately at the moment the PendingRequests are consumed
- // up.
- if (bufferedRequest.isEmpty()) {
- endIndex =
- constructBatchFromWAL(startIndex, maxIndexWhenBufferedRequestEmpty, logBatches);
+ }
+ // This condition will be executed in several scenarios:
+ // 1. restart
+ // 2. The getBatch() is invoked immediately at the moment the PendingRequests are consumed
+ // up.
+ if (bufferedRequest.isEmpty()) {
+ endIndex = constructBatchFromWAL(startIndex, maxIndexWhenBufferedRequestEmpty, logBatches);
+ batch = new PendingBatch(startIndex, endIndex, logBatches);
+ logger.debug(
+ "{} : accumulated a {} from wal when empty", impl.getThisNode().getGroupId(), batch);
+ } else {
+ // Notice that prev searchIndex >= startIndex
+ Iterator<IndexedConsensusRequest> iterator = bufferedRequest.iterator();
+ IndexedConsensusRequest prev = iterator.next();
+ // Prevents gap between logs. For example, some requests are not written into the queue
+ // when
+ // the queue is full. In this case, requests need to be loaded from the WAL
+ endIndex = constructBatchFromWAL(startIndex, prev.getSearchIndex(), logBatches);
+ if (logBatches.size() == config.getReplication().getMaxRequestPerBatch()) {
batch = new PendingBatch(startIndex, endIndex, logBatches);
- logger.debug(
- "{} : accumulated a {} from wal when empty", impl.getThisNode().getGroupId(), batch);
- } else {
- // Notice that prev searchIndex >= startIndex
- Iterator<IndexedConsensusRequest> iterator = bufferedRequest.iterator();
- IndexedConsensusRequest prev = iterator.next();
- // Prevents gap between logs. For example, some requests are not written into the queue
- // when
+ logger.debug("{} : accumulated a {} from wal", impl.getThisNode().getGroupId(), batch);
+ return batch;
+ }
+ constructBatchIndexedFromConsensusRequest(prev, logBatches);
+ endIndex = prev.getSearchIndex();
+ iterator.remove();
+ while (iterator.hasNext()
+ && logBatches.size() <= config.getReplication().getMaxRequestPerBatch()) {
+ IndexedConsensusRequest current = iterator.next();
+ // Prevents gap between logs. For example, some logs are not written into the queue when
// the queue is full. In this case, requests need to be loaded from the WAL
- endIndex = constructBatchFromWAL(startIndex, prev.getSearchIndex(), logBatches);
- if (logBatches.size() == config.getReplication().getMaxRequestPerBatch()) {
- batch = new PendingBatch(startIndex, endIndex, logBatches);
- logger.debug("{} : accumulated a {} from wal", impl.getThisNode().getGroupId(), batch);
- return batch;
- }
- constructBatchIndexedFromConsensusRequest(prev, logBatches);
- endIndex = prev.getSearchIndex();
- iterator.remove();
- while (iterator.hasNext()
- && logBatches.size() <= config.getReplication().getMaxRequestPerBatch()) {
- IndexedConsensusRequest current = iterator.next();
- // Prevents gap between logs. For example, some logs are not written into the queue when
- // the queue is full. In this case, requests need to be loaded from the WAL
- if (current.getSearchIndex() != prev.getSearchIndex() + 1) {
- endIndex =
- constructBatchFromWAL(
- prev.getSearchIndex(), current.getSearchIndex(), logBatches);
- if (logBatches.size() == config.getReplication().getMaxRequestPerBatch()) {
- batch = new PendingBatch(startIndex, endIndex, logBatches);
- logger.debug(
- "gap {} : accumulated a {} from queue and wal when gap",
- impl.getThisNode().getGroupId(),
- batch);
- return batch;
- }
+ if (current.getSearchIndex() != prev.getSearchIndex() + 1) {
+ endIndex =
+ constructBatchFromWAL(prev.getSearchIndex(), current.getSearchIndex(), logBatches);
+ if (logBatches.size() == config.getReplication().getMaxRequestPerBatch()) {
+ batch = new PendingBatch(startIndex, endIndex, logBatches);
+ logger.debug(
+ "gap {} : accumulated a {} from queue and wal when gap",
+ impl.getThisNode().getGroupId(),
+ batch);
+ return batch;
}
- constructBatchIndexedFromConsensusRequest(current, logBatches);
- endIndex = current.getSearchIndex();
- prev = current;
- // We might not be able to remove all the elements in the bufferedRequest in the
- // current function, but that's fine, we'll continue processing these elements in the
- // bufferedRequest the next time we go into the function, they're never lost
- iterator.remove();
}
- batch = new PendingBatch(startIndex, endIndex, logBatches);
- logger.debug(
- "{} : accumulated a {} from queue and wal", impl.getThisNode().getGroupId(), batch);
+ constructBatchIndexedFromConsensusRequest(current, logBatches);
+ endIndex = current.getSearchIndex();
+ prev = current;
+ // We might not be able to remove all the elements in the bufferedRequest in the
+ // current function, but that's fine, we'll continue processing these elements in the
+ // bufferedRequest the next time we go into the function, they're never lost
+ iterator.remove();
}
- return batch;
- } finally {
- StepTracker.trace("getBatch()", 25, getBatchStartTime, System.nanoTime());
+ batch = new PendingBatch(startIndex, endIndex, logBatches);
+ logger.debug(
+ "{} : accumulated a {} from queue and wal", impl.getThisNode().getGroupId(), batch);
}
+ return batch;
}
public void sendBatchAsync(PendingBatch batch, DispatchLogHandler handler) {
diff --git a/consensus/src/main/java/org/apache/iotdb/consensus/multileader/service/MultiLeaderRPCServiceProcessor.java b/consensus/src/main/java/org/apache/iotdb/consensus/multileader/service/MultiLeaderRPCServiceProcessor.java
index 1b0328975b..70a0387c40 100644
--- a/consensus/src/main/java/org/apache/iotdb/consensus/multileader/service/MultiLeaderRPCServiceProcessor.java
+++ b/consensus/src/main/java/org/apache/iotdb/consensus/multileader/service/MultiLeaderRPCServiceProcessor.java
@@ -20,7 +20,6 @@
package org.apache.iotdb.consensus.multileader.service;
import org.apache.iotdb.common.rpc.thrift.TSStatus;
-import org.apache.iotdb.commons.StepTracker;
import org.apache.iotdb.commons.consensus.ConsensusGroupId;
import org.apache.iotdb.consensus.common.request.BatchIndexedConsensusRequest;
import org.apache.iotdb.consensus.common.request.ByteBufferConsensusRequest;
@@ -54,7 +53,6 @@ public class MultiLeaderRPCServiceProcessor implements MultiLeaderConsensusIServ
@Override
public void syncLog(TSyncLogReq req, AsyncMethodCallback<TSyncLogRes> resultHandler) {
- long syncLogStartTime = System.nanoTime();
try {
ConsensusGroupId groupId =
ConsensusGroupId.Factory.createFromTConsensusGroupId(req.getConsensusGroupId());
@@ -103,8 +101,6 @@ public class MultiLeaderRPCServiceProcessor implements MultiLeaderConsensusIServ
resultHandler.onComplete(new TSyncLogRes(writeStatus.subStatus));
} catch (Exception e) {
resultHandler.onError(e);
- } finally {
- StepTracker.trace("syncLog", 25, syncLogStartTime, System.nanoTime());
}
}
diff --git a/server/src/main/java/org/apache/iotdb/db/consensus/statemachine/DataRegionStateMachine.java b/server/src/main/java/org/apache/iotdb/db/consensus/statemachine/DataRegionStateMachine.java
index 259d88b9fb..4b7a416740 100644
--- a/server/src/main/java/org/apache/iotdb/db/consensus/statemachine/DataRegionStateMachine.java
+++ b/server/src/main/java/org/apache/iotdb/db/consensus/statemachine/DataRegionStateMachine.java
@@ -123,13 +123,6 @@ public class DataRegionStateMachine extends BaseStateMachine {
}
private TSStatus cacheAndInsertLatestNode(InsertNodeWrapper insertNodeWrapper) {
- long cacheRequestStartTime = System.nanoTime();
- logger.info(
- "region = {}, queue size = {}, syncIndex = {}, nextSyncIndex = {}",
- region.getDataRegionId(),
- requestCache.size(),
- insertNodeWrapper.startSyncIndex,
- nextSyncIndex);
queueLock.lock();
try {
requestCache.add(insertNodeWrapper);
@@ -177,9 +170,9 @@ public class DataRegionStateMachine extends BaseStateMachine {
Thread.currentThread().interrupt();
}
}
- StepTracker.trace("cacheAndQueueRequest", cacheRequestStartTime, System.nanoTime());
- logger.info(
- "queue size {}, startSyncIndex = {}, endSyncIndex = {}",
+ logger.debug(
+ "region = {}, queue size {}, startSyncIndex = {}, endSyncIndex = {}",
+ region.getDataRegionId(),
requestCache.size(),
insertNodeWrapper.getStartSyncIndex(),
insertNodeWrapper.getEndSyncIndex());
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/DataRegion.java b/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/DataRegion.java
index ff1737e994..3c08cb50cb 100755
--- a/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/DataRegion.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/DataRegion.java
@@ -19,7 +19,6 @@
package org.apache.iotdb.db.engine.storagegroup;
import org.apache.iotdb.common.rpc.thrift.TSStatus;
-import org.apache.iotdb.commons.StepTracker;
import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
import org.apache.iotdb.commons.concurrent.ThreadName;
import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
@@ -1062,7 +1061,6 @@ public class DataRegion {
StorageEngineV2.blockInsertionIfReject(null);
}
writeLock("insertTablet");
- long startTime = System.nanoTime();
try {
TSStatus[] results = new TSStatus[insertTabletNode.getRowCount()];
Arrays.fill(results, RpcUtils.SUCCESS_STATUS);
@@ -1146,7 +1144,6 @@ public class DataRegion {
// TODO: trigger // fire trigger after insertion
// TriggerEngine.fire(TriggerEvent.AFTER_INSERT, insertTabletPlan, firePosition);
} finally {
- StepTracker.trace("insertTablet", startTime, System.nanoTime());
writeUnlock();
}
}
[iotdb] 02/02: Merge branch 'master' into ml_0808_test_exp1_parallel
Posted by xi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xingtanzjr pushed a commit to branch ml_0808_test_exp1_parallel
in repository https://gitbox.apache.org/repos/asf/iotdb.git
commit 8b31ab57055ffd0353c3e54b34ad445a5c5517ae
Merge: 3a743f41ae a04a1eb62b
Author: Jinrui.Zhang <xi...@gmail.com>
AuthorDate: Tue Aug 9 15:50:32 2022 +0800
Merge branch 'master' into ml_0808_test_exp1_parallel
.github/workflows/client-cpp.yml | 4 +-
.github/workflows/influxdb-protocol.yml | 2 +-
.../org/apache/iotdb/db/qp/sql/IoTDBSqlParser.g4 | 45 +-
checkstyle.xml | 7 +
client-cpp/src/main/Session.cpp | 12 +
client-cpp/src/test/cpp/sessionIT.cpp | 28 ++
.../assembly/resources/sbin/remove-confignode.sh | 40 +-
.../assembly/resources/sbin/start-confignode.sh | 36 +-
.../src/assembly/resources/sbin/stop-confignode.sh | 11 +-
.../confignode/client/DataNodeRequestType.java | 4 +-
...ava => AsyncConfigNodeHeartbeatClientPool.java} | 42 +-
.../async/datanode/AsyncDataNodeClientPool.java | 40 +-
.../datanode/AsyncDataNodeHeartbeatClientPool.java | 72 +++
.../async/handlers/ConfigNodeHeartbeatHandler.java | 16 +-
.../async/handlers/DataNodeHeartbeatHandler.java | 10 +-
.../client/async/handlers/MergeHandler.java | 83 ++++
.../sync/confignode/SyncConfigNodeClientPool.java | 22 +-
.../sync/datanode/SyncDataNodeClientPool.java | 4 +-
.../iotdb/confignode/conf/ConfigNodeConfig.java | 41 --
.../confignode/conf/ConfigNodeDescriptor.java | 17 -
.../consensus/request/ConfigPhysicalPlan.java | 4 +
.../consensus/request/ConfigPhysicalPlanType.java | 2 +
.../consensus/request/auth/AuthorPlan.java | 25 +-
.../consensus/request/write/SetTTLPlan.java | 29 +-
.../exception/AddConsensusGroupException.java | 11 +-
.../confignode/manager/ClusterSchemaManager.java | 83 ++--
.../iotdb/confignode/manager/ConfigManager.java | 101 ++--
.../apache/iotdb/confignode/manager/IManager.java | 10 +-
.../iotdb/confignode/manager/NodeManager.java | 135 ++----
.../iotdb/confignode/manager/PartitionManager.java | 21 +-
.../iotdb/confignode/manager/ProcedureManager.java | 29 +-
.../iotdb/confignode/manager/load/LoadManager.java | 350 ++++++++++++--
.../manager/load/balancer/RouteBalancer.java | 60 ++-
.../load/balancer/router/LazyGreedyRouter.java | 154 +++++++
.../load/heartbeat/ConfigNodeHeartbeatCache.java | 2 +-
.../manager/load/heartbeat/IRegionGroupCache.java | 9 +
.../manager/load/heartbeat/RegionGroupCache.java | 7 +-
.../iotdb/confignode/persistence/AuthorInfo.java | 167 ++++---
.../iotdb/confignode/persistence/NodeInfo.java | 48 +-
.../persistence/executor/ConfigPlanExecutor.java | 10 +-
.../persistence/schema/ClusterSchemaInfo.java | 34 +-
.../procedure/env/ConfigNodeProcedureEnv.java | 20 +-
.../procedure/env/DataNodeRemoveHandler.java | 23 +-
.../procedure/impl/RegionMigrateProcedure.java | 12 +-
.../procedure/state/RegionTransitionState.java | 1 -
.../iotdb/confignode/service/ConfigNode.java | 26 +-
.../service/thrift/ConfigNodeRPCService.java | 15 +-
.../thrift/ConfigNodeRPCServiceProcessor.java | 41 +-
.../request/ConfigPhysicalPlanSerDeTest.java | 123 ++++-
.../load/balancer/router/LazyGreedyRouterTest.java | 166 +++++++
.../load/balancer/router/LeaderRouterTest.java | 2 +-
.../confignode/persistence/AuthorInfoTest.java | 344 ++++++++++++--
.../thrift/ConfigNodeRPCServiceProcessorTest.java | 84 ++--
docker/src/main/Dockerfile-single-influxdb | 4 +-
docs/Download/README.md | 28 +-
.../Administration-Management/Administration.md | 97 ++--
.../UserGuide/Ecosystem Integration/Spark IoTDB.md | 2 +-
.../Maintenance-Tools/Maintenance-Command.md | 14 +-
docs/UserGuide/Maintenance-Tools/Metric-Tool.md | 10 +-
docs/UserGuide/Operate-Metadata/TTL.md | 21 +-
docs/UserGuide/Reference/SQL-Reference.md | 4 +-
docs/UserGuide/UDF-Library/Series-Processing.md | 70 +++
docs/UserGuide/Write-And-Delete-Data/CSV-Tool.md | 14 +-
docs/zh/Development/VoteRelease.md | 4 +-
docs/zh/Download/README.md | 28 +-
.../Administration-Management/Administration.md | 97 ++--
.../UserGuide/Ecosystem Integration/Spark IoTDB.md | 2 +-
.../Maintenance-Tools/Maintenance-Command.md | 16 +-
docs/zh/UserGuide/Maintenance-Tools/Metric-Tool.md | 22 +-
docs/zh/UserGuide/Operate-Metadata/TTL.md | 18 +
.../Query-Data/Performance-Tracing-Tool.md | 30 +-
docs/zh/UserGuide/Query-Data/Tracing-Tool.md | 30 +-
docs/zh/UserGuide/QuickStart/WayToGetIoTDB.md | 2 +-
docs/zh/UserGuide/Reference/SQL-Reference.md | 2 +-
docs/zh/UserGuide/UDF-Library/Series-Processing.md | 69 +++
.../zh/UserGuide/Write-And-Delete-Data/CSV-Tool.md | 14 +-
.../Apache IoTDB ConfigNode Dashboard v0.14.0.json | 275 ++++++++++-
.../Apache IoTDB DataNode Dashboard v0.14.0.json | 289 +++++++++++-
grafana-plugin/backend-compile.bat | 31 ++
grafana-plugin/go.mod | 2 +-
grafana-plugin/go.sum | 15 +-
grafana-plugin/pom.xml | 54 ++-
integration-test/import-control.xml | 8 +
.../java/org/apache/iotdb/it/env/AbstractEnv.java | 115 ++---
.../org/apache/iotdb/it/env/ConfigFactory.java | 1 +
.../org/apache/iotdb/it/env/ConfigNodeWrapper.java | 4 +
.../org/apache/iotdb/it/env/DataNodeWrapper.java | 16 +
.../java/org/apache/iotdb/it/env/EnvUtils.java | 90 ++++
.../java/org/apache/iotdb/it/env/MppConfig.java | 22 +
.../org/apache/iotdb/it/env/RemoteServerEnv.java | 28 ++
.../apache/iotdb/it/env/StandaloneOnMppEnv.java | 5 +-
.../BuiltinTimeSeriesGeneratingFunctionEnum.java | 1 +
.../org/apache/iotdb/itbase/env/BaseConfig.java | 25 +
.../java/org/apache/iotdb/itbase/env/BaseEnv.java | 15 +
.../org/apache/iotdb/db/it/IoTDBConfigNodeIT.java | 209 +++++++++
.../apache/iotdb/db/it/IoTDBFlushQueryMergeIT.java | 193 ++++++++
.../java/org/apache/iotdb/db/it/IoTDBTtlIT.java | 269 +++++++++++
.../IoTDBAlignedSeriesQueryWithDeletionIT.java | 139 ++++++
.../db/it/aligned/IoTDBInsertAlignedValuesIT.java | 12 +
.../org/apache/iotdb/db/it/env/StandaloneEnv.java | 30 ++
.../db/it/udf/IoTDBUDTFBuiltinFunctionIT.java | 89 +++-
.../iotdb/db/integration/IoTDBAuthorizationIT.java | 20 +-
.../apache/iotdb/db/integration/IoTDBTtlIT.java | 130 ++++++
library-udf/pom.xml | 6 -
.../micrometer/MicrometerMetricManagerTest.java | 11 +
.../apache/iotdb/commons/client/ClientManager.java | 13 +-
.../iotdb/commons/client/ClientPoolFactory.java | 56 ++-
.../iotdb/commons/client/IClientManager.java | 5 +
.../AsyncConfigNodeHeartbeatServiceClient.java | 140 ++++++
.../async/AsyncDataNodeHeartbeatServiceClient.java | 140 ++++++
.../apache/iotdb/commons/cluster/NodeStatus.java | 4 +
.../apache/iotdb/commons/conf/CommonConfig.java | 40 ++
.../iotdb/commons/conf/CommonDescriptor.java | 17 +
.../iotdb/commons/partition/DataPartition.java | 1 +
.../BuiltinTimeSeriesGeneratingFunction.java | 1 +
.../commons/udf/builtin/UDTFDeDuplication.java | 139 ++++++
.../commons/utils/BasicStructureSerDeUtil.java | 31 ++
pom.xml | 8 +-
.../resources/conf/iotdb-datanode.properties | 4 +-
.../iotdb/db/auth/ClusterAuthorityFetcher.java | 9 +-
.../apache/iotdb/db/client/ConfigNodeClient.java | 28 +-
.../java/org/apache/iotdb/db/conf/IoTDBConfig.java | 6 +-
.../org/apache/iotdb/db/conf/IoTDBDescriptor.java | 23 +-
.../apache/iotdb/db/engine/StorageEngineV2.java | 30 +-
.../db/engine/compaction/CompactionWorker.java | 34 +-
.../RewriteCrossSpaceCompactionSelector.java | 7 -
.../impl/ReadPointCompactionPerformer.java | 18 +-
.../writer/AbstractCompactionWriter.java | 7 +-
.../writer/CrossSpaceCompactionWriter.java | 19 +-
.../writer/InnerSpaceCompactionWriter.java | 15 +-
.../iotdb/db/localconfignode/LocalConfigNode.java | 173 ++++---
.../db/metadata/cache/DataNodeSchemaCache.java | 9 +-
.../{SchemaTree.java => ClusterSchemaTree.java} | 22 +-
.../db/mpp/common/schematree/ISchemaTree.java | 70 +++
.../iotdb/db/mpp/execution/driver/Driver.java | 3 +-
.../db/mpp/execution/exchange/ISinkHandle.java | 2 +-
.../db/mpp/execution/exchange/LocalSinkHandle.java | 10 +-
.../db/mpp/execution/exchange/SinkHandle.java | 52 ++-
.../iotdb/db/mpp/execution/memory/MemoryPool.java | 2 +-
.../operator/process/DeviceMergeOperator.java | 2 +-
.../operator/process/FilterAndProjectOperator.java | 2 +-
.../RowBasedTimeJoinOperator.java} | 62 +--
.../process/{ => join}/TimeJoinOperator.java | 7 +-
.../{ => join}/merge/AscTimeComparator.java | 2 +-
.../process/{ => join}/merge/ColumnMerger.java | 24 +-
.../{ => join}/merge/DescTimeComparator.java | 2 +-
.../{ => join}/merge/MultiColumnMerger.java | 56 ++-
.../merge/NonOverlappedMultiColumnMerger.java | 25 +-
.../{ => join}/merge/SingleColumnMerger.java | 44 +-
.../process/{ => join}/merge/TimeComparator.java | 2 +-
.../operator/schema/SchemaFetchMergeOperator.java | 4 +-
.../operator/schema/SchemaFetchScanOperator.java | 8 +-
.../schema/TimeSeriesSchemaScanOperator.java | 25 +-
.../org/apache/iotdb/db/mpp/plan/Coordinator.java | 5 +-
.../apache/iotdb/db/mpp/plan/analyze/Analysis.java | 8 +-
.../iotdb/db/mpp/plan/analyze/AnalyzeVisitor.java | 51 +-
.../db/mpp/plan/analyze/ClusterSchemaFetcher.java | 41 +-
.../db/mpp/plan/analyze/ExpressionAnalyzer.java | 10 +-
.../db/mpp/plan/analyze/FakeSchemaFetcherImpl.java | 11 +-
.../iotdb/db/mpp/plan/analyze/ISchemaFetcher.java | 8 +-
.../iotdb/db/mpp/plan/analyze/SchemaValidator.java | 6 +-
.../mpp/plan/analyze/StandaloneSchemaFetcher.java | 33 +-
.../db/mpp/plan/execution/QueryExecution.java | 3 +-
.../mpp/plan/execution/config/ConfigExecution.java | 6 +-
.../plan/execution/config/ConfigTaskVisitor.java | 29 ++
.../config/executor/ClusterConfigTaskExecutor.java | 52 ++-
.../config/executor/IConfigTaskExecutor.java | 5 +-
.../executor/StandaloneConfigTaskExecutor.java | 36 +-
.../{ => metadata}/CountStorageGroupTask.java | 4 +-
.../config/{ => metadata}/CreateFunctionTask.java | 4 +-
.../{ => metadata}/DeleteStorageGroupTask.java | 4 +-
.../config/{ => metadata}/DropFunctionTask.java | 4 +-
.../config/{ => metadata}/SetStorageGroupTask.java | 4 +-
.../config/{ => metadata}/SetTTLTask.java | 4 +-
.../config/{ => metadata}/ShowClusterTask.java | 4 +-
.../config/{ => metadata}/ShowConfigNodesTask.java | 4 +-
.../config/{ => metadata}/ShowDataNodesTask.java | 4 +-
.../config/{ => metadata}/ShowFunctionsTask.java | 4 +-
.../config/{ => metadata}/ShowRegionTask.java | 4 +-
.../{ => metadata}/ShowStorageGroupTask.java | 4 +-
.../config/{ => metadata}/ShowTTLTask.java | 4 +-
.../config/{ => metadata}/UnSetTTLTask.java | 2 +-
.../template}/CreateSchemaTemplateTask.java | 4 +-
.../template}/SetSchemaTemplateTask.java | 4 +-
.../template}/ShowNodesInSchemaTemplateTask.java | 4 +-
.../template}/ShowPathSetTemplateTask.java | 4 +-
.../template}/ShowSchemaTemplateTask.java | 4 +-
.../execution/config/{ => sys}/AuthorizerTask.java | 4 +-
.../execution/config/{ => sys}/ClearCacheTask.java | 6 +-
.../plan/execution/config/{ => sys}/FlushTask.java | 4 +-
.../{ClearCacheTask.java => sys/MergeTask.java} | 29 +-
.../iotdb/db/mpp/plan/expression/Expression.java | 144 +++---
.../plan/expression/binary/AdditionExpression.java | 12 -
.../plan/expression/binary/BinaryExpression.java | 74 +--
.../plan/expression/binary/DivisionExpression.java | 12 -
.../plan/expression/binary/EqualToExpression.java | 11 -
.../expression/binary/GreaterEqualExpression.java | 12 -
.../expression/binary/GreaterThanExpression.java | 12 -
.../expression/binary/LessEqualExpression.java | 12 -
.../plan/expression/binary/LessThanExpression.java | 12 -
.../plan/expression/binary/LogicAndExpression.java | 11 -
.../plan/expression/binary/LogicOrExpression.java | 11 -
.../plan/expression/binary/ModuloExpression.java | 12 -
.../binary/MultiplicationExpression.java | 12 -
.../plan/expression/binary/NonEqualExpression.java | 12 -
.../expression/binary/SubtractionExpression.java | 12 -
.../mpp/plan/expression/leaf/ConstantOperand.java | 37 +-
.../plan/expression/leaf/TimeSeriesOperand.java | 36 +-
.../mpp/plan/expression/leaf/TimestampOperand.java | 35 +-
.../plan/expression/multi/FunctionExpression.java | 93 +---
.../plan/expression/ternary/BetweenExpression.java | 18 -
.../plan/expression/ternary/TernaryExpression.java | 91 +---
.../db/mpp/plan/expression/unary/InExpression.java | 9 -
.../plan/expression/unary/IsNullExpression.java | 9 -
.../mpp/plan/expression/unary/LikeExpression.java | 9 -
.../plan/expression/unary/LogicNotExpression.java | 9 -
.../plan/expression/unary/NegationExpression.java | 9 -
.../plan/expression/unary/RegularExpression.java | 9 -
.../mpp/plan/expression/unary/UnaryExpression.java | 61 +--
.../visitor/ColumnTransformerVisitor.java | 478 +++++++++++++++++++
.../plan/expression/visitor/ExpressionVisitor.java | 73 +++
.../iotdb/db/mpp/plan/parser/ASTVisitor.java | 125 ++---
.../db/mpp/plan/planner/LogicalPlanVisitor.java | 9 -
.../db/mpp/plan/planner/OperatorTreeGenerator.java | 52 ++-
.../mpp/plan/planner/plan/node/PlanNodeType.java | 4 +
.../planner/plan/node/write/DeleteDataNode.java | 6 +-
.../plan/node/write/InsertMultiTabletsNode.java | 7 +-
.../plan/planner/plan/node/write/InsertNode.java | 4 +-
.../planner/plan/node/write/InsertRowNode.java | 20 +-
.../planner/plan/node/write/InsertRowsNode.java | 7 +-
.../plan/node/write/InsertRowsOfOneDeviceNode.java | 7 +-
.../planner/plan/node/write/InsertTabletNode.java | 6 +-
.../scheduler/AbstractFragInsStateTracker.java | 28 +-
.../db/mpp/plan/scheduler/ClusterScheduler.java | 3 +-
.../scheduler/FixedRateFragInsStateTracker.java | 14 +-
.../mpp/plan/scheduler/SimpleQueryTerminator.java | 4 -
.../db/mpp/plan/scheduler/StandaloneScheduler.java | 28 +-
.../db/mpp/plan/statement/StatementVisitor.java | 10 +-
.../db/mpp/plan/statement/sys/AuthorStatement.java | 20 +-
.../MergeStatement.java} | 36 +-
.../iotdb/db/protocol/influxdb/dto/IoTDBPoint.java | 40 +-
.../function/aggregator/InfluxMeanFunction.java | 4 +-
.../function/selector/InfluxFirstFunction.java | 5 +-
.../function/selector/InfluxLastFunction.java | 5 +-
.../influxdb/handler/AbstractQueryHandler.java | 511 +++++++++++++++++++++
.../protocol/influxdb/handler/NewQueryHandler.java | 200 ++++++++
.../db/protocol/influxdb/handler/QueryHandler.java | 468 +------------------
.../influxdb/meta/AbstractInfluxDBMetaManager.java | 114 +++++
.../influxdb/meta/InfluxDBMetaManager.java | 96 +---
.../influxdb/meta/NewInfluxDBMetaManager.java | 129 ++++++
.../db/protocol/influxdb/meta/TagInfoRecords.java | 28 +-
.../influxdb/util/InfluxReqAndRespUtils.java | 55 +++
.../protocol/influxdb/util/QueryResultUtils.java | 153 ++++++
.../db/protocol/influxdb/util/StringUtils.java | 11 +
.../handler/PhysicalPlanValidationHandler.java | 4 +-
.../apache/iotdb/db/qp/executor/PlanExecutor.java | 92 +++-
.../iotdb/db/qp/logical/sys/AuthorOperator.java | 26 +-
.../iotdb/db/qp/physical/sys/AuthorPlan.java | 62 +--
.../apache/iotdb/db/qp/sql/IoTDBSqlVisitor.java | 101 ++--
.../iotdb/db/service/InfluxDBRPCService.java | 26 +-
.../java/org/apache/iotdb/db/service/NewIoTDB.java | 3 -
.../iotdb/db/service/RegionMigrateService.java | 25 +-
.../recorder/CompactionMetricsRecorder.java | 2 +-
.../handler/InfluxDBServiceThriftHandler.java | 10 +-
.../impl/DataNodeInternalRPCServiceImpl.java | 26 +-
.../thrift/impl/IInfluxDBServiceWithHandler.java | 11 +-
.../service/thrift/impl/InfluxDBServiceImpl.java | 20 +-
.../thrift/impl/NewInfluxDBServiceImpl.java | 128 ++++++
.../iotdb/db/wal/recover/WALNodeRecoverTask.java | 3 +-
.../iotdb/db/wal/recover/WALRecoverWriter.java | 26 +-
.../db/metadata/cache/DataNodeSchemaCacheTest.java | 11 +-
...emaTreeTest.java => ClusterSchemaTreeTest.java} | 28 +-
.../db/mpp/execution/ConfigExecutionTest.java | 18 +-
.../iotdb/db/mpp/execution/DataDriverTest.java | 6 +-
.../execution/exchange/LocalSinkHandleTest.java | 6 +-
.../db/mpp/execution/exchange/SinkHandleTest.java | 70 +--
.../db/mpp/execution/exchange/StubSinkHandle.java | 4 +-
.../iotdb/db/mpp/execution/exchange/Utils.java | 11 +-
.../operator/AlignedSeriesScanOperatorTest.java | 8 +-
.../operator/DeviceMergeOperatorTest.java | 2 +-
.../mpp/execution/operator/LimitOperatorTest.java | 6 +-
.../execution/operator/MultiColumnMergerTest.java | 2 +-
.../NonOverlappedMultiColumnMergerTest.java | 4 +-
.../mpp/execution/operator/OffsetOperatorTest.java | 6 +-
.../operator/RawDataAggregationOperatorTest.java | 6 +-
.../execution/operator/SingleColumnMergerTest.java | 6 +-
.../execution/operator/TimeJoinOperatorTest.java | 8 +-
.../schema/SchemaFetchScanOperatorTest.java | 5 +-
.../write/InsertMultiTabletsNodeSerdeTest.java | 2 +-
.../plan/node/write/InsertRowNodeSerdeTest.java | 8 +-
.../plan/node/write/InsertRowsNodeSerdeTest.java | 2 +-
.../write/InsertRowsOfOneDeviceNodeSerdeTest.java | 3 +-
.../plan/node/write/InsertTabletNodeSerdeTest.java | 6 +-
.../plan/scheduler/StandaloneSchedulerTest.java | 5 -
.../java/org/apache/iotdb/db/qp/PlannerTest.java | 13 +-
.../iotdb/db/qp/physical/PhysicalPlanTest.java | 2 +-
.../iotdb/db/wal/recover/WALRecoverWriterTest.java | 198 ++++++++
.../org/apache/iotdb/session/pool/SessionPool.java | 3 +
thrift-commons/src/main/thrift/common.thrift | 14 +-
.../src/main/thrift/confignode.thrift | 229 +++++++--
thrift/src/main/thrift/datanode.thrift | 10 +-
.../file/metadata/statistics/Statistics.java | 10 +
.../file/metadata/statistics/TimeStatistics.java | 8 +
.../iotdb/tsfile/read/common/block/TsBlock.java | 4 +
.../tsfile/read/common/block/TsBlockBuilder.java | 16 +
.../block/column/BinaryArrayColumnEncoder.java | 26 +-
.../read/common/block/column/BinaryColumn.java | 16 +
.../read/common/block/column/BooleanColumn.java | 16 +
.../block/column/ByteArrayColumnEncoder.java | 19 +-
.../tsfile/read/common/block/column/Column.java | 38 ++
.../read/common/block/column/DoubleColumn.java | 16 +
.../read/common/block/column/FloatColumn.java | 16 +
.../block/column/Int32ArrayColumnEncoder.java | 37 +-
.../block/column/Int64ArrayColumnEncoder.java | 64 ++-
.../tsfile/read/common/block/column/IntColumn.java | 16 +
.../read/common/block/column/LongColumn.java | 16 +
.../block/column/RunLengthEncodedColumn.java | 58 +++
.../read/common/block/column/TimeColumn.java | 11 +
.../tsfile/read/reader/page/AlignedPageReader.java | 98 +++-
.../tsfile/read/reader/page/ValuePageReader.java | 99 ++++
.../tsfile/write/chunk/AlignedChunkWriterImpl.java | 63 +++
.../iotdb/tsfile/write/chunk/TimeChunkWriter.java | 10 +-
.../iotdb/tsfile/write/chunk/ValueChunkWriter.java | 24 +-
.../iotdb/tsfile/write/page/TimePageWriter.java | 6 +-
.../iotdb/tsfile/write/page/ValuePageWriter.java | 72 ++-
.../tsfile/common/block/TsBlockSerdeTest.java | 139 ++++--
326 files changed, 9437 insertions(+), 3494 deletions(-)