You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2024/03/07 08:19:33 UTC
(doris) branch branch-2.1 updated (2913a111f60 -> 677c9348a7d)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
from 2913a111f60 [Bug](top-n) init query_ctx runtime predicate before _build_pipelines #31895
new f5991d8b029 [docs](jdbc catalog) add sqlserver timestamp type mapping doc (#31832)
new 2e94512e075 [fix](build)Fix index_tool build error (#31834)
new 538032a75c6 [fix](partition) add log when tablet partition id eq 0 (#31796)
new 28f0b7eb321 [Improvement](profile)Add tvf active_be_tasks() #31815
new ff8c679851d [fix](group commit) group commit may heap-use-after-free if execute plan failed (#31839)
new f897db666c2 [docs](flink-connector) Fix Flink Doris Connector typo (#31830)
new 2e4201e4066 [opt](session variable) max_msg_size_of_result_receiver #31809
new d0847a89531 [feat](Nereids): add struct info map in group (#31800)
new b978365a7b0 [fix](cloud) Fix FE cannot work when restarting after schema change (#31842)
new c7d86c062ad [fix](Nereids) json_object can't be odd parameters, need even parameters (#31767)
new dc9de4b6b56 [Bug](load) fix wrong data in mv when routine load with function mapping (#31787)
new 686cbffd8a7 [fix](multi-catalog) fix iceberg defalut credentials proviers (#31704)
new 5c1c73e5bb4 [Fix](Nereids) fix missing comment when creating table (#31869)
new 3716f8a171f [Bug](partition) fix npe when prune partition with not exist partition column in mv #31860
new 1d9e9fc8843 [regression test] Test the unique model by modify a value type from TINYINT to other type (#31682)
new d276610694e [deps](docker) update docker build env to add jdk17 (#31644)
new ad2f7fc3165 [bugfix](coordinator) should use fragment id not profile fragment id to cancel fragment (#31852)
new 561709451c5 [fix](Nereids) fix group_concat(distinct) failed (#31873)
new 80c80b67196 [enhancement](Nereids): support more condition Date/DateTime Literal (#31858)
new 54a8c83f6a1 [enhancement](cloud) add CloudTabletStatMgr to capture stats in cloud mode (#31818)
new b3db6b2eae1 [improvement](jdbc catalog) opt get db2 schema list & xml type mapping (#31856)
new 8084a0a74ae [typo](doc) Add a full backup case and modify sidebars (#31801)
new 326196b93dd [Bug](fix) fix or and "<=>" cause coredump in query (#31884)
new 0baa00af159 [type](doc) Standardize the name in the sql-dialect document and modify the format (#31636)
new c83f600c05f [enhancement](nereids) allow reorder mark join (#30644)
new 4d259377f31 [enhancement](nereids) show nullaware semi join in plan (#31738)
new 6d0c1ad6328 [fix](http stream) http stream support memtable_on_sink_node header (#31866)
new 73ebeb24a3c [fix](mtmv) support null value in partition for updating (#31843)
new 13766665149 [fix](money_format) fix money_format #31883
new c0b25869a9e [fix](function) fix date_format function execution error on fe (#31645)
new 286685ff2fe [doc](apache-superset)add doc for apache superset compatible (#31314)
new 2f012b834d9 [bug](meta) exit when get RollbackException in observer (#31687)
new 44f41054d8c [cloud](point query) enable short circuit query in cloud (#31897)
new 586f155d84e [enhance](mtmv) MTMV Use partial partition of base table (#31632)
new 917b2011747 [chore](build) Using multithread to accelerate FE compilation (#31855)
new db65114f77d [Fix](Nereids) fix hint cases with random result (#31865)
new 73f6d9532dc [feat](nereids) support null safe eq runtime filter (FE part) (#31655)
new 677c9348a7d [fix] (doc)Fix invalid link on benchmark tool page #31920
The 38 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/http/action/http_stream.cpp | 4 +
be/src/index-tools/index_tool.cpp | 2 +-
be/src/olap/data_dir.cpp | 4 +-
be/src/pipeline/exec/scan_operator.cpp | 2 +-
be/src/runtime/group_commit_mgr.cpp | 20 +-
be/src/runtime/group_commit_mgr.h | 3 +-
be/src/runtime/runtime_query_statistics_mgr.cpp | 50 ++++
be/src/runtime/runtime_query_statistics_mgr.h | 5 +
be/src/vec/exec/scan/vmeta_scanner.cpp | 22 +-
be/src/vec/exec/scan/vmeta_scanner.h | 2 +
be/src/vec/exec/scan/vscan_node.cpp | 2 +-
be/src/vec/functions/function.cpp | 2 +-
be/src/vec/functions/function_string.h | 30 ++-
build.sh | 8 +-
docker/compilation/Dockerfile | 21 +-
.../en/community/developer-guide/benchmark-tool.md | 2 +-
.../ecosystem/bi-integration/apache-superset.md | 53 ++++
.../install/source-install/compilation-general.md | 33 ++-
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 41 ++--
docs/en/docs/lakehouse/sql-dialect.md | 44 ++--
.../Backup-and-Restore/BACKUP.md | 7 +
docs/sidebars.json | 2 +-
.../community/developer-guide/benchmark-tool.md | 2 +-
.../ecosystem/bi-integration/apache-superset.md | 57 +++++
docs/zh-CN/docs/ecosystem/flink-doris-connector.md | 2 +-
.../install/source-install/compilation-general.md | 12 +-
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 6 +-
docs/zh-CN/docs/lakehouse/sql-dialect.md | 46 ++--
.../Backup-and-Restore/BACKUP.md | 7 +
.../main/java/org/apache/doris/common/Config.java | 3 +
.../org/apache/doris/analysis/DateLiteral.java | 8 +-
.../doris/catalog/BuiltinTableValuedFunctions.java | 4 +-
.../apache/doris/catalog/CloudTabletStatMgr.java | 273 +++++++++++++++++++++
.../java/org/apache/doris/catalog/Database.java | 5 +
.../main/java/org/apache/doris/catalog/Env.java | 9 +-
.../apache/doris/catalog/ListPartitionItem.java | 21 ++
.../main/java/org/apache/doris/catalog/MTMV.java | 11 +-
.../org/apache/doris/catalog/PartitionItem.java | 14 ++
.../apache/doris/catalog/RangePartitionItem.java | 18 ++
.../java/org/apache/doris/catalog/Replica.java | 17 ++
.../apache/doris/common/util/PropertyAnalyzer.java | 3 +
.../apache/doris/datasource/InternalCatalog.java | 2 +-
.../datasource/iceberg/IcebergExternalCatalog.java | 9 +
.../iceberg/IcebergGlueExternalCatalog.java | 5 +-
.../iceberg/IcebergHMSExternalCatalog.java | 1 +
.../iceberg/IcebergHadoopExternalCatalog.java | 1 +
.../iceberg/IcebergRestExternalCatalog.java | 3 +-
.../datasource/jdbc/client/JdbcDB2Client.java | 36 +++
.../datasource/jdbc/client/JdbcOracleClient.java | 6 +-
.../datasource/property/PropertyConverter.java | 16 +-
.../property/constants/S3Properties.java | 8 +
.../apache/doris/journal/bdbje/BDBJEJournal.java | 7 +
.../src/main/java/org/apache/doris/load/Load.java | 10 +-
.../apache/doris/mtmv/MTMVPartitionSyncConfig.java | 57 +++++
.../doris/mtmv/MTMVPartitionSyncTimeUnit.java | 36 +++
.../org/apache/doris/mtmv/MTMVPartitionUtil.java | 11 +-
.../org/apache/doris/mtmv/MTMVPropertyUtil.java | 138 +++++++++++
.../org/apache/doris/mtmv/MTMVRelatedTableIf.java | 31 ++-
.../org/apache/doris/mtmv/MTMVRelationManager.java | 10 +-
.../main/java/org/apache/doris/mtmv/MTMVUtil.java | 127 +++++++++-
.../glue/translator/PhysicalPlanTranslator.java | 14 +-
.../java/org/apache/doris/nereids/memo/Group.java | 7 +-
.../apache/doris/nereids/memo/StructInfoMap.java | 123 ++++++++++
.../doris/nereids/parser/LogicalPlanBuilder.java | 6 +-
.../processor/post/RuntimeFilterGenerator.java | 34 ++-
.../rules/analysis/CollectJoinConstraint.java | 2 +-
.../rules/exploration/join/InnerJoinLAsscom.java | 7 +-
.../exploration/join/InnerJoinLAsscomProject.java | 1 -
.../exploration/join/InnerJoinLeftAssociate.java | 1 -
.../join/InnerJoinLeftAssociateProject.java | 1 -
.../exploration/join/InnerJoinRightAssociate.java | 1 -
.../join/InnerJoinRightAssociateProject.java | 1 -
.../rules/exploration/join/JoinCommute.java | 7 +-
.../rules/exploration/join/JoinExchange.java | 1 -
.../exploration/join/JoinExchangeBothProject.java | 1 -
.../exploration/join/JoinExchangeLeftProject.java | 1 -
.../exploration/join/JoinExchangeRightProject.java | 1 -
.../join/LogicalJoinSemiJoinTranspose.java | 8 +-
.../join/LogicalJoinSemiJoinTransposeProject.java | 7 +-
.../rules/exploration/join/OuterJoinAssoc.java | 1 -
.../exploration/join/OuterJoinAssocProject.java | 1 -
.../rules/exploration/join/OuterJoinLAsscom.java | 1 -
.../exploration/join/OuterJoinLAsscomProject.java | 1 -
.../join/PushDownProjectThroughInnerOuterJoin.java | 4 +-
.../join/SemiJoinSemiJoinTranspose.java | 1 -
.../join/SemiJoinSemiJoinTransposeProject.java | 11 +-
.../mv/InitMaterializationContextHook.java | 7 +-
.../rules/expression/ExpressionOptimization.java | 4 +-
.../expression/rules/NullSafeEqualToEqual.java | 62 +++++
.../rules/implementation/AggregateStrategies.java | 33 +--
.../rules/rewrite/FindHashConditionForJoin.java | 3 +-
.../rules/rewrite/PruneOlapScanPartition.java | 15 +-
.../rules/rewrite/PushFilterInsideJoin.java | 11 +-
.../nereids/rules/rewrite/SemiJoinCommute.java | 1 -
.../rewrite/TransposeSemiJoinLogicalJoin.java | 1 -
.../TransposeSemiJoinLogicalJoinProject.java | 10 +-
.../functions/agg/AggregateFunction.java | 3 +
.../expressions/functions/agg/GroupConcat.java | 9 +
.../expressions/functions/scalar/JsonObject.java | 3 +
.../{ActiveQueries.java => ActiveBeTasks.java} | 17 +-
.../expressions/functions/table/ActiveQueries.java | 2 +-
.../trees/expressions/literal/DateLiteral.java | 46 ++--
.../visitor/TableValuedFunctionVisitor.java | 5 +
.../nereids/trees/plans/algebra/Aggregate.java | 2 +-
.../plans/commands/UpdateMvByPartitionCommand.java | 19 +-
.../plans/commands/info/AlterMTMVPropertyInfo.java | 39 +--
.../trees/plans/commands/info/CreateMTMVInfo.java | 49 +---
.../nereids/trees/plans/logical/LogicalJoin.java | 14 +-
.../trees/plans/physical/AbstractPhysicalJoin.java | 8 +-
.../trees/plans/physical/AbstractPhysicalPlan.java | 10 +
.../plans/physical/PhysicalNestedLoopJoin.java | 7 +-
.../org/apache/doris/nereids/util/DateUtils.java | 8 +-
.../org/apache/doris/nereids/util/JoinUtils.java | 10 +
.../org/apache/doris/planner/HashJoinNode.java | 25 +-
.../org/apache/doris/planner/RuntimeFilter.java | 9 +
.../main/java/org/apache/doris/qe/Coordinator.java | 77 +-----
.../java/org/apache/doris/qe/PointQueryExec.java | 19 +-
.../java/org/apache/doris/qe/ResultReceiver.java | 22 +-
.../java/org/apache/doris/qe/SessionVariable.java | 14 ++
.../java/org/apache/doris/qe/StmtExecutor.java | 3 +-
.../org/apache/doris/rpc/BackendServiceProxy.java | 6 +-
.../apache/doris/rpc/TCustomProtocolFactory.java | 41 ++++
.../apache/doris/service/FrontendServiceImpl.java | 22 +-
....java => ActiveBeTasksTableValuedFunction.java} | 50 +---
.../ActiveQueriesTableValuedFunction.java | 4 +-
.../doris/tablefunction/TableValuedFunctionIf.java | 2 +
.../java/org/apache/doris/mtmv/MTMVUtilTest.java | 113 +++++++++
.../doris/nereids/memo/StructInfoMapTest.java | 81 ++++++
.../nereids/rules/expression/FoldConstantTest.java | 2 +-
.../expression/rules/NullSafeEqualToEqualTest.java | 69 ++++++
.../rules/rewrite/EliminateJoinByUniqueTest.java | 14 --
.../expressions/literal/DateTimeLiteralTest.java | 7 +
.../commands/UpdateMvByPartitionCommandTest.java | 21 ++
.../org/apache/doris/nereids/util/PlanChecker.java | 2 +
.../org/apache/doris/rewrite/FEFunctionsTest.java | 2 +-
.../apache/doris/utframe/TestWithFeService.java | 28 +++
gensrc/thrift/Types.thrift | 3 +-
.../data/correctness_p0/test_equal_for_null.out | 2 +
.../jdbc/test_db2_jdbc_catalog.out | 20 ++
.../mtmv_p0/test_hive_limit_partition_mtmv.out | 11 +
.../data/mtmv_p0/test_limit_partition_mtmv.out | 17 ++
.../routine_load_mapping/routine_load_mapping.out | 9 +
.../data/mv_p0/routine_load_mapping/test | 1 +
.../test_mv_partition/test_mv_partition.out} | 3 +-
.../data/nereids_p0/hint/test_distribute.out | 2 +-
.../data/nereids_p0/hint/test_leading.out | 146 +----------
.../datetime_functions/test_date_function.out | 6 +
.../string_functions/test_string_function.out | 24 ++
.../group_concat.out} | 7 +-
.../shape/query10.out | 62 ++---
.../shape/query35.out | 72 +++---
.../shape/query45.out | 48 ++--
.../noStatsRfPrune/query10.out | 40 +--
.../noStatsRfPrune/query35.out | 42 ++--
.../noStatsRfPrune/query45.out | 45 ++--
.../no_stats_shape/query10.out | 40 +--
.../no_stats_shape/query35.out | 42 ++--
.../no_stats_shape/query45.out | 45 ++--
.../rf_prune/query10.out | 54 ++--
.../rf_prune/query35.out | 70 +++---
.../rf_prune/query45.out | 48 ++--
.../nereids_tpcds_shape_sf100_p0/shape/query10.out | 54 ++--
.../nereids_tpcds_shape_sf100_p0/shape/query35.out | 72 +++---
.../nereids_tpcds_shape_sf100_p0/shape/query45.out | 48 ++--
.../datetime_functions/test_date_function.out | 6 +
.../test_unique_model_schema_value_change.out | 80 ++++++
.../correctness_p0/test_equal_for_null.groovy | 3 +-
.../jdbc/test_db2_jdbc_catalog.groovy | 24 ++
.../mtmv_p0/test_hive_limit_partition_mtmv.groovy | 162 ++++++++++++
.../mtmv_p0/test_limit_partition_mtmv.groovy | 240 ++++++++++++++++++
.../routine_load_mapping.groovy | 52 ++++
.../test_mv_partition/test_mv_partition.groovy | 64 +++++
.../suites/nereids_p0/hint/test_distribute.groovy | 2 +-
.../suites/nereids_p0/hint/test_leading.groovy | 14 +-
.../datetime_functions/test_date_function.groovy | 2 +
.../string_functions/test_string_function.groovy | 8 +
.../suites/nereids_syntax_p0/group_concat.groovy | 45 +++-
.../datetime_functions/test_date_function.groovy | 2 +
.../test_unique_model_schema_value_change.groovy | 155 ++++++++++++
.../max_msg_size_of_result_receiver.groovy | 61 +++++
180 files changed, 3516 insertions(+), 1103 deletions(-)
create mode 100644 docs/en/docs/ecosystem/bi-integration/apache-superset.md
create mode 100644 docs/zh-CN/docs/ecosystem/bi-integration/apache-superset.md
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionSyncConfig.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionSyncTimeUnit.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqual.java
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/{ActiveQueries.java => ActiveBeTasks.java} (77%)
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/rpc/TCustomProtocolFactory.java
copy fe/fe-core/src/main/java/org/apache/doris/tablefunction/{ActiveQueriesTableValuedFunction.java => ActiveBeTasksTableValuedFunction.java} (53%)
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVUtilTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/memo/StructInfoMapTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqualTest.java
create mode 100644 regression-test/data/mtmv_p0/test_hive_limit_partition_mtmv.out
create mode 100644 regression-test/data/mtmv_p0/test_limit_partition_mtmv.out
create mode 100644 regression-test/data/mv_p0/routine_load_mapping/routine_load_mapping.out
create mode 100644 regression-test/data/mv_p0/routine_load_mapping/test
copy regression-test/data/{correctness_p0/test_equal_for_null.out => mv_p0/test_mv_partition/test_mv_partition.out} (76%)
copy regression-test/data/{correctness_p0/test_equal_for_null.out => nereids_syntax_p0/group_concat.out} (67%)
create mode 100644 regression-test/suites/mtmv_p0/test_hive_limit_partition_mtmv.groovy
create mode 100644 regression-test/suites/mtmv_p0/test_limit_partition_mtmv.groovy
create mode 100644 regression-test/suites/mv_p0/routine_load_mapping/routine_load_mapping.groovy
create mode 100644 regression-test/suites/mv_p0/test_mv_partition/test_mv_partition.groovy
create mode 100644 regression-test/suites/variable_p0/max_msg_size_of_result_receiver.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 16/38: [deps](docker) update docker build env to add jdk17 (#31644)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d276610694eccf31ac7711db942d7a2b6842674b
Author: Mingyu Chen <mo...@163.com>
AuthorDate: Wed Mar 6 21:12:25 2024 +0800
[deps](docker) update docker build env to add jdk17 (#31644)
---
docker/compilation/Dockerfile | 21 +++++++++-----
.../install/source-install/compilation-general.md | 33 ++++++++++------------
.../install/source-install/compilation-general.md | 12 ++++----
3 files changed, 34 insertions(+), 32 deletions(-)
diff --git a/docker/compilation/Dockerfile b/docker/compilation/Dockerfile
index 2dcab8d2ee0..d24c1e72929 100644
--- a/docker/compilation/Dockerfile
+++ b/docker/compilation/Dockerfile
@@ -22,25 +22,32 @@ RUN yum install epel-release -y && yum clean all && yum makecache
# install dependencies
RUN yum install -y byacc patch automake libtool perf vim make which file ncurses-devel gettext-devel unzip bzip2 zip util-linux \
- wget git python2 bison java-11-openjdk-devel java-1.8.0-openjdk-devel
+ wget git python2 bison java-1.8.0-openjdk-devel
# clean cache
RUN yum clean all
# install maven 3.6.3
-RUN mkdir -p /usr/share/maven /usr/share/maven/ref && \
- wget -q -O /tmp/apache-maven.tar.gz https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/apache-maven-3.6.3-bin.tar.gz && \
- tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 && \
- rm -f /tmp/apache-maven.tar.gz && \
- ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
+RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
+ && wget -q -O /tmp/apache-maven.tar.gz https://doris-thirdparty-1308700295.cos.ap-beijing.myqcloud.com/tools/apache-maven-3.6.3-bin.tar.gz \
+ && tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
+ && rm -f /tmp/apache-maven.tar.gz \
+ && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
# install nodejs
-RUN wget https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/node-v12.13.0-linux-x64.tar.gz \
+RUN wget https://doris-thirdparty-1308700295.cos.ap-beijing.myqcloud.com/tools/node-v12.13.0-linux-x64.tar.gz \
-q -O /tmp/node-v12.13.0-linux-x64.tar.gz \
&& cd /tmp/ && tar -xf node-v12.13.0-linux-x64.tar.gz \
&& cp -r node-v12.13.0-linux-x64/* /usr/local/ \
&& rm /tmp/node-v12.13.0-linux-x64.tar.gz && rm -rf node-v12.13.0-linux-x64
+# install jdk17
+RUN wget https://doris-thirdparty-1308700295.cos.ap-beijing.myqcloud.com/tools/openjdk-17.0.2_linux-x64_bin.tar.gz \
+ -q -O /tmp/openjdk-17.0.2_linux-x64_bin.tar.gz \
+ && cd /tmp && tar -xzf openjdk-17.0.2_linux-x64_bin.tar.gz \
+ && cp -r jdk-17.0.2/ /usr/lib/jvm/ \
+ && rm /tmp/openjdk-17.0.2_linux-x64_bin.tar.gz && rm -rf /tmp/jdk-17.0.2/
+
# install ldb-toolchain
# 0.17
RUN wget https://doris-community-bj-1308700295.cos.ap-beijing.myqcloud.com/tools/ldb_toolchain_gen.sh \
diff --git a/docs/en/docs/install/source-install/compilation-general.md b/docs/en/docs/install/source-install/compilation-general.md
index a551357171e..2a020106cc9 100644
--- a/docs/en/docs/install/source-install/compilation-general.md
+++ b/docs/en/docs/install/source-install/compilation-general.md
@@ -74,24 +74,21 @@ This topic is about how to compile Doris from source.
> 3. For Doris 0.14.0, use `apache/incubator-doris:build-env-1.2` to compile; for Doris 0.14.x, use `apache/incubator-doris:build-env-1.3.1` to compile.
-> 4. The docker images of build-env-1.3.1 and above include both OpenJDK 8 and OpenJDK 11, please confirm the default JDK version with `java -version`. You can also switch versions as follows. (It is recommended to use JDK8.)
->
-> Switch to JDK 8:
->
-> ```
-> alternatives --set java java-1.8.0-openjdk.x86_64
-> alternatives --set javac java-1.8.0-openjdk.x86_64
-> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
-> ```
->
-> Switch to JDK 11:
->
-> ```
-> alternatives --set java java-11-openjdk.x86_64
-> alternatives --set javac java-11-openjdk.x86_64
-> export JAVA_HOME=/usr/lib/jvm/java-11
-> ```
->
+> 4. The latest version of the `apache/doris:build-env-ldb-toolchain-latest` image contains both JDK 8 and JDK 17.
+ >
+ > Switch to JDK 8:
+ >
+ > ```
+ > export JAVA_HOME=/usr/lib/jvm/java-1.8.0
+ > export PATH=$JAVA_HOME/bin/:$PATH
+ > ```
+ >
+ > Switch to JDK 17:
+ >
+ > ```
+ > export JAVA_HOME=/usr/lib/jvm/jdk-17.0.2/
+ > export PATH=$JAVA_HOME/bin/:$PATH
+ > ```
2. Run the image
diff --git a/docs/zh-CN/docs/install/source-install/compilation-general.md b/docs/zh-CN/docs/install/source-install/compilation-general.md
index 30bdab1761f..01408f2f8b0 100644
--- a/docs/zh-CN/docs/install/source-install/compilation-general.md
+++ b/docs/zh-CN/docs/install/source-install/compilation-general.md
@@ -73,22 +73,20 @@ under the License.
> 3. doris 0.14.0 版本仍然使用apache/incubator-doris:build-env-1.2 编译，0.14.x 版本的代码将使用apache/incubator-doris:build-env-1.3.1。
-> 4. 从 build-env-1.3.1 的docker镜像起，同时包含了 OpenJDK 8 和 OpenJDK 11，请通过 `java -version` 确认默认 JDK 版本。也可以通过以下方式切换版本（建议默认使用 JDK8）
+> 4. 最新版本的 `apache/doris:build-env-ldb-toolchain-latest` 镜像中同时包含 JDK 8 和 JDK 17。
>
> 切换到 JDK 8：
>
> ```
- > alternatives --set java java-1.8.0-openjdk.x86_64
- > alternatives --set javac java-1.8.0-openjdk.x86_64
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
+ > export PATH=$JAVA_HOME/bin/:$PATH
> ```
>
- > 切换到 JDK 11：
+ > 切换到 JDK 17：
>
> ```
- > alternatives --set java java-11-openjdk.x86_64
- > alternatives --set javac java-11-openjdk.x86_64
- > export JAVA_HOME=/usr/lib/jvm/java-11
+ > export JAVA_HOME=/usr/lib/jvm/jdk-17.0.2/
+ > export PATH=$JAVA_HOME/bin/:$PATH
> ```
2. 运行镜像
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 09/38: [fix](cloud) Fix FE cannot work when restarting after schema change (#31842)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b978365a7b03c22d5263fca88a5717b4b36de9f9
Author: deardeng <56...@qq.com>
AuthorDate: Wed Mar 6 17:45:28 2024 +0800
[fix](cloud) Fix FE cannot work when restarting after schema change (#31842)
```
2024-03-05 21:16:43,300 INFO (stateListener|110) [RollupJobV2.replayPendingJob():749] replay waiting txn rollup job: 30405
2024-03-05 21:16:43,300 INFO (stateListener|110) [MaterializedViewHandler.replayAlterJobV2():1114] set table's state to ROLLUP, table id: 29056, job id: 30405
2024-03-05 21:16:43,300 DEBUG (stateListener|110) [Env.replayJournal():2751] journal 21241 replayed.
2024-03-05 21:16:43,300 DEBUG (stateListener|110) [JournalObservable.notifyObservers():95] notify observers: journal: 21241, pos: 0, size: 0, obs: []
2024-03-05 21:16:43,300 DEBUG (stateListener|110) [JournalEntity.readFields():193] get opcode: 45
2024-03-05 21:16:43,300 DEBUG (stateListener|110) [EditLog.loadJournal():170] replay journal op code: 45
2024-03-05 21:16:43,301 DEBUG (stateListener|110) [InternalCatalog.unprotectUpdateReplica():1048] replay update a replica table id: 29056 partition id: 29050 index id: 30406 tablet id: 30505 backend id: 10002 replica id: 30506 version: 2 schema hash: -1 data size: 0 row count: 0 last failed version: -1 last success version: 2
2024-03-05 21:16:43,301 DEBUG (stateListener|110) [CloudReplica.getBackendId():143] connect context is null in getBackendId
2024-03-05 21:16:43,301 ERROR (stateListener|110) [EditLog.loadJournal():1226] Operation Type 45
java.lang.NullPointerException: table id: 29056 partition id: 29050 index id: 30406 tablet id: 30505 backend id: 10002 replica id: 30506 version: 2 schema hash: -1 data size: 0 row count: 0 last failed version: -1 last success version: 2
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:921) ~[guava-32.1.2-jre.jar:?]
at org.apache.doris.datasource.InternalCatalog.unprotectUpdateReplica(InternalCatalog.java:1054) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.InternalCatalog.replayUpdateReplica(InternalCatalog.java:1075) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.replayUpdateReplica(Env.java:3706) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:398) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.replayJournal(Env.java:2747) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.transferToMaster(Env.java:1459) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.access$1400(Env.java:319) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2638) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
```
---
.../src/main/java/org/apache/doris/datasource/InternalCatalog.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index d05f9cce48d..3057de0a381 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -1016,7 +1016,7 @@ public class InternalCatalog implements CatalogIf<Database> {
Partition partition = olapTable.getPartition(info.getPartitionId());
MaterializedIndex materializedIndex = partition.getIndex(info.getIndexId());
Tablet tablet = materializedIndex.getTablet(info.getTabletId());
- Replica replica = tablet.getReplicaByBackendId(info.getBackendId());
+ Replica replica = tablet.getReplicaById(info.getReplicaId());
Preconditions.checkNotNull(replica, info);
replica.updateVersionInfo(info.getVersion(), info.getDataSize(), info.getRemoteDataSize(), info.getRowCount());
replica.setBad(false);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 30/38: [fix](function) fix date_format function execution error on fe (#31645)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c0b25869a9e9af7fff4c1f7c0a122dda5aa40276
Author: lw112 <13...@users.noreply.github.com>
AuthorDate: Thu Mar 7 12:04:48 2024 +0800
[fix](function) fix date_format function execution error on fe (#31645)
---
.../src/main/java/org/apache/doris/analysis/DateLiteral.java | 8 +++++++-
.../src/main/java/org/apache/doris/nereids/util/DateUtils.java | 8 +++++++-
.../apache/doris/nereids/rules/expression/FoldConstantTest.java | 2 +-
.../src/test/java/org/apache/doris/rewrite/FEFunctionsTest.java | 2 +-
.../sql_functions/datetime_functions/test_date_function.out | 6 ++++++
.../sql_functions/datetime_functions/test_date_function.out | 6 ++++++
.../sql_functions/datetime_functions/test_date_function.groovy | 2 ++
.../sql_functions/datetime_functions/test_date_function.groovy | 2 ++
8 files changed, 32 insertions(+), 4 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
index 37171b70fac..5c16814707b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
@@ -45,6 +45,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.sql.Timestamp;
+import java.time.DayOfWeek;
import java.time.LocalDateTime;
import java.time.Year;
import java.time.ZoneId;
@@ -53,10 +54,12 @@ import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeFormatterBuilder;
import java.time.format.DateTimeParseException;
import java.time.format.ResolverStyle;
+import java.time.format.SignStyle;
import java.time.format.TextStyle;
import java.time.temporal.ChronoField;
import java.time.temporal.IsoFields;
import java.time.temporal.TemporalAccessor;
+import java.time.temporal.WeekFields;
import java.util.Collections;
import java.util.List;
import java.util.Map;
@@ -108,6 +111,7 @@ public class DateLiteral extends LiteralExpr {
private static Map<String, Integer> WEEK_DAY_NAME_DICT = Maps.newHashMap();
private static Set<Character> TIME_PART_SET = Sets.newHashSet();
private static final int[] DAYS_IN_MONTH = new int[] {0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
+ private static final WeekFields weekFields = WeekFields.of(DayOfWeek.SUNDAY, 7);
static {
try {
@@ -1047,7 +1051,7 @@ public class DateLiteral extends LiteralExpr {
builder.appendPattern("HH:mm:ss");
break;
case 'V': // %V Week (01..53), where Sunday is the first day of the week; used with %X
- builder.appendValue(ChronoField.ALIGNED_WEEK_OF_YEAR, 2);
+ builder.appendValue(weekFields.weekOfWeekBasedYear(), 2);
break;
case 'v': // %v Week (01..53), where Monday is the first day of the week; used with %x
builder.appendValue(IsoFields.WEEK_OF_WEEK_BASED_YEAR, 2);
@@ -1059,6 +1063,8 @@ public class DateLiteral extends LiteralExpr {
builder.appendValue(IsoFields.WEEK_BASED_YEAR, 4);
break;
case 'X':
+ builder.appendValue(weekFields.weekBasedYear(), 4, 10, SignStyle.EXCEEDS_PAD);
+ break;
case 'Y': // %Y Year, numeric, four digits
// %X Year for the week, where Sunday is the first day of the week,
// numeric, four digits; used with %v
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateUtils.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateUtils.java
index bd09bfa3429..70f25b25fc5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateUtils.java
@@ -20,19 +20,23 @@ package org.apache.doris.nereids.util;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.qe.ConnectContext;
+import java.time.DayOfWeek;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeFormatterBuilder;
+import java.time.format.SignStyle;
import java.time.format.TextStyle;
import java.time.temporal.ChronoField;
import java.time.temporal.IsoFields;
import java.time.temporal.TemporalAccessor;
+import java.time.temporal.WeekFields;
/**
* date util tools.
*/
public class DateUtils {
+ private static final WeekFields weekFields = WeekFields.of(DayOfWeek.SUNDAY, 7);
/**
* format builder.
@@ -103,7 +107,7 @@ public class DateUtils {
builder.appendPattern("HH:mm:ss");
break;
case 'V': // %V Week (01..53), where Sunday is the first day of the week; used with %X
- builder.appendValue(ChronoField.ALIGNED_WEEK_OF_YEAR, 2);
+ builder.appendValue(weekFields.weekOfWeekBasedYear(), 2);
break;
case 'v': // %v Week (01..53), where Monday is the first day of the week; used with %x
builder.appendValue(IsoFields.WEEK_OF_WEEK_BASED_YEAR, 2);
@@ -115,6 +119,8 @@ public class DateUtils {
builder.appendValue(IsoFields.WEEK_BASED_YEAR, 4);
break;
case 'X':
+ builder.appendValue(weekFields.weekBasedYear(), 4, 10, SignStyle.EXCEEDS_PAD);
+ break;
case 'Y': // %Y Year, numeric, four digits
// %X Year for the week, where Sunday is the first day of the week,
// numeric, four digits; used with %v
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantTest.java
index 5132fd27ff4..3b8fbc8526b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantTest.java
@@ -491,7 +491,7 @@ class FoldConstantTest extends ExpressionRewriteTestHelper {
Assertions.assertEquals(DateTimeExtractAndTransform.date(dateLiteral).toSql(), answer[answerIdx++]);
Assertions.assertEquals(DateTimeExtractAndTransform.dateV2(dateLiteral).toSql(), answer[answerIdx]);
- Assertions.assertEquals("'2021 52 2022 01'", DateTimeExtractAndTransform.dateFormat(
+ Assertions.assertEquals("'2021 52 2021 52'", DateTimeExtractAndTransform.dateFormat(
new DateTimeLiteral("2022-01-01 00:12:42"),
new VarcharLiteral("%x %v %X %V")).toSql());
Assertions.assertEquals("'2023 18 2023 19'", DateTimeExtractAndTransform.dateFormat(
diff --git a/fe/fe-core/src/test/java/org/apache/doris/rewrite/FEFunctionsTest.java b/fe/fe-core/src/test/java/org/apache/doris/rewrite/FEFunctionsTest.java
index 727a6635620..b405d787219 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/rewrite/FEFunctionsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/rewrite/FEFunctionsTest.java
@@ -386,7 +386,7 @@ public class FEFunctionsTest {
new StringLiteral("2013-05-17 08:07:05 PM"), new StringLiteral("%Y-%m-%d %r")).getStringValue());
Assert.assertEquals("2013-05-17 08:07:05", FEFunctions.dateParse(new StringLiteral("2013-05-17 08:07:05"),
new StringLiteral("%Y-%m-%d %T")).getStringValue());
- Assert.assertEquals("2021 52 2022 01", FEFunctions.dateFormat(new DateLiteral("2022-01-01 00:12:42", Type.DATETIMEV2),
+ Assert.assertEquals("2021 52 2021 52", FEFunctions.dateFormat(new DateLiteral("2022-01-01 00:12:42", Type.DATETIMEV2),
new StringLiteral("%x %v %X %V")).getStringValue());
Assert.assertEquals("2023 18 2023 19", FEFunctions.dateFormat(new DateLiteral("2023-05-07 02:41:42", Type.DATETIMEV2),
new StringLiteral("%x %v %X %V")).getStringValue());
diff --git a/regression-test/data/nereids_p0/sql_functions/datetime_functions/test_date_function.out b/regression-test/data/nereids_p0/sql_functions/datetime_functions/test_date_function.out
index 086faf0c859..46a6256615c 100644
--- a/regression-test/data/nereids_p0/sql_functions/datetime_functions/test_date_function.out
+++ b/regression-test/data/nereids_p0/sql_functions/datetime_functions/test_date_function.out
@@ -479,6 +479,12 @@ true
-- !sql --
true
+-- !sql --
+1998 52
+
+-- !sql --
+2024 52
+
-- !sql --
2022 31 4
diff --git a/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out b/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out
index dc1d05c9d48..ffd8d5b8cd2 100644
--- a/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out
+++ b/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out
@@ -496,6 +496,12 @@ true
-- !sql --
true
+-- !sql --
+1998 52
+
+-- !sql --
+2024 52
+
-- !sql --
2022 31 4
diff --git a/regression-test/suites/nereids_p0/sql_functions/datetime_functions/test_date_function.groovy b/regression-test/suites/nereids_p0/sql_functions/datetime_functions/test_date_function.groovy
index 633e884dad0..58413e29cee 100644
--- a/regression-test/suites/nereids_p0/sql_functions/datetime_functions/test_date_function.groovy
+++ b/regression-test/suites/nereids_p0/sql_functions/datetime_functions/test_date_function.groovy
@@ -470,6 +470,8 @@ suite("test_date_function") {
sql """ drop table ${tableName} """
+ qt_sql """ select date_format('1999-01-01', '%X %V'); """
+ qt_sql """ select date_format('2025-01-01', '%X %V'); """
qt_sql """ select date_format('2022-08-04', '%X %V %w'); """
qt_sql """ select STR_TO_DATE('Tue Jul 12 20:00:45 CST 2022', '%a %b %e %H:%i:%s %Y'); """
qt_sql """ select STR_TO_DATE('Tue Jul 12 20:00:45 CST 2022', '%a %b %e %T CST %Y'); """
diff --git a/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy b/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
index cbef7072b26..ba48c34f409 100644
--- a/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
+++ b/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
@@ -473,6 +473,8 @@ suite("test_date_function") {
sql """ drop table ${tableName} """
+ qt_sql """ select date_format('1999-01-01', '%X %V'); """
+ qt_sql """ select date_format('2025-01-01', '%X %V'); """
qt_sql """ select date_format('2022-08-04', '%X %V %w'); """
qt_sql """ select STR_TO_DATE('Tue Jul 12 20:00:45 CST 2022', '%a %b %e %H:%i:%s %Y'); """
qt_sql """ select STR_TO_DATE('Tue Jul 12 20:00:45 CST 2022', '%a %b %e %T CST %Y'); """
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 14/38: [Bug](partition) fix npe when prune partition with not exist partition column in mv #31860
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3716f8a171f0aea70720313d7a1676ec487659af
Author: Pxl <px...@qq.com>
AuthorDate: Wed Mar 6 20:47:06 2024 +0800
[Bug](partition) fix npe when prune partition with not exist partition column in mv #31860
---
.../rules/rewrite/PruneOlapScanPartition.java | 15 +++--
.../mv_p0/test_mv_partition/test_mv_partition.out | 4 ++
.../test_mv_partition/test_mv_partition.groovy | 64 ++++++++++++++++++++++
3 files changed, 79 insertions(+), 4 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java
index e9e59f1c113..eef8dc33fa0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java
@@ -17,6 +17,7 @@
package org.apache.doris.nereids.rules.rewrite;
+import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PartitionInfo;
import org.apache.doris.catalog.PartitionItem;
@@ -62,10 +63,16 @@ public class PruneOlapScanPartition extends OneRewriteRuleFactory {
.collect(Collectors.toMap(slot -> slot.getName().toLowerCase(), Function.identity()));
PartitionInfo partitionInfo = table.getPartitionInfo();
- List<Slot> partitionSlots = partitionInfo.getPartitionColumns()
- .stream()
- .map(column -> scanOutput.get(column.getName().toLowerCase()))
- .collect(Collectors.toList());
+ List<Slot> partitionSlots = new ArrayList<>();
+ for (Column column : partitionInfo.getPartitionColumns()) {
+ Slot slot = scanOutput.get(column.getName().toLowerCase());
+ if (slot == null) {
+ return filter;
+ } else {
+ partitionSlots.add(slot);
+ }
+ }
+
List<Long> manuallySpecifiedPartitions = scan.getManuallySpecifiedPartitions();
Map<Long, PartitionItem> idToPartitions;
diff --git a/regression-test/data/mv_p0/test_mv_partition/test_mv_partition.out b/regression-test/data/mv_p0/test_mv_partition/test_mv_partition.out
new file mode 100644
index 00000000000..352a8c8050f
--- /dev/null
+++ b/regression-test/data/mv_p0/test_mv_partition/test_mv_partition.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !select_mv --
+2024-03-04 2
+
diff --git a/regression-test/suites/mv_p0/test_mv_partition/test_mv_partition.groovy b/regression-test/suites/mv_p0/test_mv_partition/test_mv_partition.groovy
new file mode 100644
index 00000000000..510156e01c6
--- /dev/null
+++ b/regression-test/suites/mv_p0/test_mv_partition/test_mv_partition.groovy
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite ("test_mv_partition") {
+ sql """set enable_nereids_planner=true;"""
+ sql """set enable_fallback_to_original_planner=false;"""
+ sql """ DROP TABLE IF EXISTS chh; """
+
+ sql """
+ CREATE TABLE `chh` (
+ `event_id` varchar(50) NULL COMMENT '',
+ `time_stamp` datetime NULL COMMENT '',
+ `device_id` varchar(150) NULL DEFAULT "" COMMENT ''
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`event_id`)
+ PARTITION BY RANGE(`time_stamp`)()
+ DISTRIBUTED BY HASH(`device_id`) BUCKETS AUTO
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "is_being_synced" = "false",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "DAY",
+ "dynamic_partition.time_zone" = "Asia/Shanghai",
+ "dynamic_partition.start" = "-30",
+ "dynamic_partition.end" = "1",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.replication_allocation" = "tag.location.default: 1",
+ "dynamic_partition.buckets" = "20",
+ "dynamic_partition.create_history_partition" = "true",
+ "dynamic_partition.history_partition_num" = "-1",
+ "dynamic_partition.hot_partition_num" = "0",
+ "dynamic_partition.reserved_history_periods" = "NULL",
+ "dynamic_partition.storage_policy" = "",
+ "dynamic_partition.storage_medium" = "HDD",
+ "storage_format" = "V2",
+ "disable_auto_compaction" = "false",
+ "enable_single_replica_compaction" = "false"
+ );
+ """
+
+ sql """insert into chh(event_id,time_stamp,device_id) values('ad_sdk_request','2024-03-04 00:00:00','a');"""
+
+ createMV("create materialized view m_view as select to_date(time_stamp),count(device_id) from chh group by to_date(time_stamp);")
+
+ sql """insert into chh(event_id,time_stamp,device_id) values('ad_sdk_request','2024-03-04 00:00:00','a');"""
+
+ qt_select_mv "select * from chh index m_view where `mv_to_date(time_stamp)` = '2024-03-04';"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 36/38: [Fix](Nereids) fix hint cases with random result (#31865)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit db65114f77dbffc1ce3b3803015d9e5b30ae310d
Author: LiBinfeng <46...@users.noreply.github.com>
AuthorDate: Thu Mar 7 13:57:21 2024 +0800
[Fix](Nereids) fix hint cases with random result (#31865)
---
.../data/nereids_p0/hint/test_distribute.out | 2 +-
.../data/nereids_p0/hint/test_leading.out | 146 +--------------------
.../suites/nereids_p0/hint/test_distribute.groovy | 2 +-
.../suites/nereids_p0/hint/test_leading.groovy | 14 +-
4 files changed, 16 insertions(+), 148 deletions(-)
diff --git a/regression-test/data/nereids_p0/hint/test_distribute.out b/regression-test/data/nereids_p0/hint/test_distribute.out
index 5fae729afe6..a14b6696a8d 100644
--- a/regression-test/data/nereids_p0/hint/test_distribute.out
+++ b/regression-test/data/nereids_p0/hint/test_distribute.out
@@ -442,7 +442,7 @@ PhysicalResultSink
----------------------PhysicalOlapScan[t2]
Hint log:
-Used:
+Used: leading(t3 { t1 t2 } )
UnUsed: [shuffle]_2
SyntaxError:
diff --git a/regression-test/data/nereids_p0/hint/test_leading.out b/regression-test/data/nereids_p0/hint/test_leading.out
index 2c0ddc63142..7274d409312 100644
--- a/regression-test/data/nereids_p0/hint/test_leading.out
+++ b/regression-test/data/nereids_p0/hint/test_leading.out
@@ -1815,157 +1815,25 @@
119
-- !select70_1 --
-PhysicalResultSink
---hashAgg[GLOBAL]
-----PhysicalDistribute[DistributionSpecGather]
-------hashAgg[LOCAL]
---------PhysicalProject
-----------hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t2.c2)) otherCondition=()
-------------PhysicalProject
---------------PhysicalOlapScan[t1]
-------------PhysicalDistribute[DistributionSpecHash]
---------------PhysicalProject
-----------------PhysicalOlapScan[t2]
-
-Hint log:
-Used:
-UnUsed:
-SyntaxError: leading(t1 t3) Msg:can not find table: t3
+1719
-- !select70_2 --
-PhysicalResultSink
---hashAgg[GLOBAL]
-----PhysicalDistribute[DistributionSpecGather]
-------hashAgg[LOCAL]
---------PhysicalProject
-----------hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t2.c2)) otherCondition=()
-------------PhysicalProject
---------------PhysicalOlapScan[t1]
-------------PhysicalDistribute[DistributionSpecHash]
---------------PhysicalProject
-----------------PhysicalOlapScan[t2]
-
-Hint log:
-Used:
-UnUsed:
-SyntaxError: leading(t1 t5) Msg:can not find table: t5
+1719
-- !select71_1 --
-PhysicalResultSink
---hashAgg[GLOBAL]
-----PhysicalDistribute[DistributionSpecGather]
-------hashAgg[LOCAL]
---------PhysicalProject
-----------hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t2.c2)) otherCondition=()
-------------PhysicalProject
---------------PhysicalOlapScan[t1]
-------------PhysicalDistribute[DistributionSpecHash]
---------------PhysicalProject
-----------------PhysicalOlapScan[t2]
-
-Hint log:
-Used:
-UnUsed:
-SyntaxError: leading(t1 t1 t2) Msg:duplicated table
+1719
-- !select71_2 --
-PhysicalResultSink
---hashAgg[GLOBAL]
-----PhysicalDistribute[DistributionSpecGather]
-------hashAgg[LOCAL]
---------PhysicalProject
-----------hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t2.c2)) otherCondition=()
-------------PhysicalProject
---------------PhysicalOlapScan[t1]
-------------PhysicalDistribute[DistributionSpecHash]
---------------PhysicalProject
-----------------PhysicalOlapScan[t2]
-
-Hint log:
-Used:
-UnUsed:
-SyntaxError: leading(t1 t2 t2) Msg:duplicated table
+1719
-- !select72_1 --
-PhysicalResultSink
---hashAgg[GLOBAL]
-----PhysicalDistribute[DistributionSpecGather]
-------hashAgg[LOCAL]
---------PhysicalProject
-----------hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t2.c2)) otherCondition=()
-------------PhysicalProject
---------------PhysicalOlapScan[t1]
-------------PhysicalDistribute[DistributionSpecHash]
---------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((t2.c2 = t3.c3)) otherCondition=()
-----------------PhysicalProject
-------------------PhysicalOlapScan[t2]
-----------------PhysicalDistribute[DistributionSpecHash]
-------------------PhysicalProject
---------------------hashJoin[INNER_JOIN] hashCondition=((t3.c3 = t4.c4)) otherCondition=()
-----------------------PhysicalProject
-------------------------PhysicalOlapScan[t4]
-----------------------PhysicalDistribute[DistributionSpecHash]
-------------------------PhysicalProject
---------------------------PhysicalOlapScan[t3]
-
-Hint log:
-Used: leading(t4 t3 )
-UnUsed:
-SyntaxError:
+1235
-- !select72_2 --
-PhysicalResultSink
---hashAgg[GLOBAL]
-----PhysicalDistribute[DistributionSpecGather]
-------hashAgg[LOCAL]
---------PhysicalProject
-----------hashJoin[LEFT_SEMI_JOIN] hashCondition=((t2.c2 = t3.c3)) otherCondition=()
-------------hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t2.c2)) otherCondition=()
---------------PhysicalProject
-----------------PhysicalOlapScan[t1]
---------------PhysicalDistribute[DistributionSpecHash]
-----------------PhysicalProject
-------------------PhysicalOlapScan[t2]
-------------PhysicalDistribute[DistributionSpecHash]
---------------PhysicalProject
-----------------hashJoin[INNER_JOIN] hashCondition=((t3.c3 = t4.c4)) otherCondition=()
-------------------PhysicalProject
---------------------PhysicalOlapScan[t3]
-------------------PhysicalDistribute[DistributionSpecHash]
---------------------PhysicalProject
-----------------------PhysicalOlapScan[t4]
-
-Hint log:
-Used: leading(t1 t2 )
-UnUsed:
-SyntaxError:
+1235
-- !select73_1 --
-PhysicalResultSink
---hashAgg[GLOBAL]
-----PhysicalDistribute[DistributionSpecGather]
-------hashAgg[LOCAL]
---------PhysicalProject
-----------hashJoin[LEFT_SEMI_JOIN] hashCondition=((t2.c2 = t3.c3)) otherCondition=()
-------------hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t2.c2)) otherCondition=()
---------------PhysicalProject
-----------------PhysicalOlapScan[t1]
---------------PhysicalDistribute[DistributionSpecHash]
-----------------PhysicalProject
-------------------PhysicalOlapScan[t2]
-------------PhysicalDistribute[DistributionSpecHash]
---------------PhysicalProject
-----------------hashJoin[INNER_JOIN] hashCondition=((t3.c3 = t4.c4)) otherCondition=()
-------------------PhysicalProject
---------------------PhysicalOlapScan[t4]
-------------------PhysicalDistribute[DistributionSpecHash]
---------------------PhysicalProject
-----------------------PhysicalOlapScan[t3]
-
-Hint log:
-Used: leading(t1 t2 ) leading(t4 t3 )
-UnUsed:
-SyntaxError:
+1235
-- !select81_1 --
259
diff --git a/regression-test/suites/nereids_p0/hint/test_distribute.groovy b/regression-test/suites/nereids_p0/hint/test_distribute.groovy
index a8b4c308d44..6ac7faa1ec0 100644
--- a/regression-test/suites/nereids_p0/hint/test_distribute.groovy
+++ b/regression-test/suites/nereids_p0/hint/test_distribute.groovy
@@ -104,7 +104,7 @@ suite("test_distribute") {
qt_select5_1 """explain shape plan select count(*) from t1 join [shuffle] t2 on c1 > c2;"""
qt_select5_2 """explain shape plan select count(*) from t1 join [broadcast] t2 on c1 > c2;"""
- qt_select6_1 """explain shape plan select count(*) from t1 join [shuffle] t2 on c1 > c2 join t3 on c2 > c3 where c1 < 100;"""
+ qt_select6_1 """explain shape plan select /*+ leading(t3 {t1 t2}) */ count(*) from t1 join [shuffle] t2 on c1 > c2 join t3 on c2 > c3 where c1 < 100;"""
qt_select6_2 """explain shape plan select count(*) from t1 join t2 on c1 > c2 join t3 on c2 > c3 where c1 < 100;"""
// (A leftjoin B on (Pab)) leftjoin C on (Pac) = (A leftjoin C on (Pac)) leftjoin B on (Pab)
diff --git a/regression-test/suites/nereids_p0/hint/test_leading.groovy b/regression-test/suites/nereids_p0/hint/test_leading.groovy
index fd1abbc526c..50771186c64 100644
--- a/regression-test/suites/nereids_p0/hint/test_leading.groovy
+++ b/regression-test/suites/nereids_p0/hint/test_leading.groovy
@@ -794,19 +794,19 @@ suite("test_leading") {
qt_select68_13 """select /*+ leading(t3 {t2 t1}) */ count(*) from t1 left semi join t2 on c1 = c2 left anti join t3 on c1 = c3 where exists (select 1 from t3 join t4 on c3 = c4);"""
// wrong table name
- qt_select70_1 """explain shape plan select /*+ leading(t1 t3) */ count(*) from t1 join t2 on c1 = c2;"""
- qt_select70_2 """explain shape plan select /*+ leading(t1 t5) */ count(*) from t1 join t2 on c1 = c2;"""
+ qt_select70_1 """select /*+ leading(t1 t3) */ count(*) from t1 join t2 on c1 = c2;"""
+ qt_select70_2 """select /*+ leading(t1 t5) */ count(*) from t1 join t2 on c1 = c2;"""
// duplicate table name
- qt_select71_1 """explain shape plan select /*+ leading(t1 t1 t2) */ count(*) from t1 join t2 on c1 = c2;"""
- qt_select71_2 """explain shape plan select /*+ leading(t1 t2 t2) */ count(*) from t1 join t2 on c1 = c2;"""
+ qt_select71_1 """select /*+ leading(t1 t1 t2) */ count(*) from t1 join t2 on c1 = c2;"""
+ qt_select71_2 """select /*+ leading(t1 t2 t2) */ count(*) from t1 join t2 on c1 = c2;"""
// different scope
- qt_select72_1 """explain shape plan select count(*) from t1 join t2 on c1 = c2 where c2 in (select /*+ leading(t4 t3) */ c3 from t3 join t4 on c3 = c4);"""
- qt_select72_2 """explain shape plan select /*+ leading(t1 t2) */ count(*) from t1 join t2 on c1 = c2 where c2 in (select c3 from t3 join t4 on c3 = c4);"""
+ qt_select72_1 """select count(*) from t1 join t2 on c1 = c2 where c2 in (select /*+ leading(t4 t3) */ c3 from t3 join t4 on c3 = c4);"""
+ qt_select72_2 """select /*+ leading(t1 t2) */ count(*) from t1 join t2 on c1 = c2 where c2 in (select c3 from t3 join t4 on c3 = c4);"""
// multi leading hint
- qt_select73_1 """explain shape plan select /*+ leading(t1 t2) */ count(*) from t1 join t2 on c1 = c2 where c2 in (select /*+ leading(t4 t3) */ c3 from t3 join t4 on c3 = c4);"""
+ qt_select73_1 """select /*+ leading(t1 t2) */ count(*) from t1 join t2 on c1 = c2 where c2 in (select /*+ leading(t4 t3) */ c3 from t3 join t4 on c3 = c4);"""
// alias
// inner join + anti join
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 05/38: [fix](group commit) group commit may heap-use-after-free if execute plan failed (#31839)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ff8c679851d6e87af70e2484efa83522cd5af13f
Author: meiyi <my...@gmail.com>
AuthorDate: Wed Mar 6 14:14:35 2024 +0800
[fix](group commit) group commit may heap-use-after-free if execute plan failed (#31839)
---
be/src/runtime/group_commit_mgr.cpp | 20 +++-----------------
be/src/runtime/group_commit_mgr.h | 3 +--
2 files changed, 4 insertions(+), 19 deletions(-)
diff --git a/be/src/runtime/group_commit_mgr.cpp b/be/src/runtime/group_commit_mgr.cpp
index 887dd856bb4..cadff231761 100644
--- a/be/src/runtime/group_commit_mgr.cpp
+++ b/be/src/runtime/group_commit_mgr.cpp
@@ -212,10 +212,9 @@ Status GroupCommitTable::get_first_block_load_queue(
if (!_is_creating_plan_fragment) {
_is_creating_plan_fragment = true;
RETURN_IF_ERROR(_thread_pool->submit_func([&] {
- auto st = _create_group_commit_load(load_block_queue, be_exe_version);
+ auto st = _create_group_commit_load(be_exe_version);
if (!st.ok()) {
LOG(WARNING) << "create group commit load error, st=" << st.to_string();
- load_block_queue.reset();
std::unique_lock l(_lock);
_is_creating_plan_fragment = false;
_cv.notify_all();
@@ -223,26 +222,13 @@ Status GroupCommitTable::get_first_block_load_queue(
}));
}
_cv.wait_for(l, std::chrono::seconds(4));
- if (load_block_queue != nullptr) {
- if (load_block_queue->schema_version == base_schema_version) {
- if (load_block_queue->add_load_id(load_id).ok()) {
- return Status::OK();
- }
- } else if (base_schema_version < load_block_queue->schema_version) {
- return Status::DataQualityError<false>(
- "schema version not match, maybe a schema change is in process. Please "
- "retry this load manually.");
- }
- load_block_queue.reset();
- }
}
}
return Status::InternalError<false>("can not get a block queue for table_id: " +
std::to_string(_table_id));
}
-Status GroupCommitTable::_create_group_commit_load(
- std::shared_ptr<LoadBlockQueue>& load_block_queue, int be_exe_version) {
+Status GroupCommitTable::_create_group_commit_load(int be_exe_version) {
Status st = Status::OK();
TStreamLoadPutRequest request;
UniqueId load_id = UniqueId::gen_uid();
@@ -305,7 +291,7 @@ Status GroupCommitTable::_create_group_commit_load(
<< ", txn_id=" << txn_id << ", instance_id=" << print_id(instance_id)
<< ", is_pipeline=" << is_pipeline;
{
- load_block_queue = std::make_shared<LoadBlockQueue>(
+ auto load_block_queue = std::make_shared<LoadBlockQueue>(
instance_id, label, txn_id, schema_version, _all_block_queues_bytes,
result.wait_internal_group_commit_finish, result.group_commit_interval_ms,
result.group_commit_data_bytes);
diff --git a/be/src/runtime/group_commit_mgr.h b/be/src/runtime/group_commit_mgr.h
index dc135ea1063..f77012bc2d1 100644
--- a/be/src/runtime/group_commit_mgr.h
+++ b/be/src/runtime/group_commit_mgr.h
@@ -134,8 +134,7 @@ public:
std::shared_ptr<LoadBlockQueue>& load_block_queue);
private:
- Status _create_group_commit_load(std::shared_ptr<LoadBlockQueue>& load_block_queue,
- int be_exe_version);
+ Status _create_group_commit_load(int be_exe_version);
Status _exec_plan_fragment(int64_t db_id, int64_t table_id, const std::string& label,
int64_t txn_id, bool is_pipeline,
const TExecPlanFragmentParams& params,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 22/38: [typo](doc) Add a full backup case and modify sidebars (#31801)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8084a0a74ae09992db9f2407c4daa514a77c0b03
Author: yuanyuan8983 <99...@users.noreply.github.com>
AuthorDate: Wed Mar 6 23:16:24 2024 +0800
[typo](doc) Add a full backup case and modify sidebars (#31801)
---
.../Data-Definition-Statements/Backup-and-Restore/BACKUP.md | 7 +++++++
docs/sidebars.json | 1 -
.../Data-Definition-Statements/Backup-and-Restore/BACKUP.md | 7 +++++++
3 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
index 5d425b9669c..ef5f7a286c9 100644
--- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
+++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
@@ -88,6 +88,13 @@ TO example_repo
EXCLUDE (example_tbl);
```
+4. Fully back up tables under example_db to the repository example_repo:
+
+```sql
+BACKUP SNAPSHOT example_db.snapshot_label3
+TO example_repo;
+```
+
### Keywords
BACKUP
diff --git a/docs/sidebars.json b/docs/sidebars.json
index 1f14a36cd58..2e009562a82 100644
--- a/docs/sidebars.json
+++ b/docs/sidebars.json
@@ -820,7 +820,6 @@
"sql-manual/sql-functions/table-functions/catalogs",
"sql-manual/sql-functions/table-functions/frontends_disks",
"sql-manual/sql-functions/table-functions/active_queries",
- "sql-manual/sql-functions/table-functions/job",
"sql-manual/sql-functions/table-functions/jobs",
"sql-manual/sql-functions/table-functions/mv_infos",
"sql-manual/sql-functions/table-functions/tasks"
diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
index 8b283f4f5ee..82081e55372 100644
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
@@ -90,6 +90,13 @@ TO example_repo
EXCLUDE (example_tbl);
```
+4. 全量备份 example_db 下的表到仓库 example_repo 中：
+
+```sql
+BACKUP SNAPSHOT example_db.snapshot_label3
+TO example_repo;
+```
+
### Keywords
```text
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 04/38: [Improvement](profile)Add tvf active_be_tasks() #31815
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 28f0b7eb321ec6839808d252bf0d04775df18e1a
Author: wangbo <wa...@apache.org>
AuthorDate: Wed Mar 6 13:53:57 2024 +0800
[Improvement](profile)Add tvf active_be_tasks() #31815
---
be/src/runtime/runtime_query_statistics_mgr.cpp | 50 ++++++++++++++++++++++
be/src/runtime/runtime_query_statistics_mgr.h | 5 +++
be/src/vec/exec/scan/vmeta_scanner.cpp | 22 +++++++++-
be/src/vec/exec/scan/vmeta_scanner.h | 2 +
.../doris/catalog/BuiltinTableValuedFunctions.java | 4 +-
.../{ActiveQueries.java => ActiveBeTasks.java} | 17 ++++----
.../expressions/functions/table/ActiveQueries.java | 2 +-
.../visitor/TableValuedFunctionVisitor.java | 5 +++
....java => ActiveBeTasksTableValuedFunction.java} | 50 ++++++----------------
.../ActiveQueriesTableValuedFunction.java | 4 +-
.../doris/tablefunction/TableValuedFunctionIf.java | 2 +
gensrc/thrift/Types.thrift | 3 +-
12 files changed, 114 insertions(+), 52 deletions(-)
diff --git a/be/src/runtime/runtime_query_statistics_mgr.cpp b/be/src/runtime/runtime_query_statistics_mgr.cpp
index cb137fbe82a..ee09b0c30dc 100644
--- a/be/src/runtime/runtime_query_statistics_mgr.cpp
+++ b/be/src/runtime/runtime_query_statistics_mgr.cpp
@@ -199,4 +199,54 @@ void RuntimeQueryStatiticsMgr::set_workload_group_id(std::string query_id, int64
}
}
+std::vector<TRow> RuntimeQueryStatiticsMgr::get_active_be_tasks_statistics(
+ std::vector<std::string> filter_columns) {
+ std::shared_lock<std::shared_mutex> read_lock(_qs_ctx_map_lock);
+ std::vector<TRow> table_rows;
+ int64_t be_id = ExecEnv::GetInstance()->master_info()->backend_id;
+
+ for (auto& [query_id, qs_ctx_ptr] : _query_statistics_ctx_map) {
+ TRow trow;
+
+ TQueryStatistics tqs;
+ qs_ctx_ptr->collect_query_statistics(&tqs);
+
+ for (auto iter = filter_columns.begin(); iter != filter_columns.end(); iter++) {
+ std::string col_name = *iter;
+
+ TCell tcell;
+ if (col_name == "beid") {
+ tcell.longVal = be_id;
+ } else if (col_name == "fehost") {
+ tcell.stringVal = qs_ctx_ptr->_fe_addr.hostname;
+ } else if (col_name == "queryid") {
+ tcell.stringVal = query_id;
+ } else if (col_name == "tasktimems") {
+ if (qs_ctx_ptr->_is_query_finished) {
+ tcell.longVal = qs_ctx_ptr->_query_finish_time - qs_ctx_ptr->_query_start_time;
+ } else {
+ tcell.longVal = MonotonicMillis() - qs_ctx_ptr->_query_start_time;
+ }
+ } else if (col_name == "taskcputimems") {
+ tcell.longVal = tqs.cpu_ms;
+ } else if (col_name == "scanrows") {
+ tcell.longVal = tqs.scan_rows;
+ } else if (col_name == "scanbytes") {
+ tcell.longVal = tqs.scan_bytes;
+ } else if (col_name == "bepeakmemorybytes") {
+ tcell.longVal = tqs.max_peak_memory_bytes;
+ } else if (col_name == "currentusedmemorybytes") {
+ tcell.longVal = tqs.current_used_memory_bytes;
+ } else if (col_name == "shufflesendbytes") {
+ tcell.longVal = tqs.shuffle_send_bytes;
+ } else if (col_name == "shufflesendRows") {
+ tcell.longVal = tqs.shuffle_send_rows;
+ }
+ trow.column_value.push_back(tcell);
+ }
+ table_rows.push_back(trow);
+ }
+ return table_rows;
+}
+
} // namespace doris
\ No newline at end of file
diff --git a/be/src/runtime/runtime_query_statistics_mgr.h b/be/src/runtime/runtime_query_statistics_mgr.h
index 69b283b6d14..44badd196a3 100644
--- a/be/src/runtime/runtime_query_statistics_mgr.h
+++ b/be/src/runtime/runtime_query_statistics_mgr.h
@@ -17,6 +17,8 @@
#pragma once
+#include <gen_cpp/Data_types.h>
+
#include <shared_mutex>
#include <string>
@@ -66,6 +68,9 @@ public:
void get_metric_map(std::string query_id,
std::map<WorkloadMetricType, std::string>& metric_map);
+ // used for tvf active_queries
+ std::vector<TRow> get_active_be_tasks_statistics(std::vector<std::string> filter_columns);
+
private:
std::shared_mutex _qs_ctx_map_lock;
std::map<std::string, std::unique_ptr<QueryStatisticsCtx>> _query_statistics_ctx_map;
diff --git a/be/src/vec/exec/scan/vmeta_scanner.cpp b/be/src/vec/exec/scan/vmeta_scanner.cpp
index 22545fa4dce..7d438366cf2 100644
--- a/be/src/vec/exec/scan/vmeta_scanner.cpp
+++ b/be/src/vec/exec/scan/vmeta_scanner.cpp
@@ -34,6 +34,7 @@
#include "runtime/define_primitive_type.h"
#include "runtime/descriptors.h"
#include "runtime/exec_env.h"
+#include "runtime/runtime_query_statistics_mgr.h"
#include "runtime/runtime_state.h"
#include "runtime/types.h"
#include "util/thrift_rpc_helper.h"
@@ -95,7 +96,12 @@ Status VMetaScanner::prepare(RuntimeState* state, const VExprContextSPtrs& conju
return Status::InternalError("Logical error, VMetaScanner do not allow ColumnNullable");
}
- RETURN_IF_ERROR(_fetch_metadata(_scan_range.meta_scan_range));
+ if (_scan_range.meta_scan_range.metadata_type == TMetadataType::ACTIVE_BE_TASKS) {
+ // tvf active_be_tasks fetch data in be directly, it does not need to request FE for data
+ RETURN_IF_ERROR(_build_active_be_tasks_data());
+ } else {
+ RETURN_IF_ERROR(_fetch_metadata(_scan_range.meta_scan_range));
+ }
return Status::OK();
}
@@ -288,6 +294,20 @@ Status VMetaScanner::_fetch_metadata(const TMetaScanRange& meta_scan_range) {
return Status::OK();
}
+Status VMetaScanner::_build_active_be_tasks_data() {
+ std::vector<std::string> filter_columns;
+ for (const auto& slot : _tuple_desc->slots()) {
+ filter_columns.emplace_back(slot->col_name_lower_case());
+ }
+
+ std::vector<TRow> ret =
+ ExecEnv::GetInstance()->runtime_query_statistics_mgr()->get_active_be_tasks_statistics(
+ filter_columns);
+ _batch_data = std::move(ret);
+
+ return Status::OK();
+}
+
Status VMetaScanner::_build_iceberg_metadata_request(const TMetaScanRange& meta_scan_range,
TFetchSchemaTableDataRequest* request) {
VLOG_CRITICAL << "VMetaScanner::_build_iceberg_metadata_request";
diff --git a/be/src/vec/exec/scan/vmeta_scanner.h b/be/src/vec/exec/scan/vmeta_scanner.h
index 59bd55dc2d8..25cb9345311 100644
--- a/be/src/vec/exec/scan/vmeta_scanner.h
+++ b/be/src/vec/exec/scan/vmeta_scanner.h
@@ -91,6 +91,8 @@ private:
TFetchSchemaTableDataRequest* request);
Status _build_queries_metadata_request(const TMetaScanRange& meta_scan_range,
TFetchSchemaTableDataRequest* request);
+ Status _build_active_be_tasks_data();
+
bool _meta_eos;
TupleId _tuple_id;
TUserIdentity _user_identity;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java
index ac1b31fceac..acdeb683f26 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java
@@ -17,6 +17,7 @@
package org.apache.doris.catalog;
+import org.apache.doris.nereids.trees.expressions.functions.table.ActiveBeTasks;
import org.apache.doris.nereids.trees.expressions.functions.table.ActiveQueries;
import org.apache.doris.nereids.trees.expressions.functions.table.Backends;
import org.apache.doris.nereids.trees.expressions.functions.table.Catalogs;
@@ -59,7 +60,8 @@ public class BuiltinTableValuedFunctions implements FunctionHelper {
tableValued(MvInfos.class, "mv_infos"),
tableValued(Jobs.class, "jobs"),
tableValued(Tasks.class, "tasks"),
- tableValued(WorkloadGroups.class, "workload_groups")
+ tableValued(WorkloadGroups.class, "workload_groups"),
+ tableValued(ActiveBeTasks.class, "active_be_tasks")
);
public static final BuiltinTableValuedFunctions INSTANCE = new BuiltinTableValuedFunctions();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveQueries.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveBeTasks.java
similarity index 77%
copy from fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveQueries.java
copy to fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveBeTasks.java
index f8dcaa4a7ec..5737f52a2b9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveQueries.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveBeTasks.java
@@ -22,18 +22,18 @@ import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.trees.expressions.Properties;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.coercion.AnyDataType;
-import org.apache.doris.tablefunction.ActiveQueriesTableValuedFunction;
+import org.apache.doris.tablefunction.ActiveBeTasksTableValuedFunction;
import org.apache.doris.tablefunction.TableValuedFunctionIf;
import java.util.Map;
/**
- * queries tvf
+ * stands be running tasks status, currently main including select/streamload/broker load/insert select
*/
-public class ActiveQueries extends TableValuedFunction {
+public class ActiveBeTasks extends TableValuedFunction {
- public ActiveQueries(Properties properties) {
- super("active_queries", properties);
+ public ActiveBeTasks(Properties properties) {
+ super("active_be_tasks", properties);
}
@Override
@@ -45,15 +45,14 @@ public class ActiveQueries extends TableValuedFunction {
protected TableValuedFunctionIf toCatalogFunction() {
try {
Map<String, String> arguments = getTVFProperties().getMap();
- return new ActiveQueriesTableValuedFunction(arguments);
+ return new ActiveBeTasksTableValuedFunction(arguments);
} catch (Throwable t) {
- throw new AnalysisException("Can not build FrontendsTableValuedFunction by "
+ throw new AnalysisException("Can not build ActiveBeTasksTableValuedFunction by "
+ this + ": " + t.getMessage(), t);
}
}
- @Override
public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
- return visitor.visitQueries(this, context);
+ return visitor.visitActiveBeTasks(this, context);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveQueries.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveQueries.java
index f8dcaa4a7ec..1e15b674249 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveQueries.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ActiveQueries.java
@@ -47,7 +47,7 @@ public class ActiveQueries extends TableValuedFunction {
Map<String, String> arguments = getTVFProperties().getMap();
return new ActiveQueriesTableValuedFunction(arguments);
} catch (Throwable t) {
- throw new AnalysisException("Can not build FrontendsTableValuedFunction by "
+ throw new AnalysisException("Can not build ActiveQueriesTableValuedFunction by "
+ this + ": " + t.getMessage(), t);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java
index fba34d48168..36561e5b12c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java
@@ -17,6 +17,7 @@
package org.apache.doris.nereids.trees.expressions.visitor;
+import org.apache.doris.nereids.trees.expressions.functions.table.ActiveBeTasks;
import org.apache.doris.nereids.trees.expressions.functions.table.ActiveQueries;
import org.apache.doris.nereids.trees.expressions.functions.table.Backends;
import org.apache.doris.nereids.trees.expressions.functions.table.Catalogs;
@@ -102,4 +103,8 @@ public interface TableValuedFunctionVisitor<R, C> {
default R visitWorkloadGroups(WorkloadGroups workloadGroups, C context) {
return visitTableValuedFunction(workloadGroups, context);
}
+
+ default R visitActiveBeTasks(ActiveBeTasks beTasks, C context) {
+ return visitTableValuedFunction(beTasks, context);
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveQueriesTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveBeTasksTableValuedFunction.java
similarity index 53%
copy from fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveQueriesTableValuedFunction.java
copy to fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveBeTasksTableValuedFunction.java
index 27f65ed7680..99a8ba4886f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveQueriesTableValuedFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveBeTasksTableValuedFunction.java
@@ -23,74 +23,50 @@ import org.apache.doris.catalog.ScalarType;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.thrift.TMetaScanRange;
import org.apache.doris.thrift.TMetadataType;
-import org.apache.doris.thrift.TQueriesMetadataParams;
import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
import java.util.List;
import java.util.Map;
-public class ActiveQueriesTableValuedFunction extends MetadataTableValuedFunction {
- public static final String NAME = "active_queries";
+public class ActiveBeTasksTableValuedFunction extends MetadataTableValuedFunction {
+
+ public static final String NAME = "active_be_tasks";
private static final ImmutableList<Column> SCHEMA = ImmutableList.of(
- new Column("BeHost", ScalarType.createStringType()),
- new Column("BePort", PrimitiveType.BIGINT),
+ new Column("BeId", PrimitiveType.BIGINT),
+ new Column("FeHost", ScalarType.createStringType()),
new Column("QueryId", ScalarType.createStringType()),
- new Column("StartTime", ScalarType.createStringType()),
- new Column("QueryTimeMs", PrimitiveType.BIGINT),
- new Column("WorkloadGroupId", PrimitiveType.BIGINT),
- new Column("QueryCpuTimeMs", PrimitiveType.BIGINT),
+ new Column("TaskTimeMs", PrimitiveType.BIGINT),
+ new Column("TaskCpuTimeMs", PrimitiveType.BIGINT),
new Column("ScanRows", PrimitiveType.BIGINT),
new Column("ScanBytes", PrimitiveType.BIGINT),
new Column("BePeakMemoryBytes", PrimitiveType.BIGINT),
new Column("CurrentUsedMemoryBytes", PrimitiveType.BIGINT),
new Column("ShuffleSendBytes", PrimitiveType.BIGINT),
- new Column("ShuffleSendRows", PrimitiveType.BIGINT),
- new Column("Database", ScalarType.createStringType()),
- new Column("FrontendInstance", ScalarType.createStringType()),
- new Column("Sql", ScalarType.createStringType()));
-
- private static final ImmutableMap<String, Integer> COLUMN_TO_INDEX;
-
- static {
- ImmutableMap.Builder<String, Integer> builder = new ImmutableMap.Builder();
- for (int i = 0; i < SCHEMA.size(); i++) {
- builder.put(SCHEMA.get(i).getName().toLowerCase(), i);
- }
- COLUMN_TO_INDEX = builder.build();
- }
-
- public static Integer getColumnIndexFromColumnName(String columnName) {
- return COLUMN_TO_INDEX.get(columnName.toLowerCase());
- }
+ new Column("ShuffleSendRows", PrimitiveType.BIGINT));
- public ActiveQueriesTableValuedFunction(Map<String, String> params) throws AnalysisException {
+ public ActiveBeTasksTableValuedFunction(Map<String, String> params) throws AnalysisException {
if (params.size() != 0) {
- throw new AnalysisException("Queries table-valued-function does not support any params");
+ throw new AnalysisException("ActiveBeTasks table-valued-function does not support any params");
}
}
@Override
public TMetadataType getMetadataType() {
- return TMetadataType.QUERIES;
+ return TMetadataType.ACTIVE_BE_TASKS;
}
@Override
public TMetaScanRange getMetaScanRange() {
TMetaScanRange metaScanRange = new TMetaScanRange();
- metaScanRange.setMetadataType(TMetadataType.QUERIES);
- TQueriesMetadataParams queriesMetadataParams = new TQueriesMetadataParams();
- queriesMetadataParams.setClusterName("");
- queriesMetadataParams.setRelayToOtherFe(true);
- metaScanRange.setQueriesParams(queriesMetadataParams);
+ metaScanRange.setMetadataType(TMetadataType.ACTIVE_BE_TASKS);
return metaScanRange;
}
@Override
public String getTableName() {
- return "QueriesTableValuedFunction";
+ return "ActiveBeTasksTableValuedFunction";
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveQueriesTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveQueriesTableValuedFunction.java
index 27f65ed7680..41dd5484dd5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveQueriesTableValuedFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ActiveQueriesTableValuedFunction.java
@@ -68,7 +68,7 @@ public class ActiveQueriesTableValuedFunction extends MetadataTableValuedFunctio
public ActiveQueriesTableValuedFunction(Map<String, String> params) throws AnalysisException {
if (params.size() != 0) {
- throw new AnalysisException("Queries table-valued-function does not support any params");
+ throw new AnalysisException("ActiveQueries table-valued-function does not support any params");
}
}
@@ -90,7 +90,7 @@ public class ActiveQueriesTableValuedFunction extends MetadataTableValuedFunctio
@Override
public String getTableName() {
- return "QueriesTableValuedFunction";
+ return "ActiveQueriesTableValuedFunction";
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
index f9fb76a9666..4b755a97bf8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
@@ -78,6 +78,8 @@ public abstract class TableValuedFunctionIf {
return new ActiveQueriesTableValuedFunction(params);
case WorkloadSchedPolicyTableValuedFunction.NAME:
return new WorkloadSchedPolicyTableValuedFunction(params);
+ case ActiveBeTasksTableValuedFunction.NAME:
+ return new ActiveBeTasksTableValuedFunction(params);
default:
throw new AnalysisException("Could not find table function " + funcName);
}
diff --git a/gensrc/thrift/Types.thrift b/gensrc/thrift/Types.thrift
index 11ec1093da0..a1e414384d3 100644
--- a/gensrc/thrift/Types.thrift
+++ b/gensrc/thrift/Types.thrift
@@ -711,7 +711,8 @@ enum TMetadataType {
JOBS,
TASKS,
QUERIES,
- WORKLOAD_SCHED_POLICY
+ WORKLOAD_SCHED_POLICY,
+ ACTIVE_BE_TASKS,
}
enum TIcebergQueryType {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 37/38: [feat](nereids) support null safe eq runtime filter (FE part) (#31655)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 73f6d9532dce59c4610c297eaa4812e8c2c1a663
Author: minghong <en...@gmail.com>
AuthorDate: Thu Mar 7 14:29:25 2024 +0800
[feat](nereids) support null safe eq runtime filter (FE part) (#31655)
be part has been merged in #31754
---
.../processor/post/RuntimeFilterGenerator.java | 34 +++++------
.../rules/expression/ExpressionOptimization.java | 4 +-
.../expression/rules/NullSafeEqualToEqual.java | 62 +++++++++++++++++++
.../rules/rewrite/FindHashConditionForJoin.java | 3 +-
.../trees/plans/physical/AbstractPhysicalPlan.java | 10 ++++
.../org/apache/doris/planner/HashJoinNode.java | 6 --
.../org/apache/doris/planner/RuntimeFilter.java | 9 +++
.../main/java/org/apache/doris/qe/Coordinator.java | 4 +-
.../expression/rules/NullSafeEqualToEqualTest.java | 69 ++++++++++++++++++++++
.../rules/rewrite/EliminateJoinByUniqueTest.java | 14 -----
10 files changed, 174 insertions(+), 41 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
index f7a7c166bd7..8da3ede4200 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
@@ -24,6 +24,7 @@ import org.apache.doris.nereids.stats.ExpressionEstimation;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.CTEId;
import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.Expression;
@@ -268,25 +269,22 @@ public class RuntimeFilterGenerator extends PlanPostProcessor {
List<Expression> hashJoinConjuncts = join.getHashJoinConjuncts().stream().collect(Collectors.toList());
boolean buildSideContainsConsumer = hasCTEConsumerDescendant((PhysicalPlan) join.right());
for (int i = 0; i < hashJoinConjuncts.size(); i++) {
- // BE do not support RF generated from NullSafeEqual, skip them
- if (hashJoinConjuncts.get(i) instanceof EqualTo) {
- EqualTo equalTo = ((EqualTo) JoinUtils.swapEqualToForChildrenOrder(
- (EqualTo) hashJoinConjuncts.get(i), join.left().getOutputSet()));
- for (TRuntimeFilterType type : legalTypes) {
- //bitmap rf is generated by nested loop join.
- if (type == TRuntimeFilterType.BITMAP) {
- continue;
- }
- long buildSideNdv = getBuildSideNdv(join, equalTo);
- Pair<PhysicalRelation, Slot> pair = ctx.getAliasTransferMap().get(equalTo.right());
- // CteConsumer is not allowed to generate RF in order to avoid RF cycle.
- if ((pair == null && buildSideContainsConsumer)
- || (pair != null && pair.first instanceof PhysicalCTEConsumer)) {
- continue;
- }
- join.pushDownRuntimeFilter(context, generator, join, equalTo.right(),
- equalTo.left(), type, buildSideNdv, i);
+ EqualPredicate equalTo = JoinUtils.swapEqualToForChildrenOrder(
+ (EqualPredicate) hashJoinConjuncts.get(i), join.left().getOutputSet());
+ for (TRuntimeFilterType type : legalTypes) {
+ //bitmap rf is generated by nested loop join.
+ if (type == TRuntimeFilterType.BITMAP) {
+ continue;
+ }
+ long buildSideNdv = getBuildSideNdv(join, equalTo);
+ Pair<PhysicalRelation, Slot> pair = ctx.getAliasTransferMap().get(equalTo.right());
+ // CteConsumer is not allowed to generate RF in order to avoid RF cycle.
+ if ((pair == null && buildSideContainsConsumer)
+ || (pair != null && pair.first instanceof PhysicalCTEConsumer)) {
+ continue;
}
+ join.pushDownRuntimeFilter(context, generator, join, equalTo.right(),
+ equalTo.left(), type, buildSideNdv, i);
}
}
return join;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionOptimization.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionOptimization.java
index e7b3a308f0f..fdf9820c582 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionOptimization.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionOptimization.java
@@ -22,6 +22,7 @@ import org.apache.doris.nereids.rules.expression.rules.CaseWhenToIf;
import org.apache.doris.nereids.rules.expression.rules.DateFunctionRewrite;
import org.apache.doris.nereids.rules.expression.rules.DistinctPredicatesRule;
import org.apache.doris.nereids.rules.expression.rules.ExtractCommonFactorRule;
+import org.apache.doris.nereids.rules.expression.rules.NullSafeEqualToEqual;
import org.apache.doris.nereids.rules.expression.rules.OrToIn;
import org.apache.doris.nereids.rules.expression.rules.SimplifyComparisonPredicate;
import org.apache.doris.nereids.rules.expression.rules.SimplifyDecimalV3Comparison;
@@ -48,7 +49,8 @@ public class ExpressionOptimization extends ExpressionRewrite {
OrToIn.INSTANCE,
ArrayContainToArrayOverlap.INSTANCE,
CaseWhenToIf.INSTANCE,
- TopnToMax.INSTANCE
+ TopnToMax.INSTANCE,
+ NullSafeEqualToEqual.INSTANCE
);
private static final ExpressionRuleExecutor EXECUTOR = new ExpressionRuleExecutor(OPTIMIZE_REWRITE_RULES);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqual.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqual.java
new file mode 100644
index 00000000000..c215e65f722
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqual.java
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteRule;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NullSafeEqual;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+/**
+ * convert "<=>" to "=", if any side is not nullable
+ * convert "A <=> null" to "A is null"
+ */
+public class NullSafeEqualToEqual extends DefaultExpressionRewriter<ExpressionRewriteContext> implements
+ ExpressionRewriteRule<ExpressionRewriteContext> {
+ public static final NullSafeEqualToEqual INSTANCE = new NullSafeEqualToEqual();
+
+ @Override
+ public Expression rewrite(Expression expr, ExpressionRewriteContext ctx) {
+ return expr.accept(this, null);
+ }
+
+ @Override
+ public Expression visitNullSafeEqual(NullSafeEqual nullSafeEqual, ExpressionRewriteContext ctx) {
+ if (nullSafeEqual.left() instanceof NullLiteral) {
+ if (nullSafeEqual.right().nullable()) {
+ return new IsNull(nullSafeEqual.right());
+ } else {
+ return BooleanLiteral.FALSE;
+ }
+ } else if (nullSafeEqual.right() instanceof NullLiteral) {
+ if (nullSafeEqual.left().nullable()) {
+ return new IsNull(nullSafeEqual.left());
+ } else {
+ return BooleanLiteral.FALSE;
+ }
+ } else if (!nullSafeEqual.left().nullable() || !nullSafeEqual.right().nullable()) {
+ return new EqualTo(nullSafeEqual.left(), nullSafeEqual.right());
+ }
+ return nullSafeEqual;
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/FindHashConditionForJoin.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/FindHashConditionForJoin.java
index 2826d709065..86bcdf4487b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/FindHashConditionForJoin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/FindHashConditionForJoin.java
@@ -63,7 +63,8 @@ public class FindHashConditionForJoin extends OneRewriteRuleFactory {
}
List<Expression> combinedHashJoinConjuncts = Streams
- .concat(join.getHashJoinConjuncts().stream(), extractedHashJoinConjuncts.stream())
+ .concat(join.getHashJoinConjuncts().stream(),
+ extractedHashJoinConjuncts.stream())
.distinct()
.collect(ImmutableList.toImmutableList());
JoinType joinType = join.getJoinType();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
index 1e9135d600e..9f71bda0b40 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
@@ -24,7 +24,9 @@ import org.apache.doris.nereids.processor.post.RuntimeFilterContext;
import org.apache.doris.nereids.processor.post.RuntimeFilterGenerator;
import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NullSafeEqual;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.AbstractPlan;
import org.apache.doris.nereids.trees.plans.Explainable;
@@ -131,6 +133,14 @@ public abstract class AbstractPhysicalPlan extends AbstractPlan implements Physi
ctx.setTargetsOnScanNode(ctx.getAliasTransferPair(probeSlot).first, scanSlot);
}
} else {
+ // null safe equal runtime filter only support bloom filter
+ EqualPredicate eq = (EqualPredicate) builderNode.getHashJoinConjuncts().get(exprOrder);
+ if (eq instanceof NullSafeEqual && type == TRuntimeFilterType.IN_OR_BLOOM) {
+ type = TRuntimeFilterType.BLOOM;
+ }
+ if (eq instanceof NullSafeEqual && type != TRuntimeFilterType.BLOOM) {
+ return false;
+ }
filter = new RuntimeFilter(generator.getNextId(),
src, ImmutableList.of(scanSlot), ImmutableList.of(probeExpr),
type, exprOrder, builderNode, buildSideNdv,
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
index 1413d42ceee..d8cc4a77a0a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
@@ -199,12 +199,6 @@ public class HashJoinNode extends JoinNodeBase {
for (Expr eqJoinPredicate : eqJoinConjuncts) {
Preconditions.checkArgument(eqJoinPredicate instanceof BinaryPredicate);
BinaryPredicate eqJoin = (BinaryPredicate) eqJoinPredicate;
- if (eqJoin.getOp().equals(BinaryPredicate.Operator.EQ_FOR_NULL)) {
- Preconditions.checkArgument(eqJoin.getChildren().size() == 2);
- if (!eqJoin.getChild(0).isNullable() || !eqJoin.getChild(1).isNullable()) {
- eqJoin.setOp(BinaryPredicate.Operator.EQ);
- }
- }
this.eqJoinConjuncts.add(eqJoin);
}
this.distrMode = DistributionMode.NONE;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/RuntimeFilter.java b/fe/fe-core/src/main/java/org/apache/doris/planner/RuntimeFilter.java
index ade087e2de0..00117beae72 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/RuntimeFilter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/RuntimeFilter.java
@@ -230,6 +230,15 @@ public final class RuntimeFilter {
}
tFilter.setOptRemoteRf(hasRemoteTargets);
tFilter.setBloomFilterSizeCalculatedByNdv(bloomFilterSizeCalculatedByNdv);
+ if (builderNode instanceof HashJoinNode) {
+ HashJoinNode join = (HashJoinNode) builderNode;
+ BinaryPredicate eq = join.getEqJoinConjuncts().get(exprOrder);
+ if (eq.getOp().equals(BinaryPredicate.Operator.EQ_FOR_NULL)) {
+ tFilter.setNullAware(true);
+ } else {
+ tFilter.setNullAware(false);
+ }
+ }
return tFilter;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index 64f32ed2e24..872a149b176 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -94,6 +94,7 @@ import org.apache.doris.thrift.TPipelineFragmentParams;
import org.apache.doris.thrift.TPipelineFragmentParamsList;
import org.apache.doris.thrift.TPipelineInstanceParams;
import org.apache.doris.thrift.TPipelineWorkloadGroup;
+import org.apache.doris.thrift.TPlanFragment;
import org.apache.doris.thrift.TPlanFragmentDestination;
import org.apache.doris.thrift.TPlanFragmentExecParams;
import org.apache.doris.thrift.TQueryGlobals;
@@ -3703,6 +3704,7 @@ public class Coordinator implements CoordInterface {
Map<TNetworkAddress, TPipelineFragmentParams> res = new HashMap();
Map<TNetworkAddress, Integer> instanceIdx = new HashMap();
+ TPlanFragment fragmentThrift = fragment.toThrift();
for (int i = 0; i < instanceExecParams.size(); ++i) {
final FInstanceExecParam instanceExecParam = instanceExecParams.get(i);
Map<Integer, List<TScanRangeParams>> scanRanges = instanceExecParam.perNodeScanRanges;
@@ -3728,7 +3730,7 @@ public class Coordinator implements CoordInterface {
params.query_options.setMemLimit(memLimit);
params.setSendQueryStatisticsWithEveryBatch(
fragment.isTransferQueryStatisticsWithEveryBatch());
- params.setFragment(fragment.toThrift());
+ params.setFragment(fragmentThrift);
params.setLocalParams(Lists.newArrayList());
if (tWorkloadGroups != null) {
params.setWorkloadGroups(tWorkloadGroups);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqualTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqualTest.java
new file mode 100644
index 00000000000..3a2b4eb2a6e
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqualTest.java
@@ -0,0 +1,69 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteTestHelper;
+import org.apache.doris.nereids.rules.expression.ExpressionRuleExecutor;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NullSafeEqual;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
+import org.apache.doris.nereids.types.StringType;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.jupiter.api.Test;
+
+class NullSafeEqualToEqualTest extends ExpressionRewriteTestHelper {
+
+ // "A<=> Null" to "A is null"
+ @Test
+ void testNullSafeEqualToIsNull() {
+ executor = new ExpressionRuleExecutor(ImmutableList.of(NullSafeEqualToEqual.INSTANCE));
+ SlotReference slot = new SlotReference("a", StringType.INSTANCE, true);
+ assertRewrite(new NullSafeEqual(slot, NullLiteral.INSTANCE), new IsNull(slot));
+ }
+
+ // "A<=> Null" to "False", when A is not nullable
+ @Test
+ void testNullSafeEqualToFalse() {
+ executor = new ExpressionRuleExecutor(ImmutableList.of(NullSafeEqualToEqual.INSTANCE));
+ SlotReference slot = new SlotReference("a", StringType.INSTANCE, false);
+ assertRewrite(new NullSafeEqual(slot, NullLiteral.INSTANCE), BooleanLiteral.FALSE);
+ }
+
+ // "A<=> "abc" to "A = "abc"
+ @Test
+ void testNullSafeEqualToEqual() {
+ executor = new ExpressionRuleExecutor(ImmutableList.of(NullSafeEqualToEqual.INSTANCE));
+ SlotReference slot = new SlotReference("a", StringType.INSTANCE, true);
+ StringLiteral str = new StringLiteral("abc");
+ assertRewrite(new NullSafeEqual(slot, str), new EqualTo(slot, str));
+ }
+
+ // "A<=>B" not changed
+ @Test
+ void testNullSafeEqualNotChanged() {
+ executor = new ExpressionRuleExecutor(ImmutableList.of(NullSafeEqualToEqual.INSTANCE));
+ SlotReference a = new SlotReference("a", StringType.INSTANCE, true);
+ SlotReference b = new SlotReference("b", StringType.INSTANCE, true);
+ assertRewrite(new NullSafeEqual(a, b), new NullSafeEqual(a, b));
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByUniqueTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByUniqueTest.java
index c3b9c9de005..778b65f992c 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByUniqueTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByUniqueTest.java
@@ -56,13 +56,6 @@ class EliminateJoinByUniqueTest extends TestWithFeService implements MemoPattern
.nonMatch(logicalJoin())
.printlnTree();
- sql = "select t1.id1 from t1 left outer join t2 on t1.id1 <=> t2.id2";
- PlanChecker.from(connectContext)
- .analyze(sql)
- .rewrite()
- .matches(logicalJoin())
- .printlnTree();
-
sql = "select t2.id2 from t1 left outer join t2 on t1.id1 = t2.id2";
PlanChecker.from(connectContext)
.analyze(sql)
@@ -80,13 +73,6 @@ class EliminateJoinByUniqueTest extends TestWithFeService implements MemoPattern
.nonMatch(logicalJoin())
.printlnTree();
- sql = "select t1.id1 from t1 left outer join t2 on t1.id_null <=> t2.id2";
- PlanChecker.from(connectContext)
- .analyze(sql)
- .rewrite()
- .matches(logicalJoin())
- .printlnTree();
-
sql = "select t2.id2 from t1 left outer join t2 on t1.id_null = t2.id2";
PlanChecker.from(connectContext)
.analyze(sql)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 08/38: [feat](Nereids): add struct info map in group (#31800)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d0847a895312be11d9c76f359a63566d56de6508
Author: 谢健 <ji...@gmail.com>
AuthorDate: Wed Mar 6 16:49:04 2024 +0800
[feat](Nereids): add struct info map in group (#31800)
---
.../org/apache/doris/mtmv/MTMVRelationManager.java | 10 +-
.../java/org/apache/doris/nereids/memo/Group.java | 7 +-
.../apache/doris/nereids/memo/StructInfoMap.java | 123 +++++++++++++++++++++
.../mv/InitMaterializationContextHook.java | 7 +-
.../doris/nereids/memo/StructInfoMapTest.java | 81 ++++++++++++++
.../org/apache/doris/nereids/util/PlanChecker.java | 2 +
.../apache/doris/utframe/TestWithFeService.java | 28 +++++
7 files changed, 252 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
index aa7ffd2426d..723deaff740 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
@@ -35,6 +35,7 @@ import org.apache.doris.nereids.trees.plans.commands.info.TableNameInfo;
import org.apache.doris.persist.AlterMTMV;
import org.apache.doris.qe.ConnectContext;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
@@ -71,8 +72,7 @@ public class MTMVRelationManager implements MTMVHookService {
for (BaseTableInfo tableInfo : mvInfos) {
try {
MTMV mtmv = (MTMV) MTMVUtil.getTable(tableInfo);
- if (!CollectionUtils
- .isEmpty(MTMVRewriteUtil.getMTMVCanRewritePartitions(mtmv, ctx, System.currentTimeMillis()))) {
+ if (isMVPartitionValid(mtmv, ctx)) {
res.add(mtmv);
}
} catch (AnalysisException e) {
@@ -83,6 +83,12 @@ public class MTMVRelationManager implements MTMVHookService {
return res;
}
+ @VisibleForTesting
+ public boolean isMVPartitionValid(MTMV mtmv, ConnectContext ctx) {
+ return !CollectionUtils
+ .isEmpty(MTMVRewriteUtil.getMTMVCanRewritePartitions(mtmv, ctx, System.currentTimeMillis()));
+ }
+
private Set<BaseTableInfo> getMTMVInfos(List<BaseTableInfo> tableInfos) {
Set<BaseTableInfo> mvInfos = Sets.newHashSet();
for (BaseTableInfo tableInfo : tableInfos) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
index 2a0e83c72a4..5a5abd56f95 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.memo;
import org.apache.doris.common.Pair;
import org.apache.doris.nereids.cost.Cost;
-import org.apache.doris.nereids.jobs.joinorder.hypergraph.HyperGraph;
import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.rules.exploration.mv.StructInfo;
@@ -78,6 +77,8 @@ public class Group {
private List<StructInfo> structInfos = new ArrayList<>();
+ private StructInfoMap structInfoMap = new StructInfoMap();
+
/**
* Constructor for Group.
*
@@ -418,8 +419,8 @@ public class Group {
return false;
}
- public List<HyperGraph> getHyperGraphs() {
- return new ArrayList<>();
+ public StructInfoMap getstructInfoMap() {
+ return structInfoMap;
}
public boolean isProjectGroup() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java
new file mode 100644
index 00000000000..f7e09fd44d9
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.memo;
+
+import org.apache.doris.nereids.rules.exploration.mv.StructInfo;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
+
+import com.google.common.collect.Sets;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+
+/**
+ * Representation for group in cascades optimizer.
+ */
+public class StructInfoMap {
+ private final Map<BitSet, GroupExpression> groupExpressionMap = new HashMap<>();
+ private final Map<BitSet, StructInfo> infoMap = new HashMap<>();
+
+ /**
+ * get struct info according to table map
+ * @param mvTableMap the original table map
+ * @param foldTableMap the fold table map
+ * @param group the group that the mv matched
+ * @return struct info or null if not found
+ */
+ public @Nullable StructInfo getStructInfo(BitSet mvTableMap, BitSet foldTableMap, Group group) {
+ if (!infoMap.containsKey(mvTableMap)) {
+ if ((groupExpressionMap.containsKey(foldTableMap) || groupExpressionMap.isEmpty())
+ && !groupExpressionMap.containsKey(mvTableMap)) {
+ refresh(group);
+ }
+ if (groupExpressionMap.containsKey(mvTableMap)) {
+ StructInfo structInfo = constructStructInfo(groupExpressionMap.get(mvTableMap));
+ infoMap.put(mvTableMap, structInfo);
+ }
+ }
+
+ return infoMap.get(mvTableMap);
+ }
+
+ public Set<BitSet> getTableMaps() {
+ return groupExpressionMap.keySet();
+ }
+
+ private StructInfo constructStructInfo(GroupExpression groupExpression) {
+ throw new RuntimeException("has not been implemented for" + groupExpression);
+ }
+
+ /**
+ * refresh group expression map
+ * @param group the root group
+ */
+ public void refresh(Group group) {
+ List<Set<BitSet>> childrenTableMap = new ArrayList<>();
+ Set<Group> refreshedGroup = new HashSet<>();
+ for (GroupExpression groupExpression : group.getLogicalExpressions()) {
+ if (groupExpression.children().isEmpty()) {
+ groupExpressionMap.put(constructLeaf(groupExpression), groupExpression);
+ continue;
+ }
+ for (Group child : groupExpression.children()) {
+ if (!refreshedGroup.contains(child)) {
+ StructInfoMap childStructInfoMap = child.getstructInfoMap();
+ childStructInfoMap.refresh(child);
+ }
+ refreshedGroup.add(child);
+ childrenTableMap.add(child.getstructInfoMap().getTableMaps());
+ }
+ Set<BitSet> bitSets = cartesianProduct(childrenTableMap);
+ for (BitSet bitSet : bitSets) {
+ groupExpressionMap.put(bitSet, groupExpression);
+ }
+ }
+ }
+
+ private BitSet constructLeaf(GroupExpression groupExpression) {
+ Plan plan = groupExpression.getPlan();
+ BitSet tableMap = new BitSet();
+ if (plan instanceof LogicalCatalogRelation) {
+ // TODO: Bitmap is not compatible with long, use tree map instead
+ tableMap.set((int) ((LogicalCatalogRelation) plan).getTable().getId());
+ }
+ // one row relation / CTE consumer
+ return tableMap;
+ }
+
+ private Set<BitSet> cartesianProduct(List<Set<BitSet>> childrenTableMap) {
+ return Sets.cartesianProduct(childrenTableMap)
+ .stream()
+ .map(bitSetList -> {
+ BitSet bitSet = new BitSet();
+ for (BitSet b : bitSetList) {
+ bitSet.or(b);
+ }
+ return bitSet;
+ })
+ .collect(Collectors.toSet());
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
index baf714fe280..8046639d1ba 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
@@ -32,6 +32,7 @@ import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.visitor.TableCollector;
import org.apache.doris.nereids.trees.plans.visitor.TableCollector.TableCollectorContext;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
@@ -56,7 +57,11 @@ public class InitMaterializationContextHook implements PlannerHook {
initMaterializationContext(planner.getCascadesContext());
}
- private void initMaterializationContext(CascadesContext cascadesContext) {
+ /**
+ * init materialization context
+ */
+ @VisibleForTesting
+ public void initMaterializationContext(CascadesContext cascadesContext) {
if (!cascadesContext.getConnectContext().getSessionVariable().isEnableMaterializedViewRewrite()) {
return;
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/memo/StructInfoMapTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/memo/StructInfoMapTest.java
new file mode 100644
index 00000000000..0699f04b92f
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/memo/StructInfoMapTest.java
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.memo;
+
+import org.apache.doris.catalog.MTMV;
+import org.apache.doris.mtmv.MTMVRelationManager;
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.sqltest.SqlTestBase;
+import org.apache.doris.nereids.util.PlanChecker;
+import org.apache.doris.qe.ConnectContext;
+
+import mockit.Mock;
+import mockit.MockUp;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.BitSet;
+import java.util.Set;
+
+class StructInfoMapTest extends SqlTestBase {
+ @Test
+ void testTableMap() throws Exception {
+ CascadesContext c1 = createCascadesContext(
+ "select T1.id from T1 inner join T2 "
+ + "on T1.id = T2.id "
+ + "inner join T3 on T1.id = T3.id",
+ connectContext
+ );
+ PlanChecker.from(c1)
+ .analyze()
+ .rewrite()
+ .optimize();
+ Group root = c1.getMemo().getRoot();
+ Set<BitSet> tableMaps = root.getstructInfoMap().getTableMaps();
+ Assertions.assertTrue(tableMaps.isEmpty());
+ root.getstructInfoMap().refresh(root);
+ Assertions.assertEquals(1, tableMaps.size());
+ new MockUp<MTMVRelationManager>() {
+ @Mock
+ public boolean isMVPartitionValid(MTMV mtmv, ConnectContext ctx) {
+ return true;
+ }
+ };
+ connectContext.getSessionVariable().enableMaterializedViewRewrite = true;
+ createMvByNereids("create materialized view mv1 BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL\n"
+ + " DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + " PROPERTIES ('replication_num' = '1') \n"
+ + " as select T1.id from T1 inner join T2 "
+ + "on T1.id = T2.id;");
+ c1 = createCascadesContext(
+ "select T1.id from T1 inner join T2 "
+ + "on T1.id = T2.id "
+ + "inner join T3 on T1.id = T3.id",
+ connectContext
+ );
+ PlanChecker.from(c1)
+ .analyze()
+ .rewrite()
+ .optimize()
+ .printlnBestPlanTree();
+ root = c1.getMemo().getRoot();
+ root.getstructInfoMap().refresh(root);
+ tableMaps = root.getstructInfoMap().getTableMaps();
+ Assertions.assertEquals(2, tableMaps.size());
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanChecker.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanChecker.java
index 0c6cc0e577e..9471e18f6a5 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanChecker.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanChecker.java
@@ -49,6 +49,7 @@ import org.apache.doris.nereids.rules.RuleFactory;
import org.apache.doris.nereids.rules.RuleSet;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.rules.analysis.BindRelation.CustomTableResolver;
+import org.apache.doris.nereids.rules.exploration.mv.InitMaterializationContextHook;
import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.Plan;
@@ -121,6 +122,7 @@ public class PlanChecker {
public PlanChecker analyze() {
this.cascadesContext.newAnalyzer().analyze();
this.cascadesContext.toMemo();
+ InitMaterializationContextHook.INSTANCE.initMaterializationContext(this.cascadesContext);
return this;
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java b/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java
index 58b55f0a7b5..7de8a9a830b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java
@@ -58,6 +58,7 @@ import org.apache.doris.common.FeConstants;
import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.common.util.SqlParserUtils;
import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.job.base.AbstractJob;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.glue.LogicalPlanAdapter;
@@ -65,10 +66,13 @@ import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.StatementScopeIdGenerator;
import org.apache.doris.nereids.trees.plans.commands.AddConstraintCommand;
+import org.apache.doris.nereids.trees.plans.commands.CreateMTMVCommand;
import org.apache.doris.nereids.trees.plans.commands.CreateTableCommand;
import org.apache.doris.nereids.trees.plans.commands.DropConstraintCommand;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import org.apache.doris.nereids.util.MemoTestUtils;
+import org.apache.doris.persist.CreateTableInfo;
+import org.apache.doris.persist.EditLog;
import org.apache.doris.planner.Planner;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.DdlExecutor;
@@ -795,6 +799,30 @@ public abstract class TestWithFeService {
Thread.sleep(100);
}
+ protected void createMvByNereids(String sql) throws Exception {
+ new MockUp<EditLog>() {
+ @Mock
+ public void logCreateTable(CreateTableInfo info) {
+ System.out.println("skip log create table...");
+ }
+
+ @Mock
+ public void logCreateJob(AbstractJob job) {
+ System.out.println("skip log create job...");
+ }
+ };
+ NereidsParser nereidsParser = new NereidsParser();
+ LogicalPlan parsed = nereidsParser.parseSingle(sql);
+ StmtExecutor stmtExecutor = new StmtExecutor(connectContext, sql);
+ if (parsed instanceof CreateMTMVCommand) {
+ ((CreateMTMVCommand) parsed).run(connectContext, stmtExecutor);
+ }
+ checkAlterJob();
+ // waiting table state to normal
+ Thread.sleep(1000);
+
+ }
+
private void updateReplicaPathHash() {
com.google.common.collect.Table<Long, Long, Replica> replicaMetaTable = Env.getCurrentInvertedIndex()
.getReplicaMetaTable();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 20/38: [enhancement](cloud) add CloudTabletStatMgr to capture stats in cloud mode (#31818)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 54a8c83f6a187d2c2392594fbf62cad292c48966
Author: zhengyu <fr...@gmail.com>
AuthorDate: Wed Mar 6 21:57:51 2024 +0800
[enhancement](cloud) add CloudTabletStatMgr to capture stats in cloud mode (#31818)
Signed-off-by: freemandealer <fr...@gmail.com>
---
.../main/java/org/apache/doris/common/Config.java | 3 +
.../apache/doris/catalog/CloudTabletStatMgr.java | 273 +++++++++++++++++++++
.../java/org/apache/doris/catalog/Database.java | 5 +
.../main/java/org/apache/doris/catalog/Env.java | 9 +-
.../java/org/apache/doris/catalog/Replica.java | 17 ++
5 files changed, 306 insertions(+), 1 deletion(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index cbbf9368720..418122745d4 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2583,6 +2583,9 @@ public class Config extends ConfigBase {
@ConfField(mutable = true)
public static int cloud_cold_read_percent = 10; // 10%
+ @ConfField(mutable = true)
+ public static int get_tablet_stat_batch_size = 1000;
+
// The original meta read lock is not enough to keep a snapshot of partition versions,
// so the execution of `createScanRangeLocations` are delayed to `Coordinator::exec`,
// to help to acquire a snapshot of partition versions.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
new file mode 100644
index 00000000000..136c9264a4e
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
@@ -0,0 +1,273 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.catalog.MaterializedIndex.IndexExtState;
+import org.apache.doris.catalog.TableIf.TableType;
+import org.apache.doris.cloud.proto.Cloud.GetTabletStatsRequest;
+import org.apache.doris.cloud.proto.Cloud.GetTabletStatsResponse;
+import org.apache.doris.cloud.proto.Cloud.MetaServiceCode;
+import org.apache.doris.cloud.proto.Cloud.TabletIndexPB;
+import org.apache.doris.cloud.proto.Cloud.TabletStatsPB;
+import org.apache.doris.cloud.rpc.MetaServiceProxy;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.rpc.RpcException;
+
+import lombok.Getter;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ForkJoinPool;
+
+/*
+ * CloudTabletStatMgr is for collecting tablet(replica) statistics from backends.
+ * Each FE will collect by itself.
+ */
+public class CloudTabletStatMgr extends MasterDaemon {
+ private static final Logger LOG = LogManager.getLogger(CloudTabletStatMgr.class);
+
+ private ForkJoinPool taskPool = new ForkJoinPool(Runtime.getRuntime().availableProcessors());
+
+ // <(dbId, tableId) -> CloudTableStats>
+ private ConcurrentHashMap<Pair<Long, Long>, CloudTableStats> cloudTableStatsMap = new ConcurrentHashMap<>();
+
+ public CloudTabletStatMgr() {
+ super("cloud tablet stat mgr", Config.tablet_stat_update_interval_second * 1000);
+ }
+
+ @Override
+ protected void runAfterCatalogReady() {
+ LOG.info("cloud tablet stat begin");
+ long start = System.currentTimeMillis();
+
+ List<GetTabletStatsRequest> reqList = new ArrayList<GetTabletStatsRequest>();
+ GetTabletStatsRequest.Builder builder = GetTabletStatsRequest.newBuilder();
+ List<Long> dbIds = Env.getCurrentInternalCatalog().getDbIds();
+ for (Long dbId : dbIds) {
+ Database db = Env.getCurrentInternalCatalog().getDbNullable(dbId);
+ if (db == null) {
+ continue;
+ }
+
+ List<Table> tableList = db.getTables();
+ for (Table table : tableList) {
+ if (table.getType() != TableType.OLAP) {
+ continue;
+ }
+
+ table.readLock();
+ try {
+ OlapTable tbl = (OlapTable) table;
+ for (Partition partition : tbl.getAllPartitions()) {
+ for (MaterializedIndex index : partition.getMaterializedIndices(IndexExtState.VISIBLE)) {
+ for (Long tabletId : index.getTabletIdsInOrder()) {
+ Tablet tablet = index.getTablet(tabletId);
+ TabletIndexPB.Builder tabletBuilder = TabletIndexPB.newBuilder();
+ tabletBuilder.setDbId(dbId);
+ tabletBuilder.setTableId(table.getId());
+ tabletBuilder.setIndexId(index.getId());
+ tabletBuilder.setPartitionId(partition.getId());
+ tabletBuilder.setTabletId(tablet.getId());
+ builder.addTabletIdx(tabletBuilder);
+
+ if (builder.getTabletIdxCount() >= Config.get_tablet_stat_batch_size) {
+ reqList.add(builder.build());
+ builder = GetTabletStatsRequest.newBuilder();
+ }
+ }
+ }
+ } // partitions
+ } finally {
+ table.readUnlock();
+ }
+ } // tables
+ } // end for dbs
+
+ if (builder.getTabletIdxCount() > 0) {
+ reqList.add(builder.build());
+ }
+
+ for (GetTabletStatsRequest req : reqList) {
+ GetTabletStatsResponse resp;
+ try {
+ resp = getTabletStats(req);
+ } catch (RpcException e) {
+ LOG.info("get tablet stats exception:", e);
+ continue;
+ }
+
+ if (resp.getStatus().getCode() != MetaServiceCode.OK) {
+ continue;
+ }
+
+ if (LOG.isDebugEnabled()) {
+ int i = 0;
+ for (TabletIndexPB idx : req.getTabletIdxList()) {
+ LOG.debug("db_id: {} table_id: {} index_id: {} tablet_id: {} size: {}",
+ idx.getDbId(), idx.getTableId(), idx.getIndexId(), idx.getTabletId(),
+ resp.getTabletStats(i++).getDataSize());
+ }
+ }
+ updateTabletStat(resp);
+ }
+
+ LOG.info("finished to get tablet stat of all backends. cost: {} ms",
+ (System.currentTimeMillis() - start));
+
+ // after update replica in all backends, update index row num
+ start = System.currentTimeMillis();
+ ConcurrentHashMap<Pair<Long, Long>, CloudTableStats> newCloudTableStatsMap = new ConcurrentHashMap<>();
+ for (Long dbId : dbIds) {
+ Database db = Env.getCurrentInternalCatalog().getDbNullable(dbId);
+ if (db == null) {
+ continue;
+ }
+
+ List<Table> tableList = db.getTables();
+ for (Table table : tableList) {
+ if (table.getType() != TableType.OLAP) {
+ continue;
+ }
+ OlapTable olapTable = (OlapTable) table;
+
+ String dbName = db.getName();
+ Long tableId = table.getId();
+ String tableName = table.getName();
+
+ Long tableDataSize = 0L;
+ Long tableRowsetCount = 0L;
+ Long tableSegmentCount = 0L;
+ Long tableRowCount = 0L;
+
+ if (!table.writeLockIfExist()) {
+ continue;
+ }
+
+ try {
+ for (Partition partition : olapTable.getAllPartitions()) {
+ for (MaterializedIndex index : partition.getMaterializedIndices(IndexExtState.VISIBLE)) {
+ long indexRowCount = 0L;
+ for (Tablet tablet : index.getTablets()) {
+ long tabletDataSize = 0L;
+ long tabletRowsetCount = 0L;
+ long tabletSegmentCount = 0L;
+ long tabletRowCount = 0L;
+
+ for (Replica replica : tablet.getReplicas()) {
+ if (replica.getDataSize() > tabletDataSize) {
+ tabletDataSize = replica.getDataSize();
+ }
+
+ if (replica.getRowsetCount() > tabletRowsetCount) {
+ tabletRowsetCount = replica.getRowsetCount();
+ }
+
+ if (replica.getSegmentCount() > tabletSegmentCount) {
+ tabletSegmentCount = replica.getSegmentCount();
+ }
+
+ if (replica.getRowCount() > tabletRowCount) {
+ tabletRowCount = replica.getRowCount();
+ }
+ }
+
+ tableDataSize += tabletDataSize;
+ tableRowsetCount += tabletRowsetCount;
+ tableSegmentCount += tabletSegmentCount;
+ tableRowCount += tabletRowCount;
+
+ indexRowCount += tabletRowCount;
+ } // end for tablets
+ index.setRowCount(indexRowCount);
+ } // end for indices
+ } // end for partitions
+ LOG.debug("finished to set row num for table: {} in database: {}",
+ table.getName(), db.getFullName());
+ } finally {
+ table.writeUnlock();
+ }
+
+ newCloudTableStatsMap.put(Pair.of(dbId, tableId), new CloudTableStats(dbName, tableName,
+ tableDataSize, tableRowsetCount, tableSegmentCount, tableRowCount));
+ }
+ }
+ this.cloudTableStatsMap = newCloudTableStatsMap;
+ LOG.info("finished to update index row num of all databases. cost: {} ms",
+ (System.currentTimeMillis() - start));
+ }
+
+ private void updateTabletStat(GetTabletStatsResponse response) {
+ TabletInvertedIndex invertedIndex = Env.getCurrentInvertedIndex();
+ for (TabletStatsPB stat : response.getTabletStatsList()) {
+ if (invertedIndex.getTabletMeta(stat.getIdx().getTabletId()) != null) {
+ List<Replica> replicas = invertedIndex.getReplicasByTabletId(stat.getIdx().getTabletId());
+ if (replicas != null && !replicas.isEmpty() && replicas.get(0) != null) {
+ replicas.get(0).updateCloudStat(stat.getDataSize(), stat.getNumRowsets(),
+ stat.getNumSegments(), stat.getNumRows());
+ }
+ }
+ }
+ }
+
+ private GetTabletStatsResponse getTabletStats(GetTabletStatsRequest request)
+ throws RpcException {
+ GetTabletStatsResponse response;
+ try {
+ response = MetaServiceProxy.getInstance().getTabletStats(request);
+ } catch (RpcException e) {
+ LOG.info("get tablet stat get exception:", e);
+ throw e;
+ }
+ return response;
+ }
+
+ public ConcurrentHashMap<Pair<Long, Long>, CloudTableStats> getCloudTableStatsMap() {
+ return this.cloudTableStatsMap;
+ }
+
+ public static class CloudTableStats {
+ @Getter
+ private String dbName;
+ @Getter
+ private String tableName;
+
+ @Getter
+ private Long tableDataSize;
+ @Getter
+ private Long tableRowsetCount;
+ @Getter
+ private Long tableSegmentCount;
+ @Getter
+ private Long tableRowCount;
+
+ public CloudTableStats(String dbName, String tableName, Long tableDataSize, Long tableRowsetCount,
+ Long tableSegmentCount, Long tableRowCount) {
+ this.dbName = dbName;
+ this.tableName = tableName;
+ this.tableDataSize = tableDataSize;
+ this.tableRowsetCount = tableRowsetCount;
+ this.tableSegmentCount = tableSegmentCount;
+ this.tableRowCount = tableRowCount;
+ }
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
index fd709963a7b..d1f33b2b10c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
@@ -213,6 +213,11 @@ public class Database extends MetaObject implements Writable, DatabaseIf<Table>
return fullQualifiedName;
}
+ public String getName() {
+ String[] strs = fullQualifiedName.split(":");
+ return strs.length == 2 ? strs[1] : strs[0];
+ }
+
public void setNameWithLock(String newName) {
writeLock();
try {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index effd7b6ca64..38fe843d5e7 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -448,6 +448,8 @@ public class Env {
private TabletStatMgr tabletStatMgr;
+ private CloudTabletStatMgr cloudTabletStatMgr;
+
private Auth auth;
private AccessControllerManager accessManager;
@@ -692,6 +694,7 @@ public class Env {
this.globalTransactionMgr = new GlobalTransactionMgr(this);
this.tabletStatMgr = new TabletStatMgr();
+ this.cloudTabletStatMgr = new CloudTabletStatMgr();
this.auth = new Auth();
this.accessManager = new AccessControllerManager(auth);
@@ -1670,7 +1673,11 @@ public class Env {
private void startNonMasterDaemonThreads() {
// start load manager thread
loadManager.start();
- tabletStatMgr.start();
+ if (Config.isNotCloudMode()) {
+ tabletStatMgr.start();
+ } else {
+ cloudTabletStatMgr.start();
+ }
// load and export job label cleaner thread
labelCleaner.start();
// es repository
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java
index 0dad3612023..213fc94ec38 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java
@@ -159,6 +159,8 @@ public class Replica implements Writable {
*/
private long preWatermarkTxnId = -1;
private long postWatermarkTxnId = -1;
+ private long segmentCount = 0L;
+ private long rowsetCount = 0L;
private long userDropTime = -1;
@@ -241,6 +243,14 @@ public class Replica implements Writable {
return rowCount;
}
+ public long getSegmentCount() {
+ return segmentCount;
+ }
+
+ public long getRowsetCount() {
+ return rowsetCount;
+ }
+
public long getLastFailedVersion() {
return lastFailedVersion;
}
@@ -334,6 +344,13 @@ public class Replica implements Writable {
this.versionCount = versionCount;
}
+ public synchronized void updateCloudStat(long dataSize, long rowsetNum, long segmentNum, long rowNum) {
+ this.dataSize = dataSize;
+ this.rowsetCount = rowsetNum;
+ this.segmentCount = segmentNum;
+ this.rowCount = rowNum;
+ }
+
public synchronized void updateVersionInfo(long newVersion, long newDataSize, long newRemoteDataSize,
long newRowCount) {
updateReplicaInfo(newVersion, this.lastFailedVersion, this.lastSuccessVersion, newDataSize, newRemoteDataSize,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 27/38: [fix](http stream) http stream support memtable_on_sink_node header (#31866)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6d0c1ad6328b4443934dfa489609cc9c8148c669
Author: meiyi <my...@gmail.com>
AuthorDate: Thu Mar 7 11:13:31 2024 +0800
[fix](http stream) http stream support memtable_on_sink_node header (#31866)
---
be/src/http/action/http_stream.cpp | 4 ++++
.../apache/doris/service/FrontendServiceImpl.java | 22 +++++++++++++++++-----
2 files changed, 21 insertions(+), 5 deletions(-)
diff --git a/be/src/http/action/http_stream.cpp b/be/src/http/action/http_stream.cpp
index bb5ce729df8..272ef928a4e 100644
--- a/be/src/http/action/http_stream.cpp
+++ b/be/src/http/action/http_stream.cpp
@@ -294,6 +294,10 @@ Status HttpStreamAction::process_put(HttpRequest* http_req,
TStreamLoadPutRequest request;
if (http_req != nullptr) {
request.__set_load_sql(http_req->header(HTTP_SQL));
+ if (!http_req->header(HTTP_MEMTABLE_ON_SINKNODE).empty()) {
+ bool value = iequal(http_req->header(HTTP_MEMTABLE_ON_SINKNODE), "true");
+ request.__set_memtable_on_sink_node(value);
+ }
} else {
request.__set_token(ctx->auth.token);
request.__set_load_sql(ctx->sql_str);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
index 1316425df4b..97b6f52e331 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
@@ -2036,6 +2036,11 @@ public class FrontendServiceImpl implements FrontendService.Iface {
request.getTbl(),
request.getUserIp(), PrivPredicate.LOAD);
}
+ if (request.isSetMemtableOnSinkNode()) {
+ ctx.getSessionVariable().enableMemtableOnSinkNode = request.isMemtableOnSinkNode();
+ } else {
+ ctx.getSessionVariable().enableMemtableOnSinkNode = Config.stream_load_default_memtable_on_sink_node;
+ }
SqlScanner input = new SqlScanner(new StringReader(originStmt), ctx.getSessionVariable().getSqlMode());
SqlParser parser = new SqlParser(input);
try {
@@ -2057,6 +2062,13 @@ public class FrontendServiceImpl implements FrontendService.Iface {
Coordinator coord = new Coordinator(ctx, analyzer, executor.planner());
coord.setLoadMemLimit(request.getExecMemLimit());
coord.setQueryType(TQueryType.LOAD);
+ Table table = parsedStmt.getTargetTable();
+ if (table instanceof OlapTable) {
+ boolean isEnableMemtableOnSinkNode =
+ ((OlapTable) table).getTableProperty().getUseSchemaLightChange()
+ ? coord.getQueryOptions().isEnableMemtableOnSinkNode() : false;
+ coord.getQueryOptions().setEnableMemtableOnSinkNode(isEnableMemtableOnSinkNode);
+ }
TExecPlanFragmentParams plan = coord.getStreamLoadPlan();
int loadStreamPerNode = 20;
@@ -2073,11 +2085,11 @@ public class FrontendServiceImpl implements FrontendService.Iface {
// The txn_id here is obtained from the NativeInsertStmt
result.getParams().setTxnConf(new TTxnParams().setTxnId(txn_id));
result.getParams().setImportLabel(parsedStmt.getLabel());
- result.setDbId(parsedStmt.getTargetTable().getDatabase().getId());
- result.setTableId(parsedStmt.getTargetTable().getId());
- result.setBaseSchemaVersion(((OlapTable) parsedStmt.getTargetTable()).getBaseSchemaVersion());
- result.setGroupCommitIntervalMs(((OlapTable) parsedStmt.getTargetTable()).getGroupCommitIntervalMs());
- result.setGroupCommitDataBytes(((OlapTable) parsedStmt.getTargetTable()).getGroupCommitDataBytes());
+ result.setDbId(table.getDatabase().getId());
+ result.setTableId(table.getId());
+ result.setBaseSchemaVersion(((OlapTable) table).getBaseSchemaVersion());
+ result.setGroupCommitIntervalMs(((OlapTable) table).getGroupCommitIntervalMs());
+ result.setGroupCommitDataBytes(((OlapTable) table).getGroupCommitDataBytes());
result.setWaitInternalGroupCommitFinish(Config.wait_internal_group_commit_finish);
} catch (UserException e) {
LOG.warn("exec sql error", e);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 24/38: [type](doc) Standardize the name in the sql-dialect document and modify the format (#31636)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0baa00af1591880d297c5c3ec548cbf6039de4d9
Author: echo-dundun <50...@users.noreply.github.com>
AuthorDate: Thu Mar 7 10:33:30 2024 +0800
[type](doc) Standardize the name in the sql-dialect document and modify the format (#31636)
---
docs/en/docs/lakehouse/sql-dialect.md | 44 ++++++++++++++++--------------
docs/zh-CN/docs/lakehouse/sql-dialect.md | 46 +++++++++++++++++---------------
2 files changed, 49 insertions(+), 41 deletions(-)
diff --git a/docs/en/docs/lakehouse/sql-dialect.md b/docs/en/docs/lakehouse/sql-dialect.md
index c6716205583..e3b16299d2d 100644
--- a/docs/en/docs/lakehouse/sql-dialect.md
+++ b/docs/en/docs/lakehouse/sql-dialect.md
@@ -28,32 +28,46 @@ under the License.
Starting from version 2.1, Doris can support multiple SQL dialects, such as Presto, Trino, Hive, PostgreSQL, Spark, Oracle, Clickhouse, and more. Through this feature, users can directly use the corresponding SQL dialect to query data in Doris, which facilitates users to smoothly migrate their original business to Doris.
-> 1. This function is currently an experimental function. If you encounter any problems during use, you are welcome to provide feedback through the mail group, [Github issue](https://github.com/apache/doris/issues), etc. .
+> 1. This function is currently an experimental function. If you encounter any problems during use, you are welcome to provide feedback through the mail group, [GitHub issue](https://github.com/apache/doris/issues), etc. .
>
> 2. This function only supports query statements and does not support other DDL and DML statements including Explain.
## Deploy service
-1. Download latest [SQL Transform Tool](https://doris-build-1308700295.cos.ap-beijing.myqcloud.com/transform-doris-tool/transform-doris-tool-1.0.0-bin-x86).
+1. Download latest [Doris SQL Convertor](https://doris-build-1308700295.cos.ap-beijing.myqcloud.com/transform-doris-tool/transform-doris-tool-1.0.0-bin-x86).
2. On any FE node, start the service through the following command:
- `nohup ./transform-doris-tool-1.0.0-bin-x86 run --host=0.0.0.0 --port=5001 &`
+ `nohup ./doris-sql-convertor-1.0.1-bin-x86 run --host=0.0.0.0 --port=5001 &`
- > 1. This service is a stateless service and can be started and stopped at any time.
- >
- > 2. `5001` is the service port and can be arbitrarily specified as an available port.
- >
- > 3. It is recommended to start a separate service on each FE node.
+ :::note
+ 1. This service is a stateless service and can be started and stopped at any time.
+
+ 2. `5001` is the service port and can be arbitrarily specified as an available port.
+
+ 3. It is recommended to start a separate service on each FE node.
+ :::
3. Start the Doris cluster (version 2.1 or higher)
4. Set the URL of the SQL Dialect Conversion Service with the following command in Doris:
`MySQL> set global sql_converter_service_url = "http://127.0.0.1:5001/api/v1/convert"`
-
- > 1. `127.0.0.1:5001` is the deployment node IP and port of the SQL dialect conversion service.
+
+ :::note
+ 1. `127.0.0.1:5001` is the deployment node IP and port of the SQL dialect conversion service.
+ :::
## Use SQL dialect
+Currently supported dialect types include:
+
+- `presto`
+- `trino`
+- `hive`
+- `spark`
+- `postgres`
+- `clickhouse`
+- `oracle`
+
example:
- Presto
@@ -113,13 +127,3 @@ mysql> select toString(start_time) as col1,
1 row in set (0.04 sec)
```
-Currently supported dialect types include:
-
-- `presto`
-- `trino`
-- `hive`
-- `spark`
-- `postgres`
-- `clickhouse`
-- `oracle`
-
diff --git a/docs/zh-CN/docs/lakehouse/sql-dialect.md b/docs/zh-CN/docs/lakehouse/sql-dialect.md
index 3b731b949da..3475c8dfe4f 100644
--- a/docs/zh-CN/docs/lakehouse/sql-dialect.md
+++ b/docs/zh-CN/docs/lakehouse/sql-dialect.md
@@ -28,32 +28,45 @@ under the License.
从 2.1 版本开始，Doris 可以支持多种 SQL 方言，如 Presto、Trino、Hive、PostgreSQL、Spark、Clickhouse 等等。通过这个功能，用户可以直接使用对应的 SQL 方言查询 Doris 中的数据，方便用户将原先的业务平滑的迁移到 Doris 中。
-> 1. 该功能目前是实验性功能，您在使用过程中如遇到任何问题，欢迎通过邮件组、[Github issue](https://github.com/apache/doris/issues) 等方式进行反馈。
+> 1. 该功能目前是实验性功能，您在使用过程中如遇到任何问题，欢迎通过邮件组、[GitHub issue](https://github.com/apache/doris/issues) 等方式进行反馈。
>
-> 2. 该功能只支持查询语句，不支持包括 Explain 在内的其他 DDL、DML 语句。
+> 2. 该功能只支持查询语句，不支持 DDL、DML 语句。
## 部署服务
1. 下载最新版本的 [SQL 方言转换工具](https://doris-build-1308700295.cos.ap-beijing.myqcloud.com/transform-doris-tool/transform-doris-tool-1.0.0-bin-x86)。
-2. 在任意FE节点，通过以下命令启动服务：
+2. 在任意 FE 节点，通过以下命令启动服务：
- `nohup ./transform-doris-tool-1.0.0-bin-x86 run --host=0.0.0.0 --port=5001 &`
+ `nohup ./doris-sql-convertor-1.0.1-bin-x86 run --host=0.0.0.0 --port=5001 &`
- > 1. 该服务是一个无状态的服务，可随时启停。
- >
- > 2. `5001` 是服务端口，可以任意指定为一个可用端口。
- >
- > 3. 建议在每个 FE 节点都单独启动一个服务。
+ :::note
+ 1. 该服务是一个无状态的服务，可随时启停。
+
+ 2. `5001` 是服务端口，可以任意指定为一个可用端口。
+
+ 3. 建议在每个 FE 节点都单独启动一个服务。
+ :::
3. 启动 Doris 集群（2.1 或更高版本）
-4. 通过以下命令，在Doris中设置 SQL 方言转换服务的 URL：
+4. 通过以下命令，在 Doris 中设置 SQL 方言转换服务的 URL：
`MySQL> set global sql_converter_service_url = "http://127.0.0.1:5001/api/v1/convert"`
-
- > 1. `127.0.0.1:5001` 是 SQL 方言转换服务的部署节点 ip 和端口。
+
+ :::note
+ 1. `127.0.0.1:5001` 是 SQL 方言转换服务的部署节点 ip 和端口。
+ :::
## 使用 SQL 方言
+目前支持的方言类型包括：
+
+- `presto`
+- `trino`
+- `hive`
+- `spark`
+- `postgres`
+- `clickhouse`
+
示例:
- Presto
@@ -113,12 +126,3 @@ mysql> select toString(start_time) as col1,
1 row in set (0.04 sec)
```
-目前支持的方言类型包括：
-
-- `presto`
-- `trino`
-- `hive`
-- `spark`
-- `postgres`
-- `clickhouse`
-
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 31/38: [doc](apache-superset)add doc for apache superset compatible (#31314)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 286685ff2fe85ba09e98691a7b198aa72b39a033
Author: Jiwen liu <61...@users.noreply.github.com>
AuthorDate: Thu Mar 7 12:11:56 2024 +0800
[doc](apache-superset)add doc for apache superset compatible (#31314)
---
.../ecosystem/bi-integration/apache-superset.md | 53 ++++++++++++++++++++
docs/sidebars.json | 1 +
.../ecosystem/bi-integration/apache-superset.md | 57 ++++++++++++++++++++++
3 files changed, 111 insertions(+)
diff --git a/docs/en/docs/ecosystem/bi-integration/apache-superset.md b/docs/en/docs/ecosystem/bi-integration/apache-superset.md
new file mode 100644
index 00000000000..0ada613da75
--- /dev/null
+++ b/docs/en/docs/ecosystem/bi-integration/apache-superset.md
@@ -0,0 +1,53 @@
+---
+{
+"title": "Apache Superset Integrate",
+"language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+## Introduction
+Apache Superset is an open-source data exploration platform. It supports a rich variety of data source connections and numerous visualization methods. It also enables fine-grained access control for users. The main features of this tool include self-service analysis, customizable dashboards, visualization of analytical results (with export functionality), and user/role permission control. Moreover, it integrates an SQL editor for conducting SQL editing and queries.
+
+In Apache Superset version 3.1 official support has been introduced for querying and visualizing both internal and external data from Apache Doris.
+## Preconditions
+Ensure you have completed the following tool installations:
+1. Install the Python client for Apache Doris on the Apache Superset server.
+ pip install pydoris
+2. Install Apache Superset version 3.1 or above. For detailed instructions, refer to [Installing Superset from PyPI](https://superset.apache.org/docs/installation/installing-superset-from-pypi/) or [Installing Superset Locally Using Docker Compose](https://hub.docker.com/r/apache/superset).
+## Add data source
+When creating a data source in Apache Superset, please pay attention to the following two points:
+- Choose Apache Doris as the data source in SUPPORTED DATABASES.
+- In the SQLALCHEMY URI, fill in the URI following the Doris SQLAlchemy URI format as shown below.
+
+ ```doris://<User>:<Password>@<Host>:<Port>/<Catalog>.<Database>```
+- URI parameters are explained as follows:
+ - User: The username for logging into the Doris cluster, e.g., admin.
+ - Password: The password for logging into the Doris cluster.
+ - Host: The IP address of the FE (Frontend) host in the Doris cluster.
+ - Port: The query port of the FE in the Doris cluster, e.g. 9030.
+ - Catalog: The target Catalog in the Doris cluster. Both Internal Catalog and External Catalog are supported.
+ - Database: The target database in the Doris cluster. Both internal and external databases are supported.
+
+
+:::tip
+1. When deploying Superset using the latest Docker image, if you encounter the issue of not finding the Apache Doris data source, it may be because the default [Superset Docker image](https://hub.docker.com/r/apache/superset) includes only basic data source builds. You need to manually install the pydoris package. You can refer to the 'How to extend this image' section in the Superset Docker tutorial for the deployment steps of Superset.
+2. It is recommended to use Apache Doris 2.0.4 and above.
+:::
diff --git a/docs/sidebars.json b/docs/sidebars.json
index 2e009562a82..eee48955dfa 100644
--- a/docs/sidebars.json
+++ b/docs/sidebars.json
@@ -287,6 +287,7 @@
"ecosystem/cloudcanal",
"ecosystem/hive-bitmap-udf",
"ecosystem/dbt-doris-adapter",
+ "ecosystem/bi-integration/apache-superset",
{
"type": "category",
"label": "UDF",
diff --git a/docs/zh-CN/docs/ecosystem/bi-integration/apache-superset.md b/docs/zh-CN/docs/ecosystem/bi-integration/apache-superset.md
new file mode 100644
index 00000000000..cd974bfad14
--- /dev/null
+++ b/docs/zh-CN/docs/ecosystem/bi-integration/apache-superset.md
@@ -0,0 +1,57 @@
+{
+"title": "Apache Superset 集成",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+## 介绍
+Apache Superset 是一个开源的数据挖掘平台，支持丰富的数据源连接，多种可视化方式，并能够对用户实现细粒度的权限控制。该工具主要特点是可自助分析、自定义仪表盘、分析结果可视化（导出）、用户/角色权限控制，还集成了一个 SQL 编辑器，可以进行 SQL 编辑查询等。
+
+在 Apache Superset 3.1 版本中正式支持了 Apache Doris 的内部数据和外部数据进行查询和可视化处理。
+## 前置条件
+确保您已完成如下工具安装：
+1. 在 Apache Superset 服务器上安装 Apache Doris 的 Python 客户端。
+ ```
+ pip install pydoris
+ ```
+2. 安装 Apache Superset 3.1 及其以上的版本。具体参见[安装 Superset 从 PyPI 库](https://superset.apache.org/docs/installation/installing-superset-from-pypi)或者[通过 Docker 容器的方式安装](https://hub.docker.com/r/apache/superset)
+
+## 添加数据源
+当你在 Apache Superset 中创建数据源时需要注意以下两点：
+- 在 SUPPORTED DATABASES 里选择 Apache Doris 作为数据源。
+- 在 SQLALCHEMY URI 里，按如下 Doris SQLAlchemy URI 格式填写 URI：
+
+ ```doris://<User>:<Password>@<Host>:<Port>/<Catalog>.<Database>```
+- URI 参数说明如下：
+ - User：用于登录 Doris 集群的用户名，如 admin。
+ - Password：用于登录 Doris 集群的用户密码。
+ - Host：Doris 集群的 FE 主机 IP 地址。
+ - Port：Doris 集群的 FE 查询端口，如 9030。
+ - Catalog：Doris 集群中的目标 Catalog。Internal Catalog 和 External Catalog 均支持。
+ - Database：Doris 集群中的目标数据库。内部数据库和外部数据库均支持。
+
+
+:::tip
+1. 当你使用最新的 Docker 镜像部署 Superset 时，如果发现找不到 Apache Doris 数据源，这个可能是因为 Superset Docker Image默认只包含基本的数据源构建，需要手动将 pydoris 包安装进来，您可以参考 [superset docker教程] (https://hub.docker.com/r/apache/superset)中的 How to extend this image 步骤进行 Apache Superset 的部署。
+2. 推荐使用 Apache Doris 2.0.4 及以上版本。
+:::
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 17/38: [bugfix](coordinator) should use fragment id not profile fragment id to cancel fragment (#31852)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ad2f7fc3165e5afc0498cd3c69f4d509354386db
Author: yiguolei <67...@qq.com>
AuthorDate: Wed Mar 6 21:38:14 2024 +0800
[bugfix](coordinator) should use fragment id not profile fragment id to cancel fragment (#31852)
---
.../main/java/org/apache/doris/qe/Coordinator.java | 70 ++--------------------
.../org/apache/doris/rpc/BackendServiceProxy.java | 6 +-
2 files changed, 10 insertions(+), 66 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index b91fcd15e3c..64f32ed2e24 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -136,7 +136,6 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
-import java.util.Objects;
import java.util.Optional;
import java.util.Random;
import java.util.Set;
@@ -1253,7 +1252,7 @@ public class Coordinator implements CoordInterface {
}
}
- private void updateStatus(Status status, TUniqueId instanceId) {
+ private void updateStatus(Status status) {
lock.lock();
try {
// The query is done and we are just waiting for remote fragments to clean up.
@@ -1272,10 +1271,6 @@ public class Coordinator implements CoordInterface {
}
queryStatus.setStatus(status);
- LOG.warn("one instance report fail throw updateStatus(), need cancel. job id: {},"
- + " query id: {}, instance id: {}, error message: {}",
- jobId, DebugUtil.printId(queryId), instanceId != null ? DebugUtil.printId(instanceId) : "NaN",
- status.getErrorMsg());
if (status.getErrorCode() == TStatusCode.TIMEOUT) {
cancelInternal(Types.PPlanFragmentCancelReason.TIMEOUT);
} else {
@@ -1286,38 +1281,6 @@ public class Coordinator implements CoordInterface {
}
}
- private void updateStatus(Status status, long backendId) {
- lock.lock();
- try {
- // The query is done and we are just waiting for remote fragments to clean up.
- // Ignore their cancelled updates.
- if (returnedAllResults && status.isCancelled()) {
- return;
- }
- // nothing to update
- if (status.ok()) {
- return;
- }
-
- // don't override an error status; also, cancellation has already started
- if (!queryStatus.ok()) {
- return;
- }
-
- queryStatus.setStatus(status);
- LOG.warn("one instance report fail throw updateStatus(), need cancel. job id: {},"
- + " query id: {}, error message: {}",
- jobId, DebugUtil.printId(queryId), status.getErrorMsg());
- if (status.getErrorCode() == TStatusCode.TIMEOUT) {
- cancelInternal(Types.PPlanFragmentCancelReason.TIMEOUT, backendId);
- } else {
- cancelInternal(Types.PPlanFragmentCancelReason.INTERNAL_ERROR, backendId);
- }
- } finally {
- lock.unlock();
- }
- }
-
@Override
public RowBatch getNext() throws Exception {
if (receiver == null) {
@@ -1332,7 +1295,7 @@ public class Coordinator implements CoordInterface {
DebugUtil.printId(queryId), status.toString());
}
- updateStatus(status, null /* no instance id */);
+ updateStatus(status);
Status copyStatus = null;
lock();
@@ -1487,18 +1450,6 @@ public class Coordinator implements CoordInterface {
executionProfile.onCancel();
}
- private void cancelInternal(Types.PPlanFragmentCancelReason cancelReason, long backendId) {
- if (null != receiver) {
- receiver.cancel(cancelReason.toString());
- }
- if (null != pointExec) {
- pointExec.cancel();
- return;
- }
- cancelRemoteFragmentsAsync(cancelReason, backendId);
- executionProfile.onCancel();
- }
-
private void cancelRemoteFragmentsAsync(Types.PPlanFragmentCancelReason cancelReason) {
if (enablePipelineEngine) {
for (PipelineExecContext ctx : pipelineExecContexts.values()) {
@@ -1511,15 +1462,6 @@ public class Coordinator implements CoordInterface {
}
}
- private void cancelRemoteFragmentsAsync(Types.PPlanFragmentCancelReason cancelReason, long backendId) {
- Preconditions.checkArgument(enablePipelineXEngine);
- for (PipelineExecContext ctx : pipelineExecContexts.values()) {
- if (!Objects.equals(idToBackend.get(backendId), ctx.backend)) {
- ctx.cancelFragmentInstance(cancelReason);
- }
- }
- }
-
private void computeFragmentExecParams() throws Exception {
// fill hosts field in fragmentExecParams
computeFragmentHosts();
@@ -2489,7 +2431,7 @@ public class Coordinator implements CoordInterface {
LOG.warn("one instance report fail, query_id={} instance_id={}, error message: {}",
DebugUtil.printId(queryId), DebugUtil.printId(params.getFragmentInstanceId()),
status.getErrorMsg());
- updateStatus(status, params.backend_id);
+ updateStatus(status);
}
if (params.isSetDeltaUrls()) {
updateDeltas(params.getDeltaUrls());
@@ -2546,7 +2488,7 @@ public class Coordinator implements CoordInterface {
DebugUtil.printId(queryId), params.getFragmentId(),
DebugUtil.printId(params.getFragmentInstanceId()),
params.getBackendId(), status.getErrorMsg());
- updateStatus(status, params.getFragmentInstanceId());
+ updateStatus(status);
}
// params.isDone() should be promised.
@@ -2622,7 +2564,7 @@ public class Coordinator implements CoordInterface {
LOG.warn("Instance {} of query {} report failed status, error msg: {}",
DebugUtil.printId(queryId), DebugUtil.printId(params.getFragmentInstanceId()),
status.getErrorMsg());
- updateStatus(status, params.getFragmentInstanceId());
+ updateStatus(status);
}
}
@@ -3336,7 +3278,7 @@ public class Coordinator implements CoordInterface {
try {
try {
BackendServiceProxy.getInstance().cancelPipelineXPlanFragmentAsync(brpcAddress,
- this.profileFragmentId, queryId, cancelReason);
+ this.fragmentId, queryId, cancelReason);
} catch (RpcException e) {
LOG.warn("cancel plan fragment get a exception, address={}:{}", brpcAddress.getHostname(),
brpcAddress.getPort());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/rpc/BackendServiceProxy.java b/fe/fe-core/src/main/java/org/apache/doris/rpc/BackendServiceProxy.java
index 52350d805bb..df9a90433db 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/rpc/BackendServiceProxy.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/rpc/BackendServiceProxy.java
@@ -21,6 +21,7 @@ import org.apache.doris.common.Config;
import org.apache.doris.common.ThreadPoolManager;
import org.apache.doris.common.util.NetUtils;
import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.planner.PlanFragmentId;
import org.apache.doris.proto.InternalService;
import org.apache.doris.proto.InternalService.PExecPlanFragmentStartRequest;
import org.apache.doris.proto.InternalService.PGetWalQueueSizeRequest;
@@ -241,12 +242,13 @@ public class BackendServiceProxy {
}
public Future<InternalService.PCancelPlanFragmentResult> cancelPipelineXPlanFragmentAsync(TNetworkAddress address,
- int fragmentId, TUniqueId queryId, Types.PPlanFragmentCancelReason cancelReason) throws RpcException {
+ PlanFragmentId fragmentId, TUniqueId queryId,
+ Types.PPlanFragmentCancelReason cancelReason) throws RpcException {
final InternalService.PCancelPlanFragmentRequest pRequest = InternalService.PCancelPlanFragmentRequest
.newBuilder()
.setFinstId(Types.PUniqueId.newBuilder().setHi(0).setLo(0).build())
.setCancelReason(cancelReason)
- .setFragmentId(fragmentId)
+ .setFragmentId(fragmentId.asInt())
.setQueryId(Types.PUniqueId.newBuilder().setHi(queryId.hi).setLo(queryId.lo).build()).build();
try {
final BackendServiceClient client = getProxy(address);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/38: [docs](jdbc catalog) add sqlserver timestamp type mapping doc (#31832)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f5991d8b029f4a37003439a45ba3d5bf0b0388a8
Author: zy-kkk <zh...@gmail.com>
AuthorDate: Wed Mar 6 11:38:27 2024 +0800
[docs](jdbc catalog) add sqlserver timestamp type mapping doc (#31832)
---
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 40 ++++++++++++-------------
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 5 ++--
2 files changed, 23 insertions(+), 22 deletions(-)
diff --git a/docs/en/docs/lakehouse/multi-catalog/jdbc.md b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
index fa1fa4800cd..64f598aec19 100644
--- a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
@@ -194,11 +194,11 @@ JDBC catalog has the following three stages for mapping rules for external table
* Doris 2.1.0 and later versions:
- In order to avoid confusion with the `lower_case_table_names` parameter of FE conf, this configuration name is changed to `lower_case_meta_names`, which is valid for all databases. During query, all library names, table names and column names will be converted into real names, and then Check it out. If you upgrade from an old version to 2.0.4, you need `Refresh <catalog_name>` to take effect.
+ In order to avoid confusion with the `lower_case_table_names` parameter of FE conf, this configuration name is changed to `lower_case_meta_names`, which is valid for all databases. During query, all library names, table names and column names will be converted into real names, and then Check it out. If you upgrade from an old version to 2.1.0, you need `Refresh <catalog_name>` to take effect.
For specific rules, please refer to the introduction of `lower_case_meta_names` at the beginning of this section.
- Users who have previously set the JDBC catalog `lower_case_table_names` parameter will automatically have `lower_case_table_names` converted to `lower_case_meta_names` when upgrading to 2.0.4.
+ Users who have previously set the JDBC catalog `lower_case_table_names` parameter will automatically have `lower_case_table_names` converted to `lower_case_meta_names` when upgrading to 2.1.0.
### Specify synchronization database:
@@ -594,24 +594,24 @@ As for data mapping from SQLServer to Doris, one Database in Doris corresponds t
#### Type Mapping
-| SQLServer Type | Doris Type | Comment |
-|----------------------------------------|---------------|--------------------------------------------------------------------------|
-| bit | BOOLEAN | |
-| tinyint | SMALLINT | SQLServer's tinyint is an unsigned number, so it maps to Doris' SMALLINT |
-| smallint | SMALLINT | |
-| int | INT | |
-| bigint | BIGINT | |
-| real | FLOAT | |
-| float | DOUBLE | |
-| money | DECIMAL(19,4) | |
-| smallmoney | DECIMAL(10,4) | |
-| decimal/numeric | DECIMAL | |
-| date | DATE | |
-| datetime/datetime2/smalldatetime | DATETIMEV2 | |
-| char/varchar/text/nchar/nvarchar/ntext | STRING | |
-| binary/varbinary | STRING | |
-| time/datetimeoffset | STRING | |
-| Other | UNSUPPORTED | |
+| SQLServer Type | Doris Type | Comment |
+|----------------------------------------|---------------|----------------------------------------------------------------------------------|
+| bit | BOOLEAN | |
+| tinyint | SMALLINT | SQLServer's tinyint is an unsigned number, so it maps to Doris' SMALLINT |
+| smallint | SMALLINT | |
+| int | INT | |
+| bigint | BIGINT | |
+| real | FLOAT | |
+| float | DOUBLE | |
+| money | DECIMAL(19,4) | |
+| smallmoney | DECIMAL(10,4) | |
+| decimal/numeric | DECIMAL | |
+| date | DATE | |
+| datetime/datetime2/smalldatetime | DATETIMEV2 | |
+| char/varchar/text/nchar/nvarchar/ntext | STRING | |
+| time/datetimeoffset | STRING | |
+| timestamp | STRING | Read the hexadecimal display of binary data, which has no practical significance.|
+| Other | UNSUPPORTED | |
### Doris
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
index d78cb3e9cbf..ec54699227c 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
@@ -196,11 +196,11 @@ JDBC Catalog 对于外部表大小写的映射规则存在如下三个阶段：
* Doris 2.1.0 以及之后版本：
- 为了避免和 FE conf 的 `lower_case_table_names` 参数混淆，此配置名改为 `lower_case_meta_names`，对所有的数据库都有效，在查询时，会将所有的库名和表名以及列名转换为真实的名称，再去查询，如果是从老版本升级到 2.0.4 ，需要 `Refresh <catalog_name>` 才能生效。
+ 为了避免和 FE conf 的 `lower_case_table_names` 参数混淆，此配置名改为 `lower_case_meta_names`，对所有的数据库都有效，在查询时，会将所有的库名和表名以及列名转换为真实的名称，再去查询，如果是从老版本升级到 2.1.0 ，需要 `Refresh <catalog_name>` 才能生效。
具体规则参考本小节开始对于 `lower_case_meta_names` 的介绍。
- 此前设置过 JDBC Catalog `lower_case_table_names` 参数的用户会在升级到 2.0.4 时，自动将 `lower_case_table_names` 转换为 `lower_case_meta_names`。
+ 此前设置过 JDBC Catalog `lower_case_table_names` 参数的用户会在升级到 2.1.0 时，自动将 `lower_case_table_names` 转换为 `lower_case_meta_names`。
### 指定同步数据库
@@ -614,6 +614,7 @@ CREATE CATALOG jdbc_sqlserve PROPERTIES (
| char/varchar/text/nchar/nvarchar/ntext | STRING | |
| binary/varbinary | STRING | |
| time/datetimeoffset | STRING | |
+| timestamp | STRING | 读取二进制数据的十六进制显示，无实际意义 |
| Other | UNSUPPORTED | |
### Doris
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 19/38: [enhancement](Nereids): support more condition Date/DateTime Literal (#31858)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 80c80b67196010fc1caa970d2c4cc5382f12c3de
Author: jakevin <ja...@gmail.com>
AuthorDate: Wed Mar 6 21:54:49 2024 +0800
[enhancement](Nereids): support more condition Date/DateTime Literal (#31858)
---
.../trees/expressions/literal/DateLiteral.java | 46 ++++++++++------------
.../expressions/literal/DateTimeLiteralTest.java | 7 ++++
2 files changed, 28 insertions(+), 25 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
index c0c1b18488f..7dcd86a1504 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
@@ -163,11 +163,10 @@ public class DateLiteral extends Literal {
StringBuilder sb = new StringBuilder();
int i = 0;
+ // date and time contains 6 number part at most, so we just need normal 6 number part
+ int partNumber = 0;
// handle two digit year
- if (!isPunctuation(s.charAt(2)) && !isPunctuation(s.charAt(4))) {
- throw new AnalysisException("date/datetime literal [" + s + "] is invalid");
- }
if (isPunctuation(s.charAt(2))) {
String yy = s.substring(0, 2);
int year = Integer.parseInt(yy);
@@ -178,11 +177,10 @@ public class DateLiteral extends Literal {
}
sb.append(yy);
i = 2;
+ partNumber += 1;
}
// normalize leading 0 for date and time
- // date and time contains 6 number part at most, so we just need normal 6 number part
- int partNumber = 0;
while (i < s.length() && partNumber < 6) {
char c = s.charAt(i);
if (Character.isDigit(c)) {
@@ -193,39 +191,37 @@ public class DateLiteral extends Literal {
}
int len = j - i;
if (len == 4 || len == 2) {
- for (int k = i; k < j; k++) {
- sb.append(s.charAt(k));
- }
+ sb.append(s, i, j);
} else if (len == 1) {
- sb.append('0').append(c);
+ if (partNumber == 0) {
+ sb.append("000").append(c);
+ } else {
+ sb.append('0').append(c);
+ }
} else {
throw new AnalysisException("date/datetime literal [" + s + "] is invalid");
}
i = j;
partNumber += 1;
} else if (isPunctuation(c) || c == ' ' || c == 'T') {
- sb.append(c);
i += 1;
+ if (partNumber < 3 && isPunctuation(c)) {
+ sb.append('-');
+ } else if (partNumber == 3) {
+ while (i < s.length() && (isPunctuation(s.charAt(i)) || s.charAt(i) == ' ' || s.charAt(i) == 'T')) {
+ i += 1;
+ }
+ sb.append(' ');
+ } else if (partNumber > 3 && isPunctuation(c)) {
+ sb.append(':');
+ } else {
+ throw new AnalysisException("date/datetime literal [" + s + "] is invalid");
+ }
} else {
break;
}
}
- // replace punctuation with '-'
- replacePunctuation(s, sb, '-', 4);
- replacePunctuation(s, sb, '-', 7);
- // Replace punctuation with ' '
- if (sb.length() > 10 && sb.charAt(10) != ' ') {
- if (sb.charAt(10) == 'T') {
- sb.setCharAt(10, ' ');
- } else {
- replacePunctuation(s, sb, ' ', 10);
- }
- }
- // replace punctuation with ':'
- replacePunctuation(s, sb, ':', 13);
- replacePunctuation(s, sb, ':', 16);
-
// add missing Minute Second in Time part
if (sb.length() == 13) {
sb.append(":00:00");
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
index ecae8f85d5e..8555d44d634 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
@@ -33,6 +33,13 @@ class DateTimeLiteralTest {
// });
}
+ @Test
+ void mysqlStrangeCase() {
+ new DateTimeV2Literal("0-08-01 13:21:03");
+ new DateTimeV2Literal("0001-01-01: 00:01:01.001");
+ new DateTimeV2Literal("2021?01?01 00.00.00");
+ }
+
@Test
void testBasic() {
Consumer<DateTimeV2Literal> assertFunc = (datetime) -> {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 38/38: [fix] (doc)Fix invalid link on benchmark tool page #31920
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 677c9348a7dcd19c533ce5e8c9211eb8eb2491a2
Author: kkop <45...@users.noreply.github.com>
AuthorDate: Thu Mar 7 16:03:12 2024 +0800
[fix] (doc)Fix invalid link on benchmark tool page #31920
---
docs/en/community/developer-guide/benchmark-tool.md | 2 +-
docs/zh-CN/community/developer-guide/benchmark-tool.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/en/community/developer-guide/benchmark-tool.md b/docs/en/community/developer-guide/benchmark-tool.md
index eb8277384a2..371ad5614d7 100644
--- a/docs/en/community/developer-guide/benchmark-tool.md
+++ b/docs/en/community/developer-guide/benchmark-tool.md
@@ -33,7 +33,7 @@ It can be used to test the performance of some parts of the BE storage layer (fo
## Compilation
-1. To ensure that the environment has been able to successfully compile the Doris ontology, you can refer to [Installation and deployment](/docs/install/source-install/compilation).
+1. To ensure that the environment has been able to successfully compile the Doris ontology, you can refer to [Installation and deployment](/docs/install/source-install/compilation-general).
2. Execute`run-be-ut.sh`
diff --git a/docs/zh-CN/community/developer-guide/benchmark-tool.md b/docs/zh-CN/community/developer-guide/benchmark-tool.md
index 8bb45332c7a..f4df2eebf70 100644
--- a/docs/zh-CN/community/developer-guide/benchmark-tool.md
+++ b/docs/zh-CN/community/developer-guide/benchmark-tool.md
@@ -33,7 +33,7 @@ under the License.
## 编译
-1. 确保环境已经能顺利编译Doris本体,可以参考[编译与部署](/docs/install/source-install/compilation)。
+1. 确保环境已经能顺利编译Doris本体,可以参考[编译与部署](/docs/install/source-install/compilation-general)。
2. 运行目录下的`run-be-ut.sh`
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 34/38: [enhance](mtmv) MTMV Use partial partition of base table (#31632)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 586f155d84ede48f165c592a11d44dde391edbf0
Author: zhangdong <49...@qq.com>
AuthorDate: Thu Mar 7 12:28:24 2024 +0800
[enhance](mtmv) MTMV Use partial partition of base table (#31632)
MTMV add 3 properties:
partition_sync_limit: digit
partition_sync_time_unit: DAY/MONTH/YEAR
partition_sync_date_format: like "%Y-%m-%d"/"%Y%m%d"
For example, the current time is 2020-02-03 20:10:10
- If partition_sync_limit is set to 1 and partition_sync_time_unit is set to DAY, only partitions with a time greater than or equal to 2020-02-03 00:00:00 will be synchronized to the MTMV
- If partition_sync_limit is set to 1 and partition_sync_time_unit is set to MONTH, only partitions with a time greater than or equal to 2020-02-01 00:00:00 will be synchronized to the MTMV
- If partition_sync_limit is set to 1 and partition_sync_time_unit is set to YEAR, only partitions with a time greater than or equal to 2020-01-01 00:00:00 will be synchronized to the MTMV
- If partition_sync_limit is set to 3 and partition_sync_time_unit is set to MONTH, only partitions with a time greater than or equal to 2019-12-01 00:00:00 will be synchronized to the MTMV
- If partition_sync_limit is set to 4 and partition_sync_time_unit is set to DAY, only partitions with a time greater than or equal to 2020-01-31 00:00:00 will be synchronized to the MTMV
---
.../apache/doris/catalog/ListPartitionItem.java | 21 ++
.../main/java/org/apache/doris/catalog/MTMV.java | 11 +-
.../org/apache/doris/catalog/PartitionItem.java | 14 ++
.../apache/doris/catalog/RangePartitionItem.java | 18 ++
.../apache/doris/common/util/PropertyAnalyzer.java | 3 +
.../apache/doris/mtmv/MTMVPartitionSyncConfig.java | 57 +++++
.../doris/mtmv/MTMVPartitionSyncTimeUnit.java | 36 ++++
.../org/apache/doris/mtmv/MTMVPartitionUtil.java | 11 +-
.../org/apache/doris/mtmv/MTMVPropertyUtil.java | 138 ++++++++++++
.../org/apache/doris/mtmv/MTMVRelatedTableIf.java | 31 ++-
.../main/java/org/apache/doris/mtmv/MTMVUtil.java | 127 ++++++++++-
.../plans/commands/info/AlterMTMVPropertyInfo.java | 39 +---
.../trees/plans/commands/info/CreateMTMVInfo.java | 49 +----
.../java/org/apache/doris/mtmv/MTMVUtilTest.java | 113 ++++++++++
.../mtmv_p0/test_hive_limit_partition_mtmv.out | 11 +
.../data/mtmv_p0/test_limit_partition_mtmv.out | 17 ++
.../mtmv_p0/test_hive_limit_partition_mtmv.groovy | 162 ++++++++++++++
.../mtmv_p0/test_limit_partition_mtmv.groovy | 240 +++++++++++++++++++++
18 files changed, 1010 insertions(+), 88 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionItem.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionItem.java
index d0f06dd7e55..1a4d188a0ca 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionItem.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionItem.java
@@ -20,6 +20,7 @@ package org.apache.doris.catalog;
import org.apache.doris.analysis.PartitionKeyDesc;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.mtmv.MTMVUtil;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
@@ -29,6 +30,7 @@ import java.io.DataOutput;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
+import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
@@ -99,6 +101,25 @@ public class ListPartitionItem extends PartitionItem {
return PartitionKeyDesc.createIn(Lists.newArrayList(res));
}
+ @Override
+ public boolean isGreaterThanSpecifiedTime(int pos, Optional<String> dateFormatOptional, long nowTruncSubSec)
+ throws AnalysisException {
+ for (PartitionKey partitionKey : partitionKeys) {
+ if (partitionKey.getKeys().size() <= pos) {
+ throw new AnalysisException(
+ String.format("toPartitionKeyDesc IndexOutOfBounds, partitionKey: %s, pos: %d",
+ partitionKey.toString(),
+ pos));
+ }
+ if (!isDefaultPartition() && MTMVUtil.getExprTimeSec(partitionKey.getKeys().get(pos), dateFormatOptional)
+ >= nowTruncSubSec) {
+ // As long as one of the partitionKeys meets the requirements, this partition needs to be retained
+ return true;
+ }
+ }
+ return false;
+ }
+
@Override
public void write(DataOutput out) throws IOException {
out.writeInt(partitionKeys.size());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index 05622bfc0e7..80feb8f12ae 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -38,6 +38,7 @@ import org.apache.doris.mtmv.MTMVRefreshPartitionSnapshot;
import org.apache.doris.mtmv.MTMVRefreshSnapshot;
import org.apache.doris.mtmv.MTMVRelation;
import org.apache.doris.mtmv.MTMVStatus;
+import org.apache.doris.mtmv.MTMVUtil;
import org.apache.doris.persist.gson.GsonUtils;
import com.google.common.collect.Maps;
@@ -217,7 +218,7 @@ public class MTMV extends OlapTable {
public long getGracePeriod() {
readMvLock();
try {
- if (mvProperties.containsKey(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD)) {
+ if (!StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD))) {
return Long.parseLong(mvProperties.get(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD)) * 1000;
} else {
return 0L;
@@ -243,7 +244,7 @@ public class MTMV extends OlapTable {
public int getRefreshPartitionNum() {
readMvLock();
try {
- if (mvProperties.containsKey(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM)) {
+ if (!StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM))) {
int value = Integer.parseInt(mvProperties.get(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM));
return value < 1 ? MTMVTask.DEFAULT_REFRESH_PARTITION_NUM : value;
} else {
@@ -257,7 +258,7 @@ public class MTMV extends OlapTable {
public Set<String> getExcludedTriggerTables() {
readMvLock();
try {
- if (!mvProperties.containsKey(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES)) {
+ if (StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES))) {
return Sets.newHashSet();
}
String[] split = mvProperties.get(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES).split(",");
@@ -326,8 +327,10 @@ public class MTMV extends OlapTable {
return Maps.newHashMap();
}
Map<PartitionKeyDesc, Set<Long>> res = new HashMap<>();
- Map<Long, PartitionItem> relatedPartitionItems = mvPartitionInfo.getRelatedTable().getAndCopyPartitionItems();
int relatedColPos = mvPartitionInfo.getRelatedColPos();
+ Map<Long, PartitionItem> relatedPartitionItems = mvPartitionInfo.getRelatedTable()
+ .getPartitionItemsByTimeFilter(relatedColPos,
+ MTMVUtil.generateMTMVPartitionSyncConfigByProperties(mvProperties));
for (Entry<Long, PartitionItem> entry : relatedPartitionItems.entrySet()) {
PartitionKeyDesc partitionKeyDesc = entry.getValue().toPartitionKeyDesc(relatedColPos);
if (res.containsKey(partitionKeyDesc)) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionItem.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionItem.java
index 3c29aa2f48b..af1bbc9d0e2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionItem.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionItem.java
@@ -23,6 +23,7 @@ import org.apache.doris.common.io.Writable;
import java.util.Comparator;
import java.util.Map;
+import java.util.Optional;
public abstract class PartitionItem implements Comparable<PartitionItem>, Writable {
public static final Comparator<Map.Entry<Long, PartitionItem>> ITEM_MAP_ENTRY_COMPARATOR =
@@ -46,4 +47,17 @@ public abstract class PartitionItem implements Comparable<PartitionItem>, Writab
* @throws AnalysisException
*/
public abstract PartitionKeyDesc toPartitionKeyDesc(int pos) throws AnalysisException;
+
+ /**
+ * Check if the partition meets the time requirements
+ *
+ * @param pos The position of the partition column to be checked in all partition columns
+ * @param dateFormatOptional Convert other types to date format
+ * @param nowTruncSubSec The time to compare
+ * @return
+ * @throws AnalysisException
+ */
+ public abstract boolean isGreaterThanSpecifiedTime(int pos, Optional<String> dateFormatOptional,
+ long nowTruncSubSec)
+ throws AnalysisException;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionItem.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionItem.java
index 7a9162c874c..75870de63f2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionItem.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionItem.java
@@ -18,12 +18,15 @@
package org.apache.doris.catalog;
import org.apache.doris.analysis.PartitionKeyDesc;
+import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.util.RangeUtils;
+import org.apache.doris.mtmv.MTMVUtil;
import com.google.common.collect.Range;
import java.io.DataOutput;
import java.io.IOException;
+import java.util.Optional;
public class RangePartitionItem extends PartitionItem {
private Range<PartitionKey> partitionKeyRange;
@@ -60,6 +63,21 @@ public class RangePartitionItem extends PartitionItem {
return toPartitionKeyDesc();
}
+ @Override
+ public boolean isGreaterThanSpecifiedTime(int pos, Optional<String> dateFormatOptional, long nowTruncSubSec)
+ throws AnalysisException {
+ PartitionKey partitionKey = partitionKeyRange.upperEndpoint();
+ if (partitionKey.getKeys().size() <= pos) {
+ throw new AnalysisException(
+ String.format("toPartitionKeyDesc IndexOutOfBounds, partitionKey: %s, pos: %d",
+ partitionKey.toString(),
+ pos));
+ }
+ // If the upper limit of the partition range meets the requirements, this partition needs to be retained
+ return !isDefaultPartition() && MTMVUtil.getExprTimeSec(partitionKey.getKeys().get(pos), dateFormatOptional)
+ >= nowTruncSubSec;
+ }
+
@Override
public void write(DataOutput out) throws IOException {
RangeUtils.writeRange(out, partitionKeyRange);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
index 42d62046948..a09694ada18 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
@@ -165,6 +165,9 @@ public class PropertyAnalyzer {
public static final String PROPERTIES_EXCLUDED_TRIGGER_TABLES = "excluded_trigger_tables";
public static final String PROPERTIES_REFRESH_PARTITION_NUM = "refresh_partition_num";
public static final String PROPERTIES_WORKLOAD_GROUP = "workload_group";
+ public static final String PROPERTIES_PARTITION_SYNC_LIMIT = "partition_sync_limit";
+ public static final String PROPERTIES_PARTITION_TIME_UNIT = "partition_sync_time_unit";
+ public static final String PROPERTIES_PARTITION_DATE_FORMAT = "partition_date_format";
// For unique key data model, the feature Merge-on-Write will leverage a primary
// key index and a delete-bitmap to mark duplicate keys as deleted in load stage,
// which can avoid the merging cost in read stage, and accelerate the aggregation
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionSyncConfig.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionSyncConfig.java
new file mode 100644
index 00000000000..b85978e4f46
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionSyncConfig.java
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mtmv;
+
+import java.util.Optional;
+
+public class MTMVPartitionSyncConfig {
+ private int syncLimit;
+ private MTMVPartitionSyncTimeUnit timeUnit;
+ private Optional<String> dateFormat;
+
+ public MTMVPartitionSyncConfig(int syncLimit, MTMVPartitionSyncTimeUnit timeUnit,
+ Optional<String> dateFormat) {
+ this.syncLimit = syncLimit;
+ this.timeUnit = timeUnit;
+ this.dateFormat = dateFormat;
+ }
+
+ public int getSyncLimit() {
+ return syncLimit;
+ }
+
+ public void setSyncLimit(int syncLimit) {
+ this.syncLimit = syncLimit;
+ }
+
+ public MTMVPartitionSyncTimeUnit getTimeUnit() {
+ return timeUnit;
+ }
+
+ public void setTimeUnit(MTMVPartitionSyncTimeUnit timeUnit) {
+ this.timeUnit = timeUnit;
+ }
+
+ public Optional<String> getDateFormat() {
+ return dateFormat;
+ }
+
+ public void setDateFormat(Optional<String> dateFormat) {
+ this.dateFormat = dateFormat;
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionSyncTimeUnit.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionSyncTimeUnit.java
new file mode 100644
index 00000000000..0d508b9bb22
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionSyncTimeUnit.java
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mtmv;
+
+import java.util.Optional;
+
+public enum MTMVPartitionSyncTimeUnit {
+ YEAR,
+ MONTH,
+ DAY;
+
+ public static Optional<MTMVPartitionSyncTimeUnit> fromString(String unit) {
+ for (MTMVPartitionSyncTimeUnit u : MTMVPartitionSyncTimeUnit.values()) {
+ if (u.name().equalsIgnoreCase(unit)) {
+ return Optional.of(u);
+ }
+ }
+ return Optional.empty();
+ }
+}
+
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java
index 9c85725b5bb..b80c5fc283f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java
@@ -122,10 +122,12 @@ public class MTMVPartitionUtil {
* @throws AnalysisException
*/
public static List<AllPartitionDesc> getPartitionDescsByRelatedTable(MTMVRelatedTableIf relatedTable,
- Map<String, String> tableProperties, String relatedCol) throws AnalysisException {
+ Map<String, String> tableProperties, String relatedCol, Map<String, String> mvProperties)
+ throws AnalysisException {
HashMap<String, String> partitionProperties = Maps.newHashMap();
List<AllPartitionDesc> res = Lists.newArrayList();
- Set<PartitionKeyDesc> relatedPartitionDescs = getRelatedPartitionDescs(relatedTable, relatedCol);
+ Set<PartitionKeyDesc> relatedPartitionDescs = getRelatedPartitionDescs(relatedTable, relatedCol,
+ MTMVUtil.generateMTMVPartitionSyncConfigByProperties(mvProperties));
for (PartitionKeyDesc partitionKeyDesc : relatedPartitionDescs) {
SinglePartitionDesc singlePartitionDesc = new SinglePartitionDesc(true,
generatePartitionName(partitionKeyDesc),
@@ -137,11 +139,12 @@ public class MTMVPartitionUtil {
return res;
}
- private static Set<PartitionKeyDesc> getRelatedPartitionDescs(MTMVRelatedTableIf relatedTable, String relatedCol)
+ private static Set<PartitionKeyDesc> getRelatedPartitionDescs(MTMVRelatedTableIf relatedTable, String relatedCol,
+ MTMVPartitionSyncConfig config)
throws AnalysisException {
int pos = getPos(relatedTable, relatedCol);
Set<PartitionKeyDesc> res = Sets.newHashSet();
- for (Entry<Long, PartitionItem> entry : relatedTable.getAndCopyPartitionItems().entrySet()) {
+ for (Entry<Long, PartitionItem> entry : relatedTable.getPartitionItemsByTimeFilter(pos, config).entrySet()) {
PartitionKeyDesc partitionKeyDesc = entry.getValue().toPartitionKeyDesc(pos);
res.add(partitionKeyDesc);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java
new file mode 100644
index 00000000000..a9df9b87d72
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mtmv;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.Optional;
+import java.util.Set;
+
+public class MTMVPropertyUtil {
+ public static final Set<String> mvPropertyKeys = Sets.newHashSet(
+ PropertyAnalyzer.PROPERTIES_GRACE_PERIOD,
+ PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES,
+ PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM,
+ PropertyAnalyzer.PROPERTIES_WORKLOAD_GROUP,
+ PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT,
+ PropertyAnalyzer.PROPERTIES_PARTITION_TIME_UNIT,
+ PropertyAnalyzer.PROPERTIES_PARTITION_DATE_FORMAT
+ );
+
+ public static void analyzeProperty(String key, String value) {
+ switch (key) {
+ case PropertyAnalyzer.PROPERTIES_GRACE_PERIOD:
+ analyzeGracePeriod(value);
+ break;
+ case PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM:
+ analyzeRefreshPartitionNum(value);
+ break;
+ case PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES:
+ analyzeExcludedTriggerTables(value);
+ break;
+ case PropertyAnalyzer.PROPERTIES_WORKLOAD_GROUP:
+ analyzeWorkloadGroup(value);
+ break;
+ case PropertyAnalyzer.PROPERTIES_PARTITION_TIME_UNIT:
+ analyzePartitionTimeUnit(value);
+ break;
+ case PropertyAnalyzer.PROPERTIES_PARTITION_DATE_FORMAT:
+ analyzePartitionDateFormat(value);
+ break;
+ case PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT:
+ analyzePartitionSyncLimit(value);
+ break;
+ default:
+ throw new AnalysisException("illegal key:" + key);
+
+ }
+ }
+
+ private static void analyzePartitionSyncLimit(String value) {
+ if (StringUtils.isEmpty(value)) {
+ return;
+ }
+ try {
+ Integer.parseInt(value);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("valid partition_sync_limit: " + value);
+ }
+ }
+
+ private static void analyzePartitionDateFormat(String value) {
+ // do nothing
+ }
+
+ private static void analyzePartitionTimeUnit(String value) {
+ if (StringUtils.isEmpty(value)) {
+ return;
+ }
+ Optional<MTMVPartitionSyncTimeUnit> mtmvPartitionSyncTimeUnit = MTMVPartitionSyncTimeUnit
+ .fromString(value);
+ if (!mtmvPartitionSyncTimeUnit.isPresent()) {
+ throw new AnalysisException("valid partition_sync_time_unit: " + value);
+ }
+ }
+
+ private static void analyzeWorkloadGroup(String value) {
+ if (StringUtils.isEmpty(value)) {
+ return;
+ }
+ if (!StringUtils.isEmpty(value) && !Env.getCurrentEnv().getAccessManager()
+ .checkWorkloadGroupPriv(ConnectContext.get(), value, PrivPredicate.USAGE)) {
+ String message = String
+ .format("Access denied; you need (at least one of) "
+ + "the %s privilege(s) to use workload group '%s'.",
+ "USAGE/ADMIN", value);
+ throw new AnalysisException(message);
+ }
+ }
+
+ private static void analyzeExcludedTriggerTables(String value) {
+ // do nothing
+ }
+
+ private static void analyzeGracePeriod(String value) {
+ if (StringUtils.isEmpty(value)) {
+ return;
+ }
+ try {
+ Long.parseLong(value);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("valid grace_period: " + value);
+ }
+ }
+
+ private static void analyzeRefreshPartitionNum(String value) {
+ if (StringUtils.isEmpty(value)) {
+ return;
+ }
+ try {
+ Integer.parseInt(value);
+ } catch (NumberFormatException e) {
+ throw new AnalysisException("valid refresh_partition_num: " + value);
+ }
+ }
+
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedTableIf.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedTableIf.java
index 5ec7e98a407..1adfe315a8b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedTableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedTableIf.java
@@ -24,8 +24,12 @@ import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
+import com.google.common.collect.Maps;
+
import java.util.List;
import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
import java.util.Set;
/**
@@ -40,6 +44,31 @@ public interface MTMVRelatedTableIf extends TableIf {
*/
Map<Long, PartitionItem> getAndCopyPartitionItems();
+ /**
+ * Obtain a list of partitions filtered by time
+ *
+ * @param pos The position of the partition column to be checked in all partition columns
+ * @param config
+ * @return
+ * @throws AnalysisException
+ */
+ default Map<Long, PartitionItem> getPartitionItemsByTimeFilter(int pos, MTMVPartitionSyncConfig config)
+ throws AnalysisException {
+ Map<Long, PartitionItem> partitionItems = getAndCopyPartitionItems();
+ if (config.getSyncLimit() <= 0) {
+ return partitionItems;
+ }
+ long nowTruncSubSec = MTMVUtil.getNowTruncSubSec(config.getTimeUnit(), config.getSyncLimit());
+ Optional<String> dateFormat = config.getDateFormat();
+ Map<Long, PartitionItem> res = Maps.newHashMap();
+ for (Entry<Long, PartitionItem> entry : partitionItems.entrySet()) {
+ if (entry.getValue().isGreaterThanSpecifiedTime(pos, dateFormat, nowTruncSubSec)) {
+ res.put(entry.getKey(), entry.getValue());
+ }
+ }
+ return res;
+ }
+
/**
* getPartitionType LIST/RANGE/UNPARTITIONED
*
@@ -92,7 +121,7 @@ public interface MTMVRelatedTableIf extends TableIf {
* Does the current type of table allow timed triggering
*
* @return If return false,The method of comparing whether to synchronize will directly return true,
- * otherwise the snapshot information will be compared
+ * otherwise the snapshot information will be compared
*/
boolean needAutoRefresh();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVUtil.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVUtil.java
index 3b97e35141d..16459fa1303 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVUtil.java
@@ -25,8 +25,21 @@ import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.executable.DateTimeAcquire;
+import org.apache.doris.nereids.trees.expressions.functions.executable.DateTimeArithmetic;
+import org.apache.doris.nereids.trees.expressions.functions.executable.DateTimeExtractAndTransform;
+import org.apache.doris.nereids.trees.expressions.literal.DateTimeLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral;
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.Map;
+import java.util.Optional;
import java.util.Set;
public class MTMVUtil {
@@ -52,7 +65,7 @@ public class MTMVUtil {
}
/**
- * if base tables of mtmv contains external table
+ * if base tables of mtmv contains external table
*
* @param mtmv
* @return
@@ -66,4 +79,116 @@ public class MTMVUtil {
}
return false;
}
+
+ /**
+ * Obtain the minimum second from `syncLimit` `timeUnit` ago
+ *
+ * @param timeUnit
+ * @param syncLimit
+ * @return
+ * @throws AnalysisException
+ */
+ public static long getNowTruncSubSec(MTMVPartitionSyncTimeUnit timeUnit, int syncLimit)
+ throws AnalysisException {
+ if (syncLimit < 1) {
+ throw new AnalysisException("Unexpected syncLimit, syncLimit: " + syncLimit);
+ }
+ // get current time
+ Expression now = DateTimeAcquire.now();
+ if (!(now instanceof DateTimeLiteral)) {
+ throw new AnalysisException("now() should return DateTimeLiteral, now: " + now);
+ }
+ DateTimeLiteral nowLiteral = (DateTimeLiteral) now;
+ // date trunc
+ now = DateTimeExtractAndTransform
+ .dateTrunc(nowLiteral, new VarcharLiteral(timeUnit.name()));
+ if (!(now instanceof DateTimeLiteral)) {
+ throw new AnalysisException("dateTrunc() should return DateTimeLiteral, now: " + now);
+ }
+ nowLiteral = (DateTimeLiteral) now;
+ // date sub
+ if (syncLimit > 1) {
+ nowLiteral = dateSub(nowLiteral, timeUnit, syncLimit - 1);
+ }
+ return ((IntegerLiteral) DateTimeExtractAndTransform.unixTimestamp(nowLiteral)).getValue();
+ }
+
+ private static DateTimeLiteral dateSub(
+ org.apache.doris.nereids.trees.expressions.literal.DateLiteral date, MTMVPartitionSyncTimeUnit timeUnit,
+ int num)
+ throws AnalysisException {
+ IntegerLiteral integerLiteral = new IntegerLiteral(num);
+ Expression result;
+ switch (timeUnit) {
+ case DAY:
+ result = DateTimeArithmetic.dateSub(date, integerLiteral);
+ break;
+ case YEAR:
+ result = DateTimeArithmetic.yearsSub(date, integerLiteral);
+ break;
+ case MONTH:
+ result = DateTimeArithmetic.monthsSub(date, integerLiteral);
+ break;
+ default:
+ throw new AnalysisException("MTMV partition limit not support timeUnit: " + timeUnit.name());
+ }
+ if (!(result instanceof DateTimeLiteral)) {
+ throw new AnalysisException("sub() should return DateTimeLiteral, result: " + result);
+ }
+ return (DateTimeLiteral) result;
+ }
+
+ /**
+ * Convert LiteralExpr to second
+ *
+ * @param expr
+ * @param dateFormatOptional
+ * @return
+ * @throws AnalysisException
+ */
+ public static long getExprTimeSec(org.apache.doris.analysis.LiteralExpr expr, Optional<String> dateFormatOptional)
+ throws AnalysisException {
+ if (expr instanceof org.apache.doris.analysis.MaxLiteral) {
+ return Long.MAX_VALUE;
+ }
+ if (expr instanceof org.apache.doris.analysis.NullLiteral) {
+ return Long.MIN_VALUE;
+ }
+ if (expr instanceof org.apache.doris.analysis.DateLiteral) {
+ return ((org.apache.doris.analysis.DateLiteral) expr).unixTimestamp(TimeUtils.getTimeZone()) / 1000;
+ }
+ if (!dateFormatOptional.isPresent()) {
+ throw new AnalysisException("expr is not DateLiteral and DateFormat is not present.");
+ }
+ String dateFormat = dateFormatOptional.get();
+ Expression strToDate = DateTimeExtractAndTransform
+ .strToDate(new VarcharLiteral(expr.getStringValue()), new VarcharLiteral(dateFormat));
+ if (!(strToDate instanceof DateTimeLiteral)) {
+ throw new AnalysisException(
+ String.format("strToDate failed, stringValue: %s, dateFormat: %s", expr.getStringValue(),
+ dateFormat));
+ }
+ return ((IntegerLiteral) DateTimeExtractAndTransform.unixTimestamp((DateTimeLiteral) strToDate)).getValue();
+ }
+
+ /**
+ * Generate MTMVPartitionSyncConfig based on mvProperties
+ *
+ * @param mvProperties
+ * @return
+ */
+ public static MTMVPartitionSyncConfig generateMTMVPartitionSyncConfigByProperties(
+ Map<String, String> mvProperties) {
+ int syncLimit = StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT)) ? -1
+ : Integer.parseInt(mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT));
+ MTMVPartitionSyncTimeUnit timeUnit =
+ StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_TIME_UNIT))
+ ? MTMVPartitionSyncTimeUnit.DAY : MTMVPartitionSyncTimeUnit
+ .valueOf(mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_TIME_UNIT).toUpperCase());
+ Optional<String> dateFormat =
+ StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_DATE_FORMAT))
+ ? Optional.empty()
+ : Optional.of(mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_DATE_FORMAT));
+ return new MTMVPartitionSyncConfig(syncLimit, timeUnit, dateFormat);
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVPropertyInfo.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVPropertyInfo.java
index b152aadc98d..1e0174bc1f0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVPropertyInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVPropertyInfo.java
@@ -19,13 +19,10 @@ package org.apache.doris.nereids.trees.plans.commands.info;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.UserException;
-import org.apache.doris.common.util.PropertyAnalyzer;
-import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.mtmv.MTMVPropertyUtil;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.qe.ConnectContext;
-import org.apache.commons.lang3.StringUtils;
-
import java.util.Map;
import java.util.Objects;
@@ -55,40 +52,8 @@ public class AlterMTMVPropertyInfo extends AlterMTMVInfo {
private void analyzeProperties() {
for (String key : properties.keySet()) {
- if (PropertyAnalyzer.PROPERTIES_GRACE_PERIOD.equals(key)) {
- String gracePeriod = properties.get(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD);
- try {
- Long.parseLong(gracePeriod);
- } catch (NumberFormatException e) {
- throw new org.apache.doris.nereids.exceptions.AnalysisException(
- "valid grace_period: " + properties.get(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD));
- }
- } else if (PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM.equals(key)) {
- String refreshPartitionNum = properties.get(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM);
- try {
- Integer.parseInt(refreshPartitionNum);
- } catch (NumberFormatException e) {
- throw new AnalysisException(
- "valid refresh_partition_num: " + properties
- .get(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM));
- }
- } else if (PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES.equals(key)) {
- // nothing
- } else if (PropertyAnalyzer.PROPERTIES_WORKLOAD_GROUP.equals(key)) {
- String workloadGroup = properties.get(PropertyAnalyzer.PROPERTIES_WORKLOAD_GROUP);
- if (!StringUtils.isEmpty(workloadGroup) && !Env.getCurrentEnv().getAccessManager()
- .checkWorkloadGroupPriv(ConnectContext.get(), workloadGroup, PrivPredicate.USAGE)) {
- String message = String
- .format("Access denied; you need (at least one of) "
- + "the %s privilege(s) to use workload group '%s'.",
- "USAGE/ADMIN", workloadGroup);
- throw new AnalysisException(message);
- }
- } else {
- throw new org.apache.doris.nereids.exceptions.AnalysisException("illegal key:" + key);
- }
+ MTMVPropertyUtil.analyzeProperty(key, properties.get(key));
}
-
}
public Map<String, String> getProperties() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
index 90c98a44294..d7c502a9eb8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
@@ -34,13 +34,13 @@ import org.apache.doris.catalog.View;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.FeNameFormat;
-import org.apache.doris.common.util.PropertyAnalyzer;
import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.mtmv.EnvInfo;
import org.apache.doris.mtmv.MTMVPartitionInfo;
import org.apache.doris.mtmv.MTMVPartitionInfo.MTMVPartitionType;
import org.apache.doris.mtmv.MTMVPartitionUtil;
import org.apache.doris.mtmv.MTMVPlanUtil;
+import org.apache.doris.mtmv.MTMVPropertyUtil;
import org.apache.doris.mtmv.MTMVRefreshInfo;
import org.apache.doris.mtmv.MTMVRelatedTableIf;
import org.apache.doris.mtmv.MTMVRelation;
@@ -177,46 +177,12 @@ public class CreateMTMVInfo {
}
private void analyzeProperties() {
- if (properties.containsKey(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD)) {
- String gracePeriod = properties.get(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD);
- try {
- Long.parseLong(gracePeriod);
- } catch (NumberFormatException e) {
- throw new AnalysisException(
- "valid grace_period: " + properties.get(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD));
- }
- mvProperties.put(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD, gracePeriod);
- properties.remove(PropertyAnalyzer.PROPERTIES_GRACE_PERIOD);
- }
- if (properties.containsKey(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM)) {
- String refreshPartitionNum = properties.get(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM);
- try {
- Integer.parseInt(refreshPartitionNum);
- } catch (NumberFormatException e) {
- throw new AnalysisException(
- "valid refresh_partition_num: " + properties
- .get(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM));
- }
- mvProperties.put(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM, refreshPartitionNum);
- properties.remove(PropertyAnalyzer.PROPERTIES_REFRESH_PARTITION_NUM);
- }
- if (properties.containsKey(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES)) {
- String excludedTriggerTables = properties.get(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES);
- mvProperties.put(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES, excludedTriggerTables);
- properties.remove(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES);
- }
- if (properties.containsKey(PropertyAnalyzer.PROPERTIES_WORKLOAD_GROUP)) {
- String workloadGroup = properties.get(PropertyAnalyzer.PROPERTIES_WORKLOAD_GROUP);
- if (!Env.getCurrentEnv().getAccessManager()
- .checkWorkloadGroupPriv(ConnectContext.get(), workloadGroup, PrivPredicate.USAGE)) {
- String message = String
- .format("Access denied;"
- + " you need (at least one of) the %s privilege(s) to use workload group '%s'.",
- "USAGE/ADMIN", workloadGroup);
- throw new AnalysisException(message);
+ for (String key : MTMVPropertyUtil.mvPropertyKeys) {
+ if (properties.containsKey(key)) {
+ MTMVPropertyUtil.analyzeProperty(key, properties.get(key));
+ mvProperties.put(key, properties.get(key));
+ properties.remove(key);
}
- mvProperties.put(PropertyAnalyzer.PROPERTIES_WORKLOAD_GROUP, workloadGroup);
- properties.remove(PropertyAnalyzer.PROPERTIES_WORKLOAD_GROUP);
}
}
@@ -324,7 +290,8 @@ public class CreateMTMVInfo {
List<AllPartitionDesc> allPartitionDescs = null;
try {
allPartitionDescs = MTMVPartitionUtil
- .getPartitionDescsByRelatedTable(relatedTable, properties, mvPartitionInfo.getRelatedCol());
+ .getPartitionDescsByRelatedTable(relatedTable, properties, mvPartitionInfo.getRelatedCol(),
+ mvProperties);
} catch (org.apache.doris.common.AnalysisException e) {
throw new AnalysisException("getPartitionDescsByRelatedTable failed", e);
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVUtilTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVUtilTest.java
new file mode 100644
index 00000000000..2e6df56cd1a
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVUtilTest.java
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mtmv;
+
+import org.apache.doris.analysis.DateLiteral;
+import org.apache.doris.analysis.IntLiteral;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.StringLiteral;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.nereids.trees.expressions.functions.executable.DateTimeAcquire;
+import org.apache.doris.nereids.trees.expressions.literal.DateTimeLiteral;
+
+import com.google.common.collect.Maps;
+import mockit.Expectations;
+import mockit.Mocked;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.Optional;
+
+public class MTMVUtilTest {
+ @Mocked
+ private DateTimeAcquire dateTimeAcquire;
+
+ @Test
+ public void testGenerateMTMVPartitionSyncConfigByProperties() throws AnalysisException {
+ Map<String, String> mvProperties = Maps.newHashMap();
+ MTMVPartitionSyncConfig config = MTMVUtil
+ .generateMTMVPartitionSyncConfigByProperties(mvProperties);
+ Assert.assertEquals(-1, config.getSyncLimit());
+ Assert.assertFalse(config.getDateFormat().isPresent());
+ Assert.assertEquals(MTMVPartitionSyncTimeUnit.DAY, config.getTimeUnit());
+
+ mvProperties.put(PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT, "1");
+ config = MTMVUtil.generateMTMVPartitionSyncConfigByProperties(mvProperties);
+ Assert.assertEquals(1, config.getSyncLimit());
+ Assert.assertFalse(config.getDateFormat().isPresent());
+ Assert.assertEquals(MTMVPartitionSyncTimeUnit.DAY, config.getTimeUnit());
+
+ mvProperties.put(PropertyAnalyzer.PROPERTIES_PARTITION_TIME_UNIT, "month");
+ config = MTMVUtil.generateMTMVPartitionSyncConfigByProperties(mvProperties);
+ Assert.assertEquals(1, config.getSyncLimit());
+ Assert.assertFalse(config.getDateFormat().isPresent());
+ Assert.assertEquals(MTMVPartitionSyncTimeUnit.MONTH, config.getTimeUnit());
+
+ mvProperties.put(PropertyAnalyzer.PROPERTIES_PARTITION_DATE_FORMAT, "%Y%m%d");
+ config = MTMVUtil.generateMTMVPartitionSyncConfigByProperties(mvProperties);
+ Assert.assertEquals(1, config.getSyncLimit());
+ Assert.assertEquals("%Y%m%d", config.getDateFormat().get());
+ Assert.assertEquals(MTMVPartitionSyncTimeUnit.MONTH, config.getTimeUnit());
+ }
+
+ @Test
+ public void testGetExprTimeSec() throws AnalysisException {
+ LiteralExpr expr = new DateLiteral("2020-01-01");
+ long exprTimeSec = MTMVUtil.getExprTimeSec(expr, Optional.empty());
+ Assert.assertEquals(1577808000L, exprTimeSec);
+ expr = new StringLiteral("2020-01-01");
+ exprTimeSec = MTMVUtil.getExprTimeSec(expr, Optional.of("%Y-%m-%d"));
+ Assert.assertEquals(1577808000L, exprTimeSec);
+ expr = new IntLiteral(20200101);
+ exprTimeSec = MTMVUtil.getExprTimeSec(expr, Optional.of("%Y%m%d"));
+ Assert.assertEquals(1577808000L, exprTimeSec);
+ expr = new DateLiteral(Type.DATE, true);
+ exprTimeSec = MTMVUtil.getExprTimeSec(expr, Optional.empty());
+ Assert.assertEquals(253402185600L, exprTimeSec);
+ }
+
+ @Test
+ public void testGetNowTruncSubSec() throws AnalysisException {
+ DateTimeLiteral dateTimeLiteral = new DateTimeLiteral("2020-02-03 20:10:10");
+ new Expectations() {
+ {
+ dateTimeAcquire.now();
+ minTimes = 0;
+ result = dateTimeLiteral;
+ }
+ };
+ long nowTruncSubSec = MTMVUtil.getNowTruncSubSec(MTMVPartitionSyncTimeUnit.DAY, 1);
+ // 2020-02-03
+ Assert.assertEquals(1580659200L, nowTruncSubSec);
+ nowTruncSubSec = MTMVUtil.getNowTruncSubSec(MTMVPartitionSyncTimeUnit.MONTH, 1);
+ // 2020-02-01
+ Assert.assertEquals(1580486400L, nowTruncSubSec);
+ nowTruncSubSec = MTMVUtil.getNowTruncSubSec(MTMVPartitionSyncTimeUnit.YEAR, 1);
+ // 2020-01-01
+ Assert.assertEquals(1577808000L, nowTruncSubSec);
+ nowTruncSubSec = MTMVUtil.getNowTruncSubSec(MTMVPartitionSyncTimeUnit.MONTH, 3);
+ // 2019-12-01
+ Assert.assertEquals(1575129600L, nowTruncSubSec);
+ nowTruncSubSec = MTMVUtil.getNowTruncSubSec(MTMVPartitionSyncTimeUnit.DAY, 4);
+ // 2020-01-31
+ Assert.assertEquals(1580400000L, nowTruncSubSec);
+ }
+}
diff --git a/regression-test/data/mtmv_p0/test_hive_limit_partition_mtmv.out b/regression-test/data/mtmv_p0/test_hive_limit_partition_mtmv.out
new file mode 100644
index 00000000000..3f781e54612
--- /dev/null
+++ b/regression-test/data/mtmv_p0/test_hive_limit_partition_mtmv.out
@@ -0,0 +1,11 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !select_base_table --
+1 bj 20380101
+2 sh 20380101
+3 bj 20200101
+4 sh 20200101
+
+-- !mtmv_complete --
+1 20380101 bj
+2 20380101 sh
+
diff --git a/regression-test/data/mtmv_p0/test_limit_partition_mtmv.out b/regression-test/data/mtmv_p0/test_limit_partition_mtmv.out
new file mode 100644
index 00000000000..a655b5fdeff
--- /dev/null
+++ b/regression-test/data/mtmv_p0/test_limit_partition_mtmv.out
@@ -0,0 +1,17 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !date_list --
+1 2038-01-01
+
+-- !varchar_list --
+1 20380101
+
+-- !varchar_list --
+1 20380101
+
+-- !date_range --
+1 2038-01-02
+
+-- !date_range_all --
+1 2038-01-02
+2 2020-01-02
+
diff --git a/regression-test/suites/mtmv_p0/test_hive_limit_partition_mtmv.groovy b/regression-test/suites/mtmv_p0/test_hive_limit_partition_mtmv.groovy
new file mode 100644
index 00000000000..f55df2bd924
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/test_hive_limit_partition_mtmv.groovy
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_hive_limit_partition_mtmv", "p0,external,hive,external_docker,external_docker_hive") {
+ String enabled = context.config.otherConfigs.get("enableHiveTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("diable Hive test.")
+ return;
+ }
+ // prepare data in hive
+ def hive_database = "test_hive_limit_partition_mtmv_db"
+ def hive_table = "partition2"
+
+ def drop_table_str = """ drop table if exists ${hive_database}.${hive_table} """
+ def drop_database_str = """ drop database if exists ${hive_database}"""
+ def create_database_str = """ create database ${hive_database}"""
+ def create_table_str = """ CREATE TABLE ${hive_database}.${hive_table} (
+ `k1` int)
+ PARTITIONED BY (
+ `region` string,
+ `day` string
+ )
+ STORED AS ORC;
+ """
+ def add_partition_str = """
+ alter table ${hive_database}.${hive_table} add if not exists
+ partition(region="bj",day="20380101")
+ partition(region="sh",day="20380101")
+ partition(region="bj",day="20200101")
+ partition(region="sh",day="20200101")
+ """
+ def insert_str1 = """insert into ${hive_database}.${hive_table} PARTITION(region="bj",day="20380101") values(1)"""
+ def insert_str2 = """insert into ${hive_database}.${hive_table} PARTITION(region="sh",day="20380101") values(2)"""
+ def insert_str3 = """insert into ${hive_database}.${hive_table} PARTITION(region="bj",day="20200101") values(3)"""
+ def insert_str4 = """insert into ${hive_database}.${hive_table} PARTITION(region="sh",day="20200101") values(4)"""
+
+ logger.info("hive sql: " + drop_table_str)
+ hive_docker """ ${drop_table_str} """
+ logger.info("hive sql: " + drop_database_str)
+ hive_docker """ ${drop_database_str} """
+ logger.info("hive sql: " + create_database_str)
+ hive_docker """ ${create_database_str}"""
+ logger.info("hive sql: " + create_table_str)
+ hive_docker """ ${create_table_str} """
+ logger.info("hive sql: " + add_partition_str)
+ hive_docker """ ${add_partition_str} """
+ logger.info("hive sql: " + insert_str1)
+ hive_docker """ ${insert_str1} """
+ logger.info("hive sql: " + insert_str2)
+ hive_docker """ ${insert_str2} """
+ logger.info("hive sql: " + insert_str3)
+ hive_docker """ ${insert_str3} """
+ logger.info("hive sql: " + insert_str4)
+ hive_docker """ ${insert_str4} """
+
+ // prepare catalog
+ String hms_port = context.config.otherConfigs.get("hms_port")
+ String catalog_name = "test_hive_limit_partition_mtmv_catalog"
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+
+ sql """drop catalog if exists ${catalog_name}"""
+ sql """create catalog if not exists ${catalog_name} properties (
+ "type"="hms",
+ 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}'
+ );"""
+
+ order_qt_select_base_table "SELECT * FROM ${catalog_name}.${hive_database}.${hive_table}"
+
+
+ // string type
+ def mvName = "test_hive_limit_partition_mtmv"
+ def dbName = "regression_test_mtmv_p0"
+ sql """drop materialized view if exists ${mvName};"""
+ sql """REFRESH catalog ${catalog_name}"""
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ partition by(`day`)
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES (
+ 'replication_num' = '1',
+ 'partition_sync_limit'='2',
+ 'partition_sync_time_unit'='MONTH',
+ 'partition_date_format'='%Y%m%d'
+ )
+ AS
+ SELECT k1,day,region FROM ${catalog_name}.${hive_database}.${hive_table};
+ """
+ def showPartitionsResult = sql """show partitions from ${mvName}"""
+ logger.info("showPartitionsResult: " + showPartitionsResult.toString())
+ assertEquals(1, showPartitionsResult.size())
+ assertTrue(showPartitionsResult.toString().contains("p_20380101"))
+
+ // refresh complete
+ sql """
+ REFRESH MATERIALIZED VIEW ${mvName} complete
+ """
+ def jobName = getJobName(dbName, mvName);
+ waitingMTMVTaskFinished(jobName)
+ order_qt_mtmv_complete "SELECT * FROM ${mvName} order by k1,day,region"
+
+
+ // date type
+ sql """drop materialized view if exists ${mvName};"""
+ create_table_str = """ CREATE TABLE ${hive_database}.${hive_table} (
+ `k1` int)
+ PARTITIONED BY (
+ `region` string,
+ `day` date
+ )
+ STORED AS ORC;
+ """
+ add_partition_str = """
+ alter table ${hive_database}.${hive_table} add if not exists
+ partition(region="bj",day="2038-01-01")
+ partition(region="sh",day="2038-01-01")
+ partition(region="bj",day="2020-01-01")
+ partition(region="sh",day="2020-01-01")
+ """
+ logger.info("hive sql: " + drop_table_str)
+ hive_docker """ ${drop_table_str} """
+ logger.info("hive sql: " + create_table_str)
+ hive_docker """ ${create_table_str} """
+ logger.info("hive sql: " + add_partition_str)
+ hive_docker """ ${add_partition_str} """
+
+ sql """REFRESH catalog ${catalog_name}"""
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ partition by(`day`)
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES (
+ 'replication_num' = '1',
+ 'partition_sync_limit'='2',
+ 'partition_sync_time_unit'='YEAR'
+ )
+ AS
+ SELECT k1,day,region FROM ${catalog_name}.${hive_database}.${hive_table};
+ """
+ showPartitionsResult = sql """show partitions from ${mvName}"""
+ logger.info("showPartitionsResult: " + showPartitionsResult.toString())
+ assertEquals(1, showPartitionsResult.size())
+ assertTrue(showPartitionsResult.toString().contains("p_20380101"))
+ sql """drop materialized view if exists ${mvName};"""
+ sql """drop catalog if exists ${catalog_name}"""
+}
+
diff --git a/regression-test/suites/mtmv_p0/test_limit_partition_mtmv.groovy b/regression-test/suites/mtmv_p0/test_limit_partition_mtmv.groovy
new file mode 100644
index 00000000000..fd026a6c02e
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/test_limit_partition_mtmv.groovy
@@ -0,0 +1,240 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.junit.Assert;
+
+suite("test_limit_partition_mtmv") {
+ def tableName = "t_test_limit_partition_mtmv_user"
+ def mvName = "multi_mv_test_limit_partition_mtmv"
+ def dbName = "regression_test_mtmv_p0"
+
+ // list partition date type
+ sql """drop table if exists `${tableName}`"""
+ sql """drop materialized view if exists ${mvName};"""
+ sql """
+ CREATE TABLE `${tableName}` (
+ `k1` LARGEINT NOT NULL COMMENT '\"用户id\"',
+ `k2` DATE NOT NULL COMMENT '\"数据灌入日期时间\"'
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`k1`)
+ COMMENT 'OLAP'
+ PARTITION BY list(`k2`)
+ (
+ PARTITION p_20380101 VALUES IN ("2038-01-01"),
+ PARTITION p_20200101 VALUES IN ("2020-01-01")
+ )
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 2
+ PROPERTIES ('replication_num' = '1') ;
+ """
+ sql """
+ insert into ${tableName} values(1,"2038-01-01"),(2,"2020-01-01");
+ """
+
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ partition by(`k2`)
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES (
+ 'replication_num' = '1',
+ 'partition_sync_limit'='2',
+ 'partition_sync_time_unit'='YEAR'
+ )
+ AS
+ SELECT * FROM ${tableName};
+ """
+ showPartitionsResult = sql """show partitions from ${mvName}"""
+ logger.info("showPartitionsResult: " + showPartitionsResult.toString())
+ assertEquals(1, showPartitionsResult.size())
+ assertTrue(showPartitionsResult.toString().contains("p_20380101"))
+
+ sql """
+ REFRESH MATERIALIZED VIEW ${mvName}
+ """
+ def jobName = getJobName(dbName, mvName);
+ log.info(jobName)
+ waitingMTMVTaskFinished(jobName)
+ order_qt_date_list "SELECT * FROM ${mvName} order by k1,k2"
+
+
+
+ // list partition string type
+ sql """drop table if exists `${tableName}`"""
+ sql """drop materialized view if exists ${mvName};"""
+ sql """
+ CREATE TABLE `${tableName}` (
+ `k1` LARGEINT NOT NULL COMMENT '\"用户id\"',
+ `k2` VARCHAR(100) NOT NULL COMMENT '\"数据灌入日期时间\"'
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`k1`)
+ COMMENT 'OLAP'
+ PARTITION BY list(`k2`)
+ (
+ PARTITION p_20380101 VALUES IN ("20380101"),
+ PARTITION p_20200101 VALUES IN ("20200101")
+ )
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 2
+ PROPERTIES ('replication_num' = '1') ;
+ """
+ sql """
+ insert into ${tableName} values(1,"20380101"),(2,"20200101");
+ """
+
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ partition by(`k2`)
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES (
+ 'replication_num' = '1',
+ 'partition_sync_limit'='2',
+ 'partition_sync_time_unit'='MONTH',
+ 'partition_date_format'='%Y%m%d'
+ )
+ AS
+ SELECT * FROM ${tableName};
+ """
+ showPartitionsResult = sql """show partitions from ${mvName}"""
+ logger.info("showPartitionsResult: " + showPartitionsResult.toString())
+ assertEquals(1, showPartitionsResult.size())
+ assertTrue(showPartitionsResult.toString().contains("p_20380101"))
+
+ sql """
+ REFRESH MATERIALIZED VIEW ${mvName}
+ """
+ jobName = getJobName(dbName, mvName);
+ log.info(jobName)
+ waitingMTMVTaskFinished(jobName)
+ order_qt_varchar_list "SELECT * FROM ${mvName} order by k1,k2"
+
+
+ // list partition int type
+ sql """drop table if exists `${tableName}`"""
+ sql """drop materialized view if exists ${mvName};"""
+ sql """
+ CREATE TABLE `${tableName}` (
+ `k1` LARGEINT NOT NULL COMMENT '\"用户id\"',
+ `k2` int NOT NULL COMMENT '\"数据灌入日期时间\"'
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`k1`)
+ COMMENT 'OLAP'
+ PARTITION BY list(`k2`)
+ (
+ PARTITION p_20380101 VALUES IN ("20380101"),
+ PARTITION p_20200101 VALUES IN ("20200101")
+ )
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 2
+ PROPERTIES ('replication_num' = '1') ;
+ """
+ sql """
+ insert into ${tableName} values(1,20380101),(2,20200101);
+ """
+
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ partition by(`k2`)
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES (
+ 'replication_num' = '1',
+ 'partition_sync_limit'='2',
+ 'partition_sync_time_unit'='DAY',
+ 'partition_date_format'='%Y%m%d'
+ )
+ AS
+ SELECT * FROM ${tableName};
+ """
+ showPartitionsResult = sql """show partitions from ${mvName}"""
+ logger.info("showPartitionsResult: " + showPartitionsResult.toString())
+ assertEquals(1, showPartitionsResult.size())
+ assertTrue(showPartitionsResult.toString().contains("p_20380101"))
+
+ sql """
+ REFRESH MATERIALIZED VIEW ${mvName}
+ """
+ jobName = getJobName(dbName, mvName);
+ log.info(jobName)
+ waitingMTMVTaskFinished(jobName)
+ order_qt_varchar_list "SELECT * FROM ${mvName} order by k1,k2"
+
+
+ // range partition date type
+ sql """drop table if exists `${tableName}`"""
+ sql """drop materialized view if exists ${mvName};"""
+ sql """
+ CREATE TABLE `${tableName}` (
+ `k1` LARGEINT NOT NULL COMMENT '\"用户id\"',
+ `k2` DATE NOT NULL COMMENT '\"数据灌入日期时间\"'
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`k1`)
+ COMMENT 'OLAP'
+ PARTITION BY range(`k2`)
+ (
+ PARTITION p2038 VALUES [("2038-01-01"),("2038-01-03")),
+ PARTITION p2020 VALUES [("2020-01-01"),("2020-01-03"))
+ )
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 2
+ PROPERTIES ('replication_num' = '1') ;
+ """
+ sql """
+ insert into ${tableName} values(1,"2038-01-02"),(2,"2020-01-02");
+ """
+
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH AUTO ON MANUAL
+ partition by(`k2`)
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES (
+ 'replication_num' = '1',
+ 'partition_sync_limit'='2',
+ 'partition_sync_time_unit'='YEAR'
+ )
+ AS
+ SELECT * FROM ${tableName};
+ """
+ showPartitionsResult = sql """show partitions from ${mvName}"""
+ logger.info("showPartitionsResult: " + showPartitionsResult.toString())
+ assertEquals(1, showPartitionsResult.size())
+ assertTrue(showPartitionsResult.toString().contains("p_20380101_20380103"))
+
+ sql """
+ REFRESH MATERIALIZED VIEW ${mvName}
+ """
+ jobName = getJobName(dbName, mvName);
+ log.info(jobName)
+ waitingMTMVTaskFinished(jobName)
+ order_qt_date_range "SELECT * FROM ${mvName} order by k1,k2"
+
+
+ // alter
+ sql """
+ alter Materialized View ${mvName} set("partition_sync_limit"="");
+ """
+ sql """
+ REFRESH MATERIALIZED VIEW ${mvName}
+ """
+ jobName = getJobName(dbName, mvName);
+ log.info(jobName)
+ waitingMTMVTaskFinished(jobName)
+ showPartitionsResult = sql """show partitions from ${mvName}"""
+ logger.info("showPartitionsResult: " + showPartitionsResult.toString())
+ assertEquals(2, showPartitionsResult.size())
+ assertTrue(showPartitionsResult.toString().contains("p_20380101_20380103"))
+ assertTrue(showPartitionsResult.toString().contains("p_20200101_20200103"))
+ order_qt_date_range_all "SELECT * FROM ${mvName} order by k1,k2"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 35/38: [chore](build) Using multithread to accelerate FE compilation (#31855)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 917b201174757a593d3d5e109c90790d07162001
Author: yagagagaga <zh...@gmail.com>
AuthorDate: Thu Mar 7 13:56:47 2024 +0800
[chore](build) Using multithread to accelerate FE compilation (#31855)
---
build.sh | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/build.sh b/build.sh
index ffc73dcab23..9b2ea885cd1 100755
--- a/build.sh
+++ b/build.sh
@@ -587,15 +587,15 @@ if [[ "${FE_MODULES}" != '' ]]; then
if [[ "${DISABLE_JAVA_CHECK_STYLE}" = "ON" ]]; then
# Allowed user customer set env param USER_SETTINGS_MVN_REPO means settings.xml file path
if [[ -n ${USER_SETTINGS_MVN_REPO} && -f ${USER_SETTINGS_MVN_REPO} ]]; then
- "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -Dskip.doc=true -DskipTests -Dcheckstyle.skip=true ${MVN_OPT:+${MVN_OPT}} -gs "${USER_SETTINGS_MVN_REPO}"
+ "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -Dskip.doc=true -DskipTests -Dcheckstyle.skip=true ${MVN_OPT:+${MVN_OPT}} -gs "${USER_SETTINGS_MVN_REPO}" -T 1C
else
- "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -Dskip.doc=true -DskipTests -Dcheckstyle.skip=true ${MVN_OPT:+${MVN_OPT}}
+ "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -Dskip.doc=true -DskipTests -Dcheckstyle.skip=true ${MVN_OPT:+${MVN_OPT}} -T 1C
fi
else
if [[ -n ${USER_SETTINGS_MVN_REPO} && -f ${USER_SETTINGS_MVN_REPO} ]]; then
- "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -Dskip.doc=true -DskipTests ${MVN_OPT:+${MVN_OPT}} -gs "${USER_SETTINGS_MVN_REPO}"
+ "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -Dskip.doc=true -DskipTests ${MVN_OPT:+${MVN_OPT}} -gs "${USER_SETTINGS_MVN_REPO}" -T 1C
else
- "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -Dskip.doc=true -DskipTests ${MVN_OPT:+${MVN_OPT}}
+ "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -Dskip.doc=true -DskipTests ${MVN_OPT:+${MVN_OPT}} -T 1C
fi
fi
cd "${DORIS_HOME}"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 07/38: [opt](session variable) max_msg_size_of_result_receiver #31809
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2e4201e4066fec38deb22494248473fd3dbbc9e3
Author: zhiqiang <se...@163.com>
AuthorDate: Wed Mar 6 14:50:27 2024 +0800
[opt](session variable) max_msg_size_of_result_receiver #31809
---
.../main/java/org/apache/doris/qe/Coordinator.java | 3 +-
.../java/org/apache/doris/qe/PointQueryExec.java | 19 +++++--
.../java/org/apache/doris/qe/ResultReceiver.java | 22 ++++++--
.../java/org/apache/doris/qe/SessionVariable.java | 14 +++++
.../java/org/apache/doris/qe/StmtExecutor.java | 3 +-
.../apache/doris/rpc/TCustomProtocolFactory.java | 41 +++++++++++++++
.../max_msg_size_of_result_receiver.groovy | 61 ++++++++++++++++++++++
7 files changed, 155 insertions(+), 8 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index 3ab3d40dc7b..b91fcd15e3c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -659,7 +659,8 @@ public class Coordinator implements CoordInterface {
if (topDataSink instanceof ResultSink || topDataSink instanceof ResultFileSink) {
TNetworkAddress execBeAddr = topParams.instanceExecParams.get(0).host;
receiver = new ResultReceiver(queryId, topParams.instanceExecParams.get(0).instanceId,
- addressToBackendID.get(execBeAddr), toBrpcHost(execBeAddr), this.timeoutDeadline);
+ addressToBackendID.get(execBeAddr), toBrpcHost(execBeAddr), this.timeoutDeadline,
+ context.getSessionVariable().getMaxMsgSizeOfResultReceiver());
if (!context.isReturnResultFromLocal()) {
Preconditions.checkState(context.getConnectType().equals(ConnectType.ARROW_FLIGHT_SQL));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExec.java b/fe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExec.java
index 40d244184b2..cf61d33b9a9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExec.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExec.java
@@ -35,6 +35,7 @@ import org.apache.doris.proto.InternalService.KeyTuple;
import org.apache.doris.proto.Types;
import org.apache.doris.rpc.BackendServiceProxy;
import org.apache.doris.rpc.RpcException;
+import org.apache.doris.rpc.TCustomProtocolFactory;
import org.apache.doris.system.Backend;
import org.apache.doris.thrift.TExpr;
import org.apache.doris.thrift.TExprList;
@@ -85,6 +86,8 @@ public class PointQueryExec implements CoordInterface {
// using this ID to find for this prepared statement
private UUID cacheID;
+ private final int maxMsgSizeOfResultReceiver;
+
private OlapScanNode getPlanRoot() {
List<PlanFragment> fragments = planner.getFragments();
PlanFragment fragment = fragments.get(0);
@@ -96,7 +99,7 @@ public class PointQueryExec implements CoordInterface {
return planRoot;
}
- public PointQueryExec(Planner planner, Analyzer analyzer) {
+ public PointQueryExec(Planner planner, Analyzer analyzer, int maxMessageSize) {
// init from planner
this.planner = planner;
List<PlanFragment> fragments = planner.getFragments();
@@ -117,6 +120,7 @@ public class PointQueryExec implements CoordInterface {
// TODO
// planner.getDescTable().toThrift();
}
+ this.maxMsgSizeOfResultReceiver = maxMessageSize;
}
void setScanRangeLocations() throws Exception {
@@ -310,8 +314,17 @@ public class PointQueryExec implements CoordInterface {
} else if (pResult.hasRowBatch() && pResult.getRowBatch().size() > 0) {
byte[] serialResult = pResult.getRowBatch().toByteArray();
TResultBatch resultBatch = new TResultBatch();
- TDeserializer deserializer = new TDeserializer();
- deserializer.deserialize(resultBatch, serialResult);
+ TDeserializer deserializer = new TDeserializer(
+ new TCustomProtocolFactory(this.maxMsgSizeOfResultReceiver));
+ try {
+ deserializer.deserialize(resultBatch, serialResult);
+ } catch (TException e) {
+ if (e.getMessage().contains("MaxMessageSize reached")) {
+ throw new TException("MaxMessageSize reached, try increase max_msg_size_of_result_receiver");
+ } else {
+ throw e;
+ }
+ }
rowBatch.setBatch(resultBatch);
rowBatch.setEos(true);
return rowBatch;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ResultReceiver.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ResultReceiver.java
index a9e9740963f..c473d74b919 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ResultReceiver.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ResultReceiver.java
@@ -23,6 +23,7 @@ import org.apache.doris.proto.InternalService;
import org.apache.doris.proto.Types;
import org.apache.doris.rpc.BackendServiceProxy;
import org.apache.doris.rpc.RpcException;
+import org.apache.doris.rpc.TCustomProtocolFactory;
import org.apache.doris.thrift.TNetworkAddress;
import org.apache.doris.thrift.TResultBatch;
import org.apache.doris.thrift.TStatusCode;
@@ -53,12 +54,16 @@ public class ResultReceiver {
private Future<InternalService.PFetchDataResult> fetchDataAsyncFuture = null;
public String cancelReason = "";
- public ResultReceiver(TUniqueId queryId, TUniqueId tid, Long backendId, TNetworkAddress address, long timeoutTs) {
+ int maxMsgSizeOfResultReceiver;
+
+ public ResultReceiver(TUniqueId queryId, TUniqueId tid, Long backendId, TNetworkAddress address, long timeoutTs,
+ int maxMsgSizeOfResultReceiver) {
this.queryId = Types.PUniqueId.newBuilder().setHi(queryId.hi).setLo(queryId.lo).build();
this.finstId = Types.PUniqueId.newBuilder().setHi(tid.hi).setLo(tid.lo).build();
this.backendId = backendId;
this.address = address;
this.timeoutTs = timeoutTs;
+ this.maxMsgSizeOfResultReceiver = maxMsgSizeOfResultReceiver;
}
public RowBatch getNext(Status status) throws TException {
@@ -136,8 +141,19 @@ public class ResultReceiver {
} else if (pResult.hasRowBatch() && pResult.getRowBatch().size() > 0) {
byte[] serialResult = pResult.getRowBatch().toByteArray();
TResultBatch resultBatch = new TResultBatch();
- TDeserializer deserializer = new TDeserializer();
- deserializer.deserialize(resultBatch, serialResult);
+ TDeserializer deserializer = new TDeserializer(
+ new TCustomProtocolFactory(this.maxMsgSizeOfResultReceiver));
+ try {
+ deserializer.deserialize(resultBatch, serialResult);
+ } catch (TException e) {
+ if (e.getMessage().contains("MaxMessageSize reached")) {
+ throw new TException(
+ "MaxMessageSize reached, try increase max_msg_size_of_result_receiver");
+ } else {
+ throw e;
+ }
+ }
+
rowBatch.setBatch(resultBatch);
rowBatch.setEos(pResult.getEos());
return rowBatch;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 1e137ad1192..164e06e0856 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -45,6 +45,7 @@ import com.google.common.collect.Lists;
import org.apache.commons.lang3.StringUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import org.apache.thrift.TConfiguration;
import org.json.simple.JSONObject;
import org.json.simple.JSONValue;
@@ -515,6 +516,8 @@ public class SessionVariable implements Serializable, Writable {
public static final String SHOW_ALL_FE_CONNECTION = "show_all_fe_connection";
+ public static final String MAX_MSG_SIZE_OF_RESULT_RECEIVER = "max_msg_size_of_result_receiver";
+
public static final List<String> DEBUG_VARIABLES = ImmutableList.of(
SKIP_DELETE_PREDICATE,
SKIP_DELETE_BITMAP,
@@ -1636,6 +1639,13 @@ public class SessionVariable implements Serializable, Writable {
"当变量为true时，show processlist命令展示所有fe的连接"})
public boolean showAllFeConnection = false;
+ @VariableMgr.VarAttr(name = MAX_MSG_SIZE_OF_RESULT_RECEIVER,
+ description = {"Max message size during result deserialization, change this if you meet error"
+ + " like \"MaxMessageSize reached\"",
+ "用于控制结果反序列化时 thrift 字段的最大值，当遇到类似\"MaxMessageSize reached\"这样的错误时可以考虑修改该参数"})
+ public int maxMsgSizeOfResultReceiver = TConfiguration.DEFAULT_MAX_MESSAGE_SIZE;
+
+
// CLOUD_VARIABLES_BEGIN
@VariableMgr.VarAttr(name = CLOUD_CLUSTER)
public String cloudCluster = "";
@@ -3450,4 +3460,8 @@ public class SessionVariable implements Serializable, Writable {
return this.showAllFeConnection;
}
+ public int getMaxMsgSizeOfResultReceiver() {
+ return this.maxMsgSizeOfResultReceiver;
+ }
+
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 71af58481d3..808ec0de09d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -1569,7 +1569,8 @@ public class StmtExecutor {
RowBatch batch;
CoordInterface coordBase = null;
if (queryStmt instanceof SelectStmt && ((SelectStmt) parsedStmt).isPointQueryShortCircuit()) {
- coordBase = new PointQueryExec(planner, analyzer);
+ coordBase = new PointQueryExec(planner, analyzer,
+ context.getSessionVariable().getMaxMsgSizeOfResultReceiver());
} else {
coord = new Coordinator(context, analyzer, planner, context.getStatsErrorEstimator());
QeProcessorImpl.INSTANCE.registerQuery(context.queryId(),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/rpc/TCustomProtocolFactory.java b/fe/fe-core/src/main/java/org/apache/doris/rpc/TCustomProtocolFactory.java
new file mode 100644
index 00000000000..ca586df1e39
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/rpc/TCustomProtocolFactory.java
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.rpc;
+
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.protocol.TProtocol;
+import org.apache.thrift.protocol.TProtocolFactory;
+import org.apache.thrift.transport.TTransport;
+
+/*
+* CustomProtocolFactory can change maxMessageSize of TTransport,
+* so that we can transfer message whose filed size is very large.
+*/
+public class TCustomProtocolFactory implements TProtocolFactory {
+ private final int maxMessageSize;
+
+ @Override
+ public TProtocol getProtocol(TTransport tTransport) {
+ tTransport.getConfiguration().setMaxMessageSize(maxMessageSize);
+ return new TBinaryProtocol(tTransport);
+ }
+
+ public TCustomProtocolFactory(int maxMessageSize) {
+ this.maxMessageSize = maxMessageSize;
+ }
+}
diff --git a/regression-test/suites/variable_p0/max_msg_size_of_result_receiver.groovy b/regression-test/suites/variable_p0/max_msg_size_of_result_receiver.groovy
new file mode 100644
index 00000000000..e7fead33d90
--- /dev/null
+++ b/regression-test/suites/variable_p0/max_msg_size_of_result_receiver.groovy
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("max_msg_size_of_result_receiver") {
+ def MESSAGE_SIZE_BASE=1000*1000; // 1MB
+ def DEFAULT_MAX_MESSAGE_SIZE = 104000000; // 104MB
+ def table_name = "max_msg_size_of_result_receiver"
+ sql """
+ DROP TABLE IF EXISTS ${table_name}
+ """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${table_name} (id int, str string)
+ ENGINE=OLAP DISTRIBUTED BY HASH(id)
+ PROPERTIES("replication_num"="1")
+ """
+
+ sql """
+ INSERT INTO ${table_name} VALUES (104, repeat("a", ${MESSAGE_SIZE_BASE * 104}))
+ """
+
+ sql """
+ INSERT INTO ${table_name} VALUES (105, repeat("a", ${MESSAGE_SIZE_BASE * 105}))
+ """
+
+ def with_exception = false
+ try {
+ sql "SELECT * FROM ${table_name} WHERE id = 104"
+ } catch (Exception e) {
+ with_exception = true
+ }
+ assertEquals(with_exception, false)
+
+ try {
+ sql "SELECT * FROM ${table_name} WHERE id = 105"
+ } catch (Exception e) {
+ assertTrue(e.getMessage().contains('MaxMessageSize reached, try increase max_msg_size_of_result_receiver'))
+ }
+
+ try {
+ sql "SELECT /*+SET_VAR(max_msg_size_of_result_receiver=${DEFAULT_MAX_MESSAGE_SIZE * 2})*/ * FROM ${table_name} WHERE id = 105"
+ } catch (Exception e) {
+ with_exception = true
+
+ }
+ assertEquals(with_exception, false)
+
+}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 12/38: [fix](multi-catalog) fix iceberg defalut credentials proviers (#31704)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 686cbffd8a799d77fe993eeac64d6161a301abb4
Author: slothever <18...@users.noreply.github.com>
AuthorDate: Wed Mar 6 20:40:10 2024 +0800
[fix](multi-catalog) fix iceberg defalut credentials proviers (#31704)
---
.../doris/datasource/iceberg/IcebergExternalCatalog.java | 9 +++++++++
.../datasource/iceberg/IcebergGlueExternalCatalog.java | 5 ++++-
.../datasource/iceberg/IcebergHMSExternalCatalog.java | 1 +
.../datasource/iceberg/IcebergHadoopExternalCatalog.java | 1 +
.../datasource/iceberg/IcebergRestExternalCatalog.java | 3 +--
.../doris/datasource/property/PropertyConverter.java | 16 +++++++++++++---
.../datasource/property/constants/S3Properties.java | 8 ++++++++
7 files changed, 37 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java
index 426657bc539..e5b8246571c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java
@@ -21,10 +21,14 @@ import org.apache.doris.datasource.ExternalCatalog;
import org.apache.doris.datasource.InitCatalogLog;
import org.apache.doris.datasource.SessionContext;
import org.apache.doris.datasource.operations.ExternalMetadataOperations;
+import org.apache.doris.datasource.property.PropertyConverter;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.s3a.Constants;
import org.apache.iceberg.catalog.Catalog;
import java.util.List;
+import java.util.Map;
public abstract class IcebergExternalCatalog extends ExternalCatalog {
@@ -76,4 +80,9 @@ public abstract class IcebergExternalCatalog extends ExternalCatalog {
makeSureInitialized();
return metadataOps.listTableNames(dbName);
}
+
+ protected void initS3Param(Configuration conf) {
+ Map<String, String> properties = catalogProperty.getHadoopProperties();
+ conf.set(Constants.AWS_CREDENTIALS_PROVIDER, PropertyConverter.getAWSCredentialsProviders(properties));
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergGlueExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergGlueExternalCatalog.java
index 08e7fe044ba..9f3e05757c9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergGlueExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergGlueExternalCatalog.java
@@ -21,6 +21,7 @@ import org.apache.doris.datasource.CatalogProperty;
import org.apache.doris.datasource.property.PropertyConverter;
import org.apache.doris.datasource.property.constants.S3Properties;
+import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.s3a.Constants;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.aws.glue.GlueCatalog;
@@ -45,7 +46,9 @@ public class IcebergGlueExternalCatalog extends IcebergExternalCatalog {
protected void initCatalog() {
icebergCatalogType = ICEBERG_GLUE;
GlueCatalog glueCatalog = new GlueCatalog();
- glueCatalog.setConf(getConfiguration());
+ Configuration conf = getConfiguration();
+ initS3Param(conf);
+ glueCatalog.setConf(conf);
// initialize glue catalog
Map<String, String> catalogProperties = catalogProperty.getHadoopProperties();
String warehouse = catalogProperty.getOrDefault(CatalogProperties.WAREHOUSE_LOCATION, CHECKED_WAREHOUSE);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHMSExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHMSExternalCatalog.java
index 875633da293..2ccf1e03e28 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHMSExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHMSExternalCatalog.java
@@ -60,6 +60,7 @@ public class IcebergHMSExternalCatalog extends IcebergExternalCatalog {
HadoopUGI.tryKrbLogin(this.getName(), AuthenticationConfig.getKerberosConfig(hiveConf,
AuthenticationConfig.HADOOP_KERBEROS_PRINCIPAL,
AuthenticationConfig.HADOOP_KERBEROS_KEYTAB));
+ initS3Param(hiveConf);
HMSCachedClient cachedClient = HiveMetadataOps.createCachedClient(hiveConf, 1, null);
String location = cachedClient.getCatalogLocation("hive");
catalogProperties.put(CatalogProperties.WAREHOUSE_LOCATION, location);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHadoopExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHadoopExternalCatalog.java
index 83bc70ce348..c9532f0b154 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHadoopExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHadoopExternalCatalog.java
@@ -56,6 +56,7 @@ public class IcebergHadoopExternalCatalog extends IcebergExternalCatalog {
icebergCatalogType = ICEBERG_HADOOP;
HadoopCatalog hadoopCatalog = new HadoopCatalog();
Configuration conf = getConfiguration();
+ initS3Param(conf);
// initialize hive catalog
Map<String, String> catalogProperties = new HashMap<>();
String warehouse = catalogProperty.getHadoopProperties().get(CatalogProperties.WAREHOUSE_LOCATION);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergRestExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergRestExternalCatalog.java
index e839b9a0017..77a6a7404ef 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergRestExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergRestExternalCatalog.java
@@ -54,8 +54,7 @@ public class IcebergRestExternalCatalog extends IcebergExternalCatalog {
private Configuration replaceS3Properties(Configuration conf) {
Map<String, String> catalogProperties = catalogProperty.getHadoopProperties();
- String defaultProviderList = String.join(",", S3Properties.AWS_CREDENTIALS_PROVIDERS);
- conf.set(Constants.AWS_CREDENTIALS_PROVIDER, defaultProviderList);
+ initS3Param(conf);
String usePahStyle = catalogProperties.getOrDefault(PropertyConverter.USE_PATH_STYLE, "true");
// Set path style
conf.set(PropertyConverter.USE_PATH_STYLE, usePahStyle);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
index 33859dc5f30..24457abc748 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
@@ -263,9 +263,7 @@ public class PropertyConverter {
s3Properties.put(Constants.MAX_ERROR_RETRIES, "2");
s3Properties.put("fs.s3.impl.disable.cache", "true");
s3Properties.putIfAbsent("fs.s3.impl", S3AFileSystem.class.getName());
- String defaultProviderList = String.join(",", S3Properties.AWS_CREDENTIALS_PROVIDERS);
- String credentialsProviders = properties
- .getOrDefault(S3Properties.CREDENTIALS_PROVIDER, defaultProviderList);
+ String credentialsProviders = getAWSCredentialsProviders(properties);
s3Properties.put(Constants.AWS_CREDENTIALS_PROVIDER, credentialsProviders);
if (credential.isWhole()) {
s3Properties.put(Constants.ACCESS_KEY, credential.getAccessKey());
@@ -285,6 +283,18 @@ public class PropertyConverter {
}
}
+ public static String getAWSCredentialsProviders(Map<String, String> properties) {
+ String credentialsProviders;
+ String hadoopCredProviders = properties.get(Constants.AWS_CREDENTIALS_PROVIDER);
+ if (hadoopCredProviders != null) {
+ credentialsProviders = hadoopCredProviders;
+ } else {
+ String defaultProviderList = String.join(",", S3Properties.AWS_CREDENTIALS_PROVIDERS);
+ credentialsProviders = properties.getOrDefault(S3Properties.CREDENTIALS_PROVIDER, defaultProviderList);
+ }
+ return credentialsProviders;
+ }
+
private static Map<String, String> convertToGCSProperties(Map<String, String> props, CloudCredential credential) {
// Now we use s3 client to access
return convertToS3Properties(S3Properties.prefixToS3(props), credential);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/S3Properties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/S3Properties.java
index c4d3cce9c2c..3297a4bd550 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/S3Properties.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/S3Properties.java
@@ -26,6 +26,10 @@ import org.apache.doris.datasource.property.PropertyConverter;
import org.apache.doris.thrift.TS3StorageParam;
import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
+import com.amazonaws.auth.InstanceProfileCredentialsProvider;
+import com.amazonaws.auth.SystemPropertiesCredentialsProvider;
+import com.amazonaws.auth.WebIdentityTokenCredentialsProvider;
+import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.google.common.base.Strings;
import com.google.common.collect.Maps;
import org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider;
@@ -66,6 +70,10 @@ public class S3Properties extends BaseProperties {
TemporaryAWSCredentialsProvider.class.getName(),
SimpleAWSCredentialsProvider.class.getName(),
EnvironmentVariableCredentialsProvider.class.getName(),
+ SystemPropertiesCredentialsProvider.class.getName(),
+ ProfileCredentialsProvider.class.getName(),
+ InstanceProfileCredentialsProvider.class.getName(),
+ WebIdentityTokenCredentialsProvider.class.getName(),
IAMInstanceCredentialsProvider.class.getName());
public static Map<String, String> credentialToMap(CloudCredentialWithEndpoint credential) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 13/38: [Fix](Nereids) fix missing comment when creating table (#31869)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5c1c73e5bb46f3779ce7ace088562c18de64e094
Author: zy-kkk <zh...@gmail.com>
AuthorDate: Wed Mar 6 20:46:32 2024 +0800
[Fix](Nereids) fix missing comment when creating table (#31869)
---
.../java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 03158295a7a..2ad86977c81 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -2414,6 +2414,8 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
if (ctx.INTEGER_VALUE() != null) {
bucketNum = Integer.parseInt(ctx.INTEGER_VALUE().getText());
}
+ String comment = ctx.STRING_LITERAL() == null ? "" : LogicalPlanBuilderAssistant.escapeBackSlash(
+ ctx.STRING_LITERAL().getText().substring(1, ctx.STRING_LITERAL().getText().length() - 1));
DistributionDescriptor desc = null;
if (ctx.HASH() != null) {
desc = new DistributionDescriptor(true, ctx.autoBucket != null, bucketNum,
@@ -2473,7 +2475,7 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
engineName,
keysType,
ctx.keys != null ? visitIdentifierList(ctx.keys) : ImmutableList.of(),
- "",
+ comment,
isAutoPartition,
autoPartitionExpr.build(),
partitionType,
@@ -2495,7 +2497,7 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
engineName,
keysType,
ctx.keys != null ? visitIdentifierList(ctx.keys) : ImmutableList.of(),
- "",
+ comment,
isAutoPartition,
autoPartitionExpr.build(),
partitionType,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 03/38: [fix](partition) add log when tablet partition id eq 0 (#31796)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 538032a75c615fb16e8d28a224da6abbd79af4ac
Author: deardeng <56...@qq.com>
AuthorDate: Wed Mar 6 13:49:54 2024 +0800
[fix](partition) add log when tablet partition id eq 0 (#31796)
---
be/src/olap/data_dir.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/olap/data_dir.cpp b/be/src/olap/data_dir.cpp
index 7cc57b9fee3..8713dc27428 100644
--- a/be/src/olap/data_dir.cpp
+++ b/be/src/olap/data_dir.cpp
@@ -513,8 +513,8 @@ Status DataDir::load() {
}
if (rowset_partition_id_eq_0_num > config::ignore_invalid_partition_id_rowset_num) {
LOG(FATAL) << fmt::format(
- "roswet partition id eq 0 bigger than config {}, be exit, plz check be.INFO",
- config::ignore_invalid_partition_id_rowset_num);
+ "roswet partition id eq 0 is {} bigger than config {}, be exit, plz check be.INFO",
+ rowset_partition_id_eq_0_num, config::ignore_invalid_partition_id_rowset_num);
exit(-1);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 26/38: [enhancement](nereids) show nullaware semi join in plan (#31738)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4d259377f31c32aa4c108f6174d26674a8b5ba1f
Author: starocean999 <40...@users.noreply.github.com>
AuthorDate: Thu Mar 7 10:56:54 2024 +0800
[enhancement](nereids) show nullaware semi join in plan (#31738)
---
.../glue/translator/PhysicalPlanTranslator.java | 14 +++++++++++++-
.../java/org/apache/doris/planner/HashJoinNode.java | 19 +++----------------
2 files changed, 16 insertions(+), 17 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 929bbdb23ca..81ff155f0d7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -28,6 +28,7 @@ import org.apache.doris.analysis.FunctionCallExpr;
import org.apache.doris.analysis.GroupByClause.GroupingType;
import org.apache.doris.analysis.GroupingInfo;
import org.apache.doris.analysis.IsNullPredicate;
+import org.apache.doris.analysis.JoinOperator;
import org.apache.doris.analysis.OrderByElement;
import org.apache.doris.analysis.OutFileClause;
import org.apache.doris.analysis.SlotDescriptor;
@@ -1227,6 +1228,7 @@ public class PhysicalPlanTranslator extends DefaultPlanVisitor<PlanFragment, Pla
List<Expr> markConjuncts = ImmutableList.of();
boolean isHashJoinConjunctsEmpty = hashJoin.getHashJoinConjuncts().isEmpty();
boolean isMarkJoinConjunctsEmpty = hashJoin.getMarkJoinConjuncts().isEmpty();
+ JoinOperator joinOperator = JoinType.toJoinOperator(joinType);
if (isHashJoinConjunctsEmpty) {
// if hash join conjuncts is empty, means mark join conjuncts must be EqualPredicate
// BE should use mark join conjuncts to build hash table
@@ -1236,10 +1238,20 @@ public class PhysicalPlanTranslator extends DefaultPlanVisitor<PlanFragment, Pla
.map(e -> JoinUtils.swapEqualToForChildrenOrder(e, hashJoin.left().getOutputSet()))
.map(e -> ExpressionTranslator.translate(e, context))
.collect(Collectors.toList());
+ // in order to process semi/anti join with no hash conjunct and having mark conjunct effeciently
+ // we can use mark conjunct as hash conjunct with slight different behavior if meets null values
+ // so we use null aware semi/anti join to indicate it's a null aware hash conjunct
+ // it's unnecessary to introduce new join type like NULL_AWARE_LEFT_SEMI_JOIN in nereids
+ // so we translate the join type here to let be known if the hash conjunct is null aware
+ if (joinOperator == JoinOperator.LEFT_ANTI_JOIN) {
+ joinOperator = JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN;
+ } else if (joinOperator == JoinOperator.LEFT_SEMI_JOIN) {
+ joinOperator = JoinOperator.NULL_AWARE_LEFT_SEMI_JOIN;
+ }
}
HashJoinNode hashJoinNode = new HashJoinNode(context.nextPlanNodeId(), leftPlanRoot,
- rightPlanRoot, JoinType.toJoinOperator(joinType), execEqConjuncts, Lists.newArrayList(), markConjuncts,
+ rightPlanRoot, joinOperator, execEqConjuncts, Lists.newArrayList(), markConjuncts,
null, null, null, hashJoin.isMarkJoin());
hashJoinNode.setNereidsId(hashJoin.getId());
hashJoinNode.setDistributeExprLists(distributeExprLists);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
index 0f099bdca33..1413d42ceee 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
@@ -782,11 +782,8 @@ public class HashJoinNode extends JoinNodeBase {
if (markJoinConjuncts != null) {
if (eqJoinConjuncts.isEmpty()) {
- Preconditions.checkState(joinOp == JoinOperator.LEFT_SEMI_JOIN
- || joinOp == JoinOperator.LEFT_ANTI_JOIN);
- if (joinOp == JoinOperator.LEFT_SEMI_JOIN || joinOp == JoinOperator.LEFT_ANTI_JOIN) {
- msg.hash_join_node.join_op = transformJoinOperator().toThrift();
- }
+ Preconditions.checkState(joinOp == JoinOperator.NULL_AWARE_LEFT_SEMI_JOIN
+ || joinOp == JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN);
// because eqJoinConjuncts mustn't be empty in thrift
// we have to use markJoinConjuncts instead
for (Expr e : markJoinConjuncts) {
@@ -969,19 +966,9 @@ public class HashJoinNode extends JoinNodeBase {
}
}
- private JoinOperator transformJoinOperator() {
- boolean transformToNullAware = markJoinConjuncts != null && eqJoinConjuncts.isEmpty();
- if (joinOp == JoinOperator.LEFT_ANTI_JOIN && transformToNullAware) {
- return JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN;
- } else if (joinOp == JoinOperator.LEFT_SEMI_JOIN && transformToNullAware) {
- return JoinOperator.NULL_AWARE_LEFT_SEMI_JOIN;
- }
- return joinOp;
- }
-
@Override
public boolean isNullAwareLeftAntiJoin() {
- if (transformJoinOperator() == JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN) {
+ if (joinOp == JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN) {
return true;
}
return children.stream().anyMatch(PlanNode::isNullAwareLeftAntiJoin);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 23/38: [Bug](fix) fix or and "<=>" cause coredump in query (#31884)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 326196b93dde8f0272aff660355606a5561e096d
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Thu Mar 7 10:09:35 2024 +0800
[Bug](fix) fix or and "<=>" cause coredump in query (#31884)
---
be/src/pipeline/exec/scan_operator.cpp | 2 +-
be/src/vec/exec/scan/vscan_node.cpp | 2 +-
regression-test/data/correctness_p0/test_equal_for_null.out | 2 ++
regression-test/suites/correctness_p0/test_equal_for_null.groovy | 3 ++-
4 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/be/src/pipeline/exec/scan_operator.cpp b/be/src/pipeline/exec/scan_operator.cpp
index 0fb92d793a8..2c809ffdfe3 100644
--- a/be/src/pipeline/exec/scan_operator.cpp
+++ b/be/src/pipeline/exec/scan_operator.cpp
@@ -1090,7 +1090,7 @@ Status ScanLocalState<Derived>::_normalize_binary_in_compound_predicate(
auto eq_checker = [](const std::string& fn_name) { return fn_name == "eq"; };
auto ne_checker = [](const std::string& fn_name) { return fn_name == "ne"; };
auto noneq_checker = [](const std::string& fn_name) {
- return fn_name != "ne" && fn_name != "eq";
+ return fn_name != "ne" && fn_name != "eq" && fn_name != "eq_for_null";
};
StringRef value;
diff --git a/be/src/vec/exec/scan/vscan_node.cpp b/be/src/vec/exec/scan/vscan_node.cpp
index 4ba8f924c00..4c0129b2bdf 100644
--- a/be/src/vec/exec/scan/vscan_node.cpp
+++ b/be/src/vec/exec/scan/vscan_node.cpp
@@ -1084,7 +1084,7 @@ Status VScanNode::_normalize_binary_in_compound_predicate(vectorized::VExpr* exp
auto eq_checker = [](const std::string& fn_name) { return fn_name == "eq"; };
auto ne_checker = [](const std::string& fn_name) { return fn_name == "ne"; };
auto noneq_checker = [](const std::string& fn_name) {
- return fn_name != "ne" && fn_name != "eq";
+ return fn_name != "ne" && fn_name != "eq" && fn_name != "eq_for_null";
};
StringRef value;
diff --git a/regression-test/data/correctness_p0/test_equal_for_null.out b/regression-test/data/correctness_p0/test_equal_for_null.out
index f958424e65c..357a6957798 100644
--- a/regression-test/data/correctness_p0/test_equal_for_null.out
+++ b/regression-test/data/correctness_p0/test_equal_for_null.out
@@ -1,3 +1,5 @@
-- This file is automatically generated. You should know what you did if you want to edit this
-- !select --
+-- !select --
+
diff --git a/regression-test/suites/correctness_p0/test_equal_for_null.groovy b/regression-test/suites/correctness_p0/test_equal_for_null.groovy
index d2f04e8bdde..51bef9deef1 100644
--- a/regression-test/suites/correctness_p0/test_equal_for_null.groovy
+++ b/regression-test/suites/correctness_p0/test_equal_for_null.groovy
@@ -47,4 +47,5 @@ suite("test_equal_for_null") {
"""
qt_select " select * from `test_equal_for_null_tbl` where `k1` <=> 4 order by 1, 2, 3, 4; "
-}
\ No newline at end of file
+ qt_select " select * from `test_equal_for_null_tbl` where `k1` <=> 4 or `k1` <=> 3 order by 1, 2, 3, 4; "
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 11/38: [Bug](load) fix wrong data in mv when routine load with function mapping (#31787)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit dc9de4b6b560260cca1190355b9f11b1683e40b4
Author: Pxl <px...@qq.com>
AuthorDate: Wed Mar 6 20:39:40 2024 +0800
[Bug](load) fix wrong data in mv when routine load with function mapping (#31787)
---
.../src/main/java/org/apache/doris/load/Load.java | 10 ++---
.../routine_load_mapping/routine_load_mapping.out | 9 ++++
.../data/mv_p0/routine_load_mapping/test | 1 +
.../routine_load_mapping.groovy | 52 ++++++++++++++++++++++
4 files changed, 67 insertions(+), 5 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/Load.java b/fe/fe-core/src/main/java/org/apache/doris/load/Load.java
index aaace9b37df..cbd3c21a30c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/Load.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/Load.java
@@ -568,14 +568,14 @@ public class Load {
List<SlotRef> slots = Lists.newArrayList();
entry.getValue().collect(SlotRef.class, slots);
for (SlotRef slot : slots) {
- if (slotDescByName.get(slot.getColumnName()) != null) {
- smap.getLhs().add(slot);
- smap.getRhs().add(
- getExprFromDesc(analyzer, getSlotFromDesc(slotDescByName.get(slot.getColumnName())), slot));
- } else if (exprsByName.get(slot.getColumnName()) != null) {
+ if (exprsByName.get(slot.getColumnName()) != null) {
smap.getLhs().add(slot);
smap.getRhs().add(new CastExpr(tbl.getColumn(slot.getColumnName()).getType(),
exprsByName.get(slot.getColumnName())));
+ } else if (slotDescByName.get(slot.getColumnName()) != null) {
+ smap.getLhs().add(slot);
+ smap.getRhs().add(
+ getExprFromDesc(analyzer, getSlotFromDesc(slotDescByName.get(slot.getColumnName())), slot));
} else {
if (entry.getKey().equalsIgnoreCase(Column.DELETE_SIGN)) {
throw new UserException("unknown reference column in DELETE ON clause:" + slot.getColumnName());
diff --git a/regression-test/data/mv_p0/routine_load_mapping/routine_load_mapping.out b/regression-test/data/mv_p0/routine_load_mapping/routine_load_mapping.out
new file mode 100644
index 00000000000..b70955720b3
--- /dev/null
+++ b/regression-test/data/mv_p0/routine_load_mapping/routine_load_mapping.out
@@ -0,0 +1,9 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !select --
+'ad_sdk_request' 2007-12-01T00:30:19 'b'
+ad_sdk_request 2024-03-04T00:00 a
+
+-- !select_mv --
+2007-12-01T00:30:19 1
+2024-03-04T00:00 1
+
diff --git a/regression-test/data/mv_p0/routine_load_mapping/test b/regression-test/data/mv_p0/routine_load_mapping/test
new file mode 100644
index 00000000000..98b8c2bab05
--- /dev/null
+++ b/regression-test/data/mv_p0/routine_load_mapping/test
@@ -0,0 +1 @@
+'ad_sdk_request',1196440219,'b'
\ No newline at end of file
diff --git a/regression-test/suites/mv_p0/routine_load_mapping/routine_load_mapping.groovy b/regression-test/suites/mv_p0/routine_load_mapping/routine_load_mapping.groovy
new file mode 100644
index 00000000000..1188e3031c0
--- /dev/null
+++ b/regression-test/suites/mv_p0/routine_load_mapping/routine_load_mapping.groovy
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite ("routine_load_mapping") {
+
+ sql """ DROP TABLE IF EXISTS test; """
+
+ sql """
+ CREATE TABLE `test` (
+ `event_id` varchar(50) NULL COMMENT '',
+ `time_stamp` datetime NULL COMMENT '',
+ `device_id` varchar(150) NULL DEFAULT "" COMMENT ''
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`event_id`)
+ DISTRIBUTED BY HASH(`device_id`) BUCKETS AUTO
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """insert into test(event_id,time_stamp,device_id) values('ad_sdk_request','2024-03-04 00:00:00','a');"""
+
+ createMV("""create materialized view m_view as select time_stamp, count(device_id) from test group by time_stamp;""")
+
+ streamLoad {
+ table "test"
+ set 'column_separator', ','
+ set 'columns', 'event_id,time_stamp,device_id,time_stamp=from_unixtime(`time_stamp`)'
+
+ file './test'
+ time 10000 // limit inflight 10s
+ }
+
+ qt_select "select * from test order by 1,2,3;"
+ qt_select_mv "select * from test index m_view order by 1,2;"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 32/38: [bug](meta) exit when get RollbackException in observer (#31687)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2f012b834d9482b77d26d84fb49f38551e4581dc
Author: Lijia Liu <li...@yeah.net>
AuthorDate: Thu Mar 7 12:18:21 2024 +0800
[bug](meta) exit when get RollbackException in observer (#31687)
---
.../src/main/java/org/apache/doris/journal/bdbje/BDBJEJournal.java | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBJEJournal.java b/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBJEJournal.java
index e00288eac55..13fa926c06d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBJEJournal.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBJEJournal.java
@@ -662,6 +662,13 @@ public class BDBJEJournal implements Journal { // CHECKSTYLE IGNORE THIS LINE: B
}
} catch (RollbackException rollbackEx) {
if (!Env.isCheckpointThread()) {
+ // Because Doris FE can not rollback its edit log, so it should restart and replay the new master's
+ // edit log.
+ if (rollbackEx.getEarliestTransactionId() != 0) {
+ LOG.error("Catch rollback log exception and it may have replayed outdated "
+ + "logs, so exec System.exit(-1).", rollbackEx);
+ System.exit(-1);
+ }
LOG.warn("catch rollback log exception. will reopen the ReplicatedEnvironment.", rollbackEx);
bdbEnvironment.close();
bdbEnvironment.openReplicatedEnvironment(new File(environmentPath));
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 25/38: [enhancement](nereids) allow reorder mark join (#30644)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c83f600c05fc83da7ee91c2aa58c23a315efd8b2
Author: starocean999 <40...@users.noreply.github.com>
AuthorDate: Thu Mar 7 10:51:50 2024 +0800
[enhancement](nereids) allow reorder mark join (#30644)
---
.../rules/analysis/CollectJoinConstraint.java | 2 +-
.../rules/exploration/join/InnerJoinLAsscom.java | 7 +--
.../exploration/join/InnerJoinLAsscomProject.java | 1 -
.../exploration/join/InnerJoinLeftAssociate.java | 1 -
.../join/InnerJoinLeftAssociateProject.java | 1 -
.../exploration/join/InnerJoinRightAssociate.java | 1 -
.../join/InnerJoinRightAssociateProject.java | 1 -
.../rules/exploration/join/JoinCommute.java | 7 ++-
.../rules/exploration/join/JoinExchange.java | 1 -
.../exploration/join/JoinExchangeBothProject.java | 1 -
.../exploration/join/JoinExchangeLeftProject.java | 1 -
.../exploration/join/JoinExchangeRightProject.java | 1 -
.../join/LogicalJoinSemiJoinTranspose.java | 8 +--
.../join/LogicalJoinSemiJoinTransposeProject.java | 7 +--
.../rules/exploration/join/OuterJoinAssoc.java | 1 -
.../exploration/join/OuterJoinAssocProject.java | 1 -
.../rules/exploration/join/OuterJoinLAsscom.java | 1 -
.../exploration/join/OuterJoinLAsscomProject.java | 1 -
.../join/PushDownProjectThroughInnerOuterJoin.java | 4 +-
.../join/SemiJoinSemiJoinTranspose.java | 1 -
.../join/SemiJoinSemiJoinTransposeProject.java | 11 +++-
.../rules/rewrite/PushFilterInsideJoin.java | 11 +++-
.../nereids/rules/rewrite/SemiJoinCommute.java | 1 -
.../rewrite/TransposeSemiJoinLogicalJoin.java | 1 -
.../TransposeSemiJoinLogicalJoinProject.java | 10 +--
.../nereids/trees/plans/logical/LogicalJoin.java | 14 ++---
.../trees/plans/physical/AbstractPhysicalJoin.java | 8 +--
.../plans/physical/PhysicalNestedLoopJoin.java | 7 +--
.../org/apache/doris/nereids/util/JoinUtils.java | 10 +++
.../shape/query10.out | 62 +++++++++----------
.../shape/query35.out | 72 +++++++++++-----------
.../shape/query45.out | 48 +++++++--------
.../noStatsRfPrune/query10.out | 40 ++++++------
.../noStatsRfPrune/query35.out | 42 ++++++-------
.../noStatsRfPrune/query45.out | 45 +++++++-------
.../no_stats_shape/query10.out | 40 ++++++------
.../no_stats_shape/query35.out | 42 ++++++-------
.../no_stats_shape/query45.out | 45 +++++++-------
.../rf_prune/query10.out | 54 ++++++++--------
.../rf_prune/query35.out | 70 ++++++++++-----------
.../rf_prune/query45.out | 48 +++++++--------
.../nereids_tpcds_shape_sf100_p0/shape/query10.out | 54 ++++++++--------
.../nereids_tpcds_shape_sf100_p0/shape/query35.out | 72 +++++++++++-----------
.../nereids_tpcds_shape_sf100_p0/shape/query45.out | 48 +++++++--------
44 files changed, 443 insertions(+), 461 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectJoinConstraint.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectJoinConstraint.java
index ea68f260685..3b7f22a29bc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectJoinConstraint.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectJoinConstraint.java
@@ -47,7 +47,7 @@ public class CollectJoinConstraint implements RewriteRuleFactory {
@Override
public List<Rule> buildRules() {
return ImmutableList.of(
- logicalJoin().whenNot(LogicalJoin::isMarkJoin).thenApply(ctx -> {
+ logicalJoin().thenApply(ctx -> {
if (!ctx.cascadesContext.isLeadingJoin()) {
return ctx.root;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java
index e0442738719..06d7d68ffc0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java
@@ -56,7 +56,6 @@ public class InnerJoinLAsscom extends OneExplorationRuleFactory {
return innerLogicalJoin(innerLogicalJoin(), group())
.when(topJoin -> checkReorder(topJoin, topJoin.left(), leftZigZag))
.whenNot(join -> join.hasDistributeHint() || join.left().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().isMarkJoin())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.left();
GroupPlan a = bottomJoin.left();
@@ -104,12 +103,10 @@ public class InnerJoinLAsscom extends OneExplorationRuleFactory {
double bRows = bottomJoin.right().getGroup().getStatistics().getRowCount();
double cRows = topJoin.right().getGroup().getStatistics().getRowCount();
return bRows < cRows && !bottomJoin.getJoinReorderContext().hasCommuteZigZag()
- && !topJoin.getJoinReorderContext().hasLAsscom()
- && (!bottomJoin.isMarkJoin() && !topJoin.isMarkJoin());
+ && !topJoin.getJoinReorderContext().hasLAsscom();
} else {
return !bottomJoin.getJoinReorderContext().hasCommuteZigZag()
- && !topJoin.getJoinReorderContext().hasLAsscom()
- && (!bottomJoin.isMarkJoin() && !topJoin.isMarkJoin());
+ && !topJoin.getJoinReorderContext().hasLAsscom();
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
index fc9f8f98a58..635c89d8995 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
@@ -62,7 +62,6 @@ public class InnerJoinLAsscomProject extends OneExplorationRuleFactory {
return innerLogicalJoin(logicalProject(innerLogicalJoin()), group())
.when(topJoin -> InnerJoinLAsscom.checkReorder(topJoin, topJoin.left().child(), enableLeftZigZag))
.whenNot(join -> join.hasDistributeHint() || join.left().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().child().isMarkJoin())
.when(join -> join.left().isAllSlots())
.then(topJoin -> {
/* ********** init ********** */
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLeftAssociate.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLeftAssociate.java
index a661db81bf6..624ddb842c8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLeftAssociate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLeftAssociate.java
@@ -51,7 +51,6 @@ public class InnerJoinLeftAssociate extends OneExplorationRuleFactory {
return innerLogicalJoin(group(), innerLogicalJoin())
.when(InnerJoinLeftAssociate::checkReorder)
.whenNot(join -> join.hasDistributeHint() || join.right().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.right().isMarkJoin())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.right();
GroupPlan a = topJoin.left();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLeftAssociateProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLeftAssociateProject.java
index 45363ea4266..8cec11b0ec0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLeftAssociateProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLeftAssociateProject.java
@@ -52,7 +52,6 @@ public class InnerJoinLeftAssociateProject extends OneExplorationRuleFactory {
return innerLogicalJoin(group(), logicalProject(innerLogicalJoin()))
.when(InnerJoinLeftAssociate::checkReorder)
.whenNot(join -> join.hasDistributeHint() || join.right().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.right().child().isMarkJoin())
.when(join -> join.right().isAllSlots())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.right().child();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinRightAssociate.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinRightAssociate.java
index 4fb21bbe37e..cf0aa4d0775 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinRightAssociate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinRightAssociate.java
@@ -49,7 +49,6 @@ public class InnerJoinRightAssociate extends OneExplorationRuleFactory {
return innerLogicalJoin(innerLogicalJoin(), group())
.when(InnerJoinRightAssociate::checkReorder)
.whenNot(join -> join.hasDistributeHint() || join.left().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().isMarkJoin())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.left();
GroupPlan a = bottomJoin.left();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinRightAssociateProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinRightAssociateProject.java
index 916078a0ad6..012fe6e9efc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinRightAssociateProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinRightAssociateProject.java
@@ -50,7 +50,6 @@ public class InnerJoinRightAssociateProject extends OneExplorationRuleFactory {
return innerLogicalJoin(logicalProject(innerLogicalJoin()), group())
.when(InnerJoinRightAssociate::checkReorder)
.whenNot(join -> join.hasDistributeHint() || join.left().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().child().isMarkJoin())
.when(join -> join.left().isAllSlots())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.left().child();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
index 5d402feec20..ef8868998e0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
@@ -27,6 +27,7 @@ import org.apache.doris.nereids.trees.expressions.functions.scalar.BitmapContain
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.JoinUtils;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.thrift.TRuntimeFilterType;
@@ -59,7 +60,11 @@ public class JoinCommute extends OneExplorationRuleFactory {
.when(join -> check(swapType, join))
.whenNot(LogicalJoin::hasDistributeHint)
.whenNot(join -> joinOrderMatchBitmapRuntimeFilterOrder(join))
- .whenNot(LogicalJoin::isMarkJoin)
+ // null aware mark join will be translated to null aware left semi/anti join
+ // we don't support null aware right semi/anti join, so should not commute
+ .whenNot(join -> JoinUtils.isNullAwareMarkJoin(join))
+ // commuting nest loop mark join is not supported by be
+ .whenNot(join -> join.isMarkJoin() && join.getHashJoinConjuncts().isEmpty())
.then(join -> {
LogicalJoin<Plan, Plan> newJoin = join.withTypeChildren(join.getJoinType().swap(),
join.right(), join.left(), null);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchange.java
index 4f37a28efd3..a660a84ee35 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchange.java
@@ -56,7 +56,6 @@ public class JoinExchange extends OneExplorationRuleFactory {
.when(JoinExchange::checkReorder)
.whenNot(join -> join.hasDistributeHint()
|| join.left().hasDistributeHint() || join.right().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().isMarkJoin() || join.right().isMarkJoin())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> leftJoin = topJoin.left();
LogicalJoin<GroupPlan, GroupPlan> rightJoin = topJoin.right();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeBothProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeBothProject.java
index 97e37c37165..9d5a6d8deca 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeBothProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeBothProject.java
@@ -59,7 +59,6 @@ public class JoinExchangeBothProject extends OneExplorationRuleFactory {
.when(join -> join.left().isAllSlots() && join.right().isAllSlots())
.whenNot(join -> join.hasDistributeHint()
|| join.left().child().hasDistributeHint() || join.right().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().child().isMarkJoin() || join.right().child().isMarkJoin())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> leftJoin = topJoin.left().child();
LogicalJoin<GroupPlan, GroupPlan> rightJoin = topJoin.right().child();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeLeftProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeLeftProject.java
index b8f88e08d64..d01339893c9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeLeftProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeLeftProject.java
@@ -59,7 +59,6 @@ public class JoinExchangeLeftProject extends OneExplorationRuleFactory {
.when(join -> join.left().isAllSlots())
.whenNot(join -> join.hasDistributeHint()
|| join.left().child().hasDistributeHint() || join.right().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().child().isMarkJoin() || join.right().isMarkJoin())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> leftJoin = topJoin.left().child();
LogicalJoin<GroupPlan, GroupPlan> rightJoin = topJoin.right();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeRightProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeRightProject.java
index dce4f75724c..76f3fb82f8d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeRightProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinExchangeRightProject.java
@@ -59,7 +59,6 @@ public class JoinExchangeRightProject extends OneExplorationRuleFactory {
.when(join -> join.right().isAllSlots())
.whenNot(join -> join.hasDistributeHint()
|| join.left().hasDistributeHint() || join.right().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().isMarkJoin() || join.right().child().isMarkJoin())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> leftJoin = topJoin.left();
LogicalJoin<GroupPlan, GroupPlan> rightJoin = topJoin.right().child();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/LogicalJoinSemiJoinTranspose.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/LogicalJoinSemiJoinTranspose.java
index 2504cd18d3b..7f0409e45e6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/LogicalJoinSemiJoinTranspose.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/LogicalJoinSemiJoinTranspose.java
@@ -43,9 +43,7 @@ public class LogicalJoinSemiJoinTranspose implements ExplorationRuleFactory {
.when(topJoin -> (topJoin.left().getJoinType().isLeftSemiOrAntiJoin()
&& (topJoin.getJoinType().isInnerJoin()
|| topJoin.getJoinType().isLeftOuterJoin())))
- .whenNot(topJoin -> topJoin.hasDistributeHint() || topJoin.left().hasDistributeHint()
- || topJoin.left().isMarkJoin())
- .whenNot(LogicalJoin::isMarkJoin)
+ .whenNot(topJoin -> topJoin.hasDistributeHint() || topJoin.left().hasDistributeHint())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.left();
GroupPlan a = bottomJoin.left();
@@ -60,9 +58,7 @@ public class LogicalJoinSemiJoinTranspose implements ExplorationRuleFactory {
.when(topJoin -> (topJoin.right().getJoinType().isLeftSemiOrAntiJoin()
&& (topJoin.getJoinType().isInnerJoin()
|| topJoin.getJoinType().isRightOuterJoin())))
- .whenNot(topJoin -> topJoin.hasDistributeHint() || topJoin.right().hasDistributeHint()
- || topJoin.right().isMarkJoin())
- .whenNot(LogicalJoin::isMarkJoin)
+ .whenNot(topJoin -> topJoin.hasDistributeHint() || topJoin.right().hasDistributeHint())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.right();
GroupPlan a = topJoin.left();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/LogicalJoinSemiJoinTransposeProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/LogicalJoinSemiJoinTransposeProject.java
index cf2889095b4..3b2c6e0dbc7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/LogicalJoinSemiJoinTransposeProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/LogicalJoinSemiJoinTransposeProject.java
@@ -44,9 +44,7 @@ public class LogicalJoinSemiJoinTransposeProject implements ExplorationRuleFacto
&& (topJoin.getJoinType().isInnerJoin()
|| topJoin.getJoinType().isLeftOuterJoin())))
.whenNot(topJoin -> topJoin.hasDistributeHint()
- || topJoin.left().child().hasDistributeHint()
- || topJoin.left().child().isMarkJoin())
- .whenNot(LogicalJoin::isMarkJoin)
+ || topJoin.left().child().hasDistributeHint())
.when(join -> join.left().isAllSlots())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.left().child();
@@ -66,8 +64,7 @@ public class LogicalJoinSemiJoinTransposeProject implements ExplorationRuleFacto
&& (topJoin.getJoinType().isInnerJoin()
|| topJoin.getJoinType().isRightOuterJoin())))
.whenNot(topJoin -> topJoin.hasDistributeHint()
- || topJoin.right().child().hasDistributeHint()
- || topJoin.right().child().isMarkJoin())
+ || topJoin.right().child().hasDistributeHint())
.when(join -> join.right().isAllSlots())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.right().child();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
index 5b7f430ca59..97e09598fa2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
@@ -59,7 +59,6 @@ public class OuterJoinAssoc extends OneExplorationRuleFactory {
.when(join -> VALID_TYPE_PAIR_SET.contains(Pair.of(join.left().getJoinType(), join.getJoinType())))
.when(topJoin -> OuterJoinLAsscom.checkReorder(topJoin, topJoin.left()))
.when(topJoin -> checkCondition(topJoin, topJoin.left().left().getOutputSet()))
- .whenNot(join -> join.isMarkJoin() || join.left().isMarkJoin())
.thenApply(ctx -> {
LogicalJoin<LogicalJoin<GroupPlan, GroupPlan>, GroupPlan> topJoin = ctx.root;
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.left();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
index 709cd129c80..77cfd806917 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
@@ -57,7 +57,6 @@ public class OuterJoinAssocProject extends OneExplorationRuleFactory {
Pair.of(join.left().child().getJoinType(), join.getJoinType())))
.when(topJoin -> OuterJoinLAsscom.checkReorder(topJoin, topJoin.left().child()))
.whenNot(join -> join.hasDistributeHint() || join.left().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().child().isMarkJoin())
.when(join -> OuterJoinAssoc.checkCondition(join, join.left().child().left().getOutputSet()))
.when(join -> join.left().isAllSlots())
.thenApply(ctx -> {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscom.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscom.java
index b2485171f59..40ef9b96229 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscom.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscom.java
@@ -64,7 +64,6 @@ public class OuterJoinLAsscom extends OneExplorationRuleFactory {
.when(topJoin -> checkReorder(topJoin, topJoin.left()))
.whenNot(join -> join.hasDistributeHint() || join.left().hasDistributeHint())
.when(topJoin -> checkCondition(topJoin, topJoin.left().right().getOutputExprIdSet()))
- .whenNot(LogicalJoin::isMarkJoin)
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.left();
GroupPlan a = bottomJoin.left();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscomProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscomProject.java
index 4e2833f2ede..bafe2d8fb4b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscomProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinLAsscomProject.java
@@ -54,7 +54,6 @@ public class OuterJoinLAsscomProject extends OneExplorationRuleFactory {
Pair.of(join.left().child().getJoinType(), join.getJoinType())))
.when(topJoin -> OuterJoinLAsscom.checkReorder(topJoin, topJoin.left().child()))
.whenNot(join -> join.hasDistributeHint() || join.left().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().child().isMarkJoin())
.when(topJoin -> OuterJoinLAsscom.checkCondition(topJoin,
topJoin.left().child().right().getOutputExprIdSet()))
.when(join -> join.left().isAllSlots())
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/PushDownProjectThroughInnerOuterJoin.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/PushDownProjectThroughInnerOuterJoin.java
index d920fdffc0e..85095223ac7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/PushDownProjectThroughInnerOuterJoin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/PushDownProjectThroughInnerOuterJoin.java
@@ -56,7 +56,7 @@ public class PushDownProjectThroughInnerOuterJoin implements ExplorationRuleFact
@Override
public List<Rule> buildRules() {
return ImmutableList.of(
- logicalJoin(logicalProject(logicalJoin().whenNot(LogicalJoin::isMarkJoin)), group())
+ logicalJoin(logicalProject(logicalJoin()), group())
.when(j -> j.left().child().getJoinType().isOuterJoin()
|| j.left().child().getJoinType().isInnerJoin())
// Just pushdown project with non-column expr like (t.id + 1)
@@ -70,7 +70,7 @@ public class PushDownProjectThroughInnerOuterJoin implements ExplorationRuleFact
}
return topJoin.withChildren(newLeft, topJoin.right());
}).toRule(RuleType.PUSH_DOWN_PROJECT_THROUGH_INNER_OUTER_JOIN_LEFT),
- logicalJoin(group(), logicalProject(logicalJoin().whenNot(LogicalJoin::isMarkJoin)))
+ logicalJoin(group(), logicalProject(logicalJoin()))
.when(j -> j.right().child().getJoinType().isOuterJoin()
|| j.right().child().getJoinType().isInnerJoin())
// Just pushdown project with non-column expr like (t.id + 1)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTranspose.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTranspose.java
index 9dbc6894571..e973cd7f5be 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTranspose.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTranspose.java
@@ -63,7 +63,6 @@ public class SemiJoinSemiJoinTranspose extends OneExplorationRuleFactory {
return logicalJoin(logicalJoin(), group())
.when(this::typeChecker)
.whenNot(join -> join.hasDistributeHint() || join.left().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().isMarkJoin())
.then(topJoin -> {
LogicalJoin<GroupPlan, GroupPlan> bottomJoin = topJoin.left();
GroupPlan a = bottomJoin.left();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTransposeProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTransposeProject.java
index f7cc68ef8ac..2a70ae157b5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTransposeProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTransposeProject.java
@@ -23,6 +23,7 @@ import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.rules.exploration.CBOUtils;
import org.apache.doris.nereids.rules.exploration.OneExplorationRuleFactory;
import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.MarkJoinSlotReference;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
@@ -31,8 +32,8 @@ import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
-import java.util.HashSet;
import java.util.Set;
+import java.util.stream.Collectors;
/**
* rule for semi-semi transpose
@@ -55,7 +56,6 @@ public class SemiJoinSemiJoinTransposeProject extends OneExplorationRuleFactory
.when(this::typeChecker)
.when(topSemi -> InnerJoinLAsscom.checkReorder(topSemi, topSemi.left().child(), false))
.whenNot(join -> join.hasDistributeHint() || join.left().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().child().isMarkJoin())
.when(join -> join.left().isAllSlots())
.then(topSemi -> {
LogicalJoin<GroupPlan, GroupPlan> bottomSemi = topSemi.left().child();
@@ -64,7 +64,9 @@ public class SemiJoinSemiJoinTransposeProject extends OneExplorationRuleFactory
GroupPlan b = bottomSemi.right();
GroupPlan c = topSemi.right();
Set<ExprId> aOutputExprIdSet = a.getOutputExprIdSet();
- Set<NamedExpression> acProjects = new HashSet<NamedExpression>(abProject.getProjects());
+ Set<NamedExpression> acProjects = (Set<NamedExpression>) abProject.getProjects()
+ .stream().filter(slot -> !(slot instanceof MarkJoinSlotReference))
+ .collect(Collectors.toSet());
bottomSemi.getConditionSlot()
.forEach(slot -> {
@@ -73,6 +75,9 @@ public class SemiJoinSemiJoinTransposeProject extends OneExplorationRuleFactory
}
});
LogicalJoin newBottomSemi = topSemi.withChildrenNoContext(a, c, null);
+ if (topSemi.isMarkJoin()) {
+ acProjects.add(topSemi.getMarkJoinSlotReference().get());
+ }
newBottomSemi.getJoinReorderContext().copyFrom(bottomSemi.getJoinReorderContext());
newBottomSemi.getJoinReorderContext().setHasCommute(false);
newBottomSemi.getJoinReorderContext().setHasLAsscom(false);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushFilterInsideJoin.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushFilterInsideJoin.java
index c9174e1fbdb..69d990c3bdb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushFilterInsideJoin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushFilterInsideJoin.java
@@ -21,12 +21,16 @@ import org.apache.doris.nereids.annotation.DependsRules;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.MarkJoinSlotReference;
+import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.ExpressionUtils;
import com.google.common.collect.Lists;
import java.util.List;
+import java.util.Set;
/**
* Push the predicate in the LogicalFilter to the join children.
@@ -39,13 +43,18 @@ public class PushFilterInsideJoin extends OneRewriteRuleFactory {
@Override
public Rule build() {
return logicalFilter(logicalJoin())
- .whenNot(filter -> filter.child().isMarkJoin())
// TODO: current just handle cross/inner join.
.when(filter -> filter.child().getJoinType().isCrossJoin()
|| filter.child().getJoinType().isInnerJoin())
.then(filter -> {
List<Expression> otherConditions = Lists.newArrayList(filter.getConjuncts());
LogicalJoin<Plan, Plan> join = filter.child();
+ Set<Slot> childOutput = join.getOutputSet();
+ if (ExpressionUtils.getInputSlotSet(otherConditions).stream()
+ .filter(MarkJoinSlotReference.class::isInstance)
+ .anyMatch(slot -> childOutput.contains(slot))) {
+ return null;
+ }
otherConditions.addAll(join.getOtherJoinConjuncts());
return new LogicalJoin<>(join.getJoinType(), join.getHashJoinConjuncts(),
otherConditions, join.getDistributeHint(), join.getMarkJoinSlotReference(),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SemiJoinCommute.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SemiJoinCommute.java
index 23d9844927e..3aa4925d58b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SemiJoinCommute.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SemiJoinCommute.java
@@ -35,7 +35,6 @@ public class SemiJoinCommute extends OneRewriteRuleFactory {
.whenNot(join -> ConnectContext.get().getSessionVariable().isDisableJoinReorder())
.whenNot(join -> join.isLeadingJoin())
.whenNot(LogicalJoin::hasDistributeHint)
- .whenNot(LogicalJoin::isMarkJoin)
.then(join -> join.withTypeChildren(join.getJoinType().swap(), join.right(), join.left(), null))
.toRule(RuleType.LOGICAL_SEMI_JOIN_COMMUTE);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoin.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoin.java
index 572d5ff4bf3..d0068f9faea 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoin.java
@@ -44,7 +44,6 @@ public class TransposeSemiJoinLogicalJoin extends OneRewriteRuleFactory {
|| topJoin.left().getJoinType().isLeftOuterJoin()
|| topJoin.left().getJoinType().isRightOuterJoin())))
.whenNot(topJoin -> topJoin.hasDistributeHint() || topJoin.left().hasDistributeHint())
- .whenNot(LogicalJoin::isMarkJoin)
.whenNot(topJoin -> topJoin.isLeadingJoin() || topJoin.left().isLeadingJoin())
.then(topSemiJoin -> {
LogicalJoin<Plan, Plan> bottomJoin = topSemiJoin.left();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoinProject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoinProject.java
index f8c03d2c59e..25e1879acc0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoinProject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoinProject.java
@@ -20,6 +20,7 @@ package org.apache.doris.nereids.rules.rewrite;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.Plan;
@@ -29,6 +30,7 @@ import org.apache.doris.nereids.util.Utils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
import java.util.Set;
@@ -49,7 +51,6 @@ public class TransposeSemiJoinLogicalJoinProject extends OneRewriteRuleFactory {
|| topJoin.left().child().getJoinType().isRightOuterJoin())))
.when(join -> join.left().isAllSlots())
.whenNot(join -> join.hasDistributeHint() || join.left().child().hasDistributeHint())
- .whenNot(join -> join.isMarkJoin() || join.left().child().isMarkJoin())
.whenNot(topJoin -> topJoin.isLeadingJoin() || topJoin.left().child().isLeadingJoin())
.when(join -> join.left().getProjects().stream().allMatch(expr -> expr instanceof Slot))
.then(topSemiJoin -> {
@@ -65,7 +66,8 @@ public class TransposeSemiJoinLogicalJoinProject extends OneRewriteRuleFactory {
if (containsType == ContainsType.ALL) {
return null;
}
-
+ ImmutableList<NamedExpression> topProjects = topSemiJoin.getOutput().stream()
+ .map(slot -> (NamedExpression) slot).collect(ImmutableList.toImmutableList());
if (containsType == ContainsType.LEFT) {
/*-
* topSemiJoin project
@@ -85,7 +87,7 @@ public class TransposeSemiJoinLogicalJoinProject extends OneRewriteRuleFactory {
Plan newBottomSemiJoin = topSemiJoin.withChildren(a, c);
Plan newTopJoin = bottomJoin.withChildren(newBottomSemiJoin, b);
- return project.withChildren(newTopJoin);
+ return project.withProjectsAndChild(topProjects, newTopJoin);
} else {
/*-
* topSemiJoin project
@@ -105,7 +107,7 @@ public class TransposeSemiJoinLogicalJoinProject extends OneRewriteRuleFactory {
Plan newBottomSemiJoin = topSemiJoin.withChildren(b, c);
Plan newTopJoin = bottomJoin.withChildren(a, newBottomSemiJoin);
- return project.withChildren(newTopJoin);
+ return project.withProjectsAndChild(topProjects, newTopJoin);
}
}).toRule(RuleType.TRANSPOSE_LOGICAL_SEMI_JOIN_LOGICAL_JOIN_PROJECT);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java
index ba34a13c2e0..c8db68081ad 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java
@@ -185,11 +185,8 @@ public class LogicalJoin<LEFT_CHILD_TYPE extends Plan, RIGHT_CHILD_TYPE extends
* getConditionSlot
*/
public Set<Slot> getConditionSlot() {
- // this function is called by rules which reject mark join
- // so markJoinConjuncts is not processed here
- Preconditions.checkState(!isMarkJoin(),
- "shouldn't call mark join's getConditionSlot method");
- return Stream.concat(hashJoinConjuncts.stream(), otherJoinConjuncts.stream())
+ return Stream.concat(Stream.concat(hashJoinConjuncts.stream(), otherJoinConjuncts.stream()),
+ markJoinConjuncts.stream())
.flatMap(expr -> expr.getInputSlots().stream())
.collect(ImmutableSet.toImmutableSet());
}
@@ -198,11 +195,8 @@ public class LogicalJoin<LEFT_CHILD_TYPE extends Plan, RIGHT_CHILD_TYPE extends
* getConditionExprId
*/
public Set<ExprId> getConditionExprId() {
- // this function is called by rules which reject mark join
- // so markJoinConjuncts is not processed here
- Preconditions.checkState(!isMarkJoin(),
- "shouldn't call mark join's getConditionExprId method");
- return Stream.concat(getHashJoinConjuncts().stream(), getOtherJoinConjuncts().stream())
+ return Stream.concat(Stream.concat(hashJoinConjuncts.stream(), otherJoinConjuncts.stream()),
+ markJoinConjuncts.stream())
.flatMap(expr -> expr.getInputSlotExprIds().stream()).collect(Collectors.toSet());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalJoin.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalJoin.java
index e2847970370..87ac8ba0798 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalJoin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalJoin.java
@@ -36,7 +36,6 @@ import org.apache.doris.nereids.util.JoinUtils;
import org.apache.doris.nereids.util.Utils;
import org.apache.doris.statistics.Statistics;
-import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableList.Builder;
import com.google.common.collect.ImmutableSet;
@@ -260,11 +259,8 @@ public abstract class AbstractPhysicalJoin<
* getConditionSlot
*/
public Set<Slot> getConditionSlot() {
- // this function is called by rules which reject mark join
- // so markJoinConjuncts is not processed here
- Preconditions.checkState(!isMarkJoin(),
- "shouldn't call mark join's getConditionSlot method");
- return Stream.concat(hashJoinConjuncts.stream(), otherJoinConjuncts.stream())
+ return Stream.concat(Stream.concat(hashJoinConjuncts.stream(), otherJoinConjuncts.stream()),
+ markJoinConjuncts.stream())
.flatMap(expr -> expr.getInputSlots().stream()).collect(ImmutableSet.toImmutableSet());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalNestedLoopJoin.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalNestedLoopJoin.java
index 1f4e2bb1712..b8e5c17e034 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalNestedLoopJoin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalNestedLoopJoin.java
@@ -210,11 +210,8 @@ public class PhysicalNestedLoopJoin<
* getConditionSlot
*/
public Set<Slot> getConditionSlot() {
- // this function is called by rules which reject mark join
- // so markJoinConjuncts is not processed here
- Preconditions.checkState(!isMarkJoin(),
- "shouldn't call mark join's getConditionSlot method");
- return Stream.concat(hashJoinConjuncts.stream(), otherJoinConjuncts.stream())
+ return Stream.concat(Stream.concat(hashJoinConjuncts.stream(), otherJoinConjuncts.stream()),
+ markJoinConjuncts.stream())
.flatMap(expr -> expr.getInputSlots().stream())
.collect(ImmutableSet.toImmutableSet());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java
index 27125504791..3c439f8b99f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java
@@ -382,4 +382,14 @@ public class JoinUtils {
.build();
}
}
+
+ public static boolean hasMarkConjuncts(Join join) {
+ return !join.getMarkJoinConjuncts().isEmpty();
+ }
+
+ public static boolean isNullAwareMarkJoin(Join join) {
+ // if mark join's hash conjuncts is empty, we use mark conjuncts as hash conjuncts
+ // and translate join type to NULL_AWARE_LEFT_SEMI_JOIN or NULL_AWARE_LEFT_ANTI_JOIN
+ return join.getHashJoinConjuncts().isEmpty() && !join.getMarkJoinConjuncts().isEmpty();
+ }
}
diff --git a/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query10.out b/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query10.out
index 83741371afc..0ee3636703c 100644
--- a/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query10.out
+++ b/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query10.out
@@ -10,48 +10,48 @@ PhysicalResultSink
--------------hashAgg[LOCAL]
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
---------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF5 c_current_cdemo_sk->[cd_demo_sk]
-------------------------------PhysicalOlapScan[customer_demographics] apply RFs: RF5
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF4 c_customer_sk->[ss_customer_sk]
+--------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF5 d_date_sk->[cs_sold_date_sk]
+----------------------------PhysicalProject
+------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF5
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------PhysicalProject
+--------------------------------filter((date_dim.d_moy <= 6) and (date_dim.d_moy >= 3) and (date_dim.d_year = 2001))
+----------------------------------PhysicalOlapScan[date_dim]
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF4 c_current_cdemo_sk->[cd_demo_sk]
+----------------------------PhysicalOlapScan[customer_demographics] apply RFs: RF4
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------PhysicalProject
+--------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF3 c_customer_sk->[ss_customer_sk]
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF2 d_date_sk->[ss_sold_date_sk]
+----------------------------------------PhysicalProject
+------------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF2 RF3
+----------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------------------PhysicalProject
+--------------------------------------------filter((date_dim.d_moy <= 6) and (date_dim.d_moy >= 3) and (date_dim.d_year = 2001))
+----------------------------------------------PhysicalOlapScan[date_dim]
+----------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
------------------------------------PhysicalDistribute[DistributionSpecHash]
--------------------------------------PhysicalProject
-----------------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[ss_sold_date_sk]
+----------------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
------------------------------------------PhysicalProject
---------------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF3 RF4
+--------------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
--------------------------------------------PhysicalProject
----------------------------------------------filter((date_dim.d_moy <= 6) and (date_dim.d_moy >= 3) and (date_dim.d_year = 2001))
------------------------------------------------PhysicalOlapScan[date_dim]
------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF2 ca_address_sk->[c_current_addr_sk]
+--------------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF0 ca_address_sk->[c_current_addr_sk]
----------------------------------------PhysicalProject
-------------------------------------------PhysicalOlapScan[customer] apply RFs: RF2
+------------------------------------------PhysicalOlapScan[customer] apply RFs: RF0
----------------------------------------PhysicalDistribute[DistributionSpecReplicated]
------------------------------------------PhysicalProject
--------------------------------------------filter(ca_county IN ('Campbell County', 'Cleburne County', 'Escambia County', 'Fairfield County', 'Washtenaw County'))
----------------------------------------------PhysicalOlapScan[customer_address]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
-------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_moy <= 6) and (date_dim.d_moy >= 3) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
-----------------------PhysicalDistribute[DistributionSpecHash]
-------------------------PhysicalProject
---------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
-----------------------------PhysicalProject
-------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF0
-----------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------PhysicalProject
---------------------------------filter((date_dim.d_moy <= 6) and (date_dim.d_moy >= 3) and (date_dim.d_year = 2001))
-----------------------------------PhysicalOlapScan[date_dim]
diff --git a/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query35.out b/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query35.out
index a72d3f8673e..de0cd9c7852 100644
--- a/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query35.out
+++ b/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query35.out
@@ -10,50 +10,50 @@ PhysicalResultSink
--------------hashAgg[LOCAL]
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
---------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF5 cd_demo_sk->[c_current_cdemo_sk]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF4 ca_address_sk->[c_current_addr_sk]
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF3 c_customer_sk->[ss_customer_sk]
+--------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[cs_sold_date_sk]
+----------------------------PhysicalProject
+------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF3
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------PhysicalProject
+--------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 1999))
+----------------------------------PhysicalOlapScan[date_dim]
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=()
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------PhysicalProject
+--------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=()
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF2 c_customer_sk->[ss_customer_sk]
+--------------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------------PhysicalProject
+------------------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ss_sold_date_sk]
+--------------------------------------------PhysicalProject
+----------------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF1 RF2
+--------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------------------PhysicalProject
+------------------------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 1999))
+--------------------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
----------------------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------------------PhysicalProject
---------------------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF2 d_date_sk->[ss_sold_date_sk]
+--------------------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[ws_sold_date_sk]
----------------------------------------------PhysicalProject
-------------------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF2 RF3
+------------------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF0
----------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
------------------------------------------------PhysicalProject
--------------------------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 1999))
----------------------------------------------------PhysicalOlapScan[date_dim]
----------------------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------------------PhysicalProject
---------------------------------------------PhysicalOlapScan[customer] apply RFs: RF4 RF5
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[customer_address]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_demographics]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+--------------------------------------------PhysicalOlapScan[customer]
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------PhysicalOlapScan[customer_address]
+----------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 1999))
-------------------------------------PhysicalOlapScan[date_dim]
-----------------------PhysicalDistribute[DistributionSpecHash]
-------------------------PhysicalProject
---------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
-----------------------------PhysicalProject
-------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF0
-----------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------PhysicalProject
---------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 1999))
-----------------------------------PhysicalOlapScan[date_dim]
+--------------------------------PhysicalOlapScan[customer_demographics]
diff --git a/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query45.out b/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query45.out
index 939588dcdf9..cb8a7ac5923 100644
--- a/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query45.out
+++ b/regression-test/data/nereids_tpcds_shape_sf1000_p0/shape/query45.out
@@ -9,34 +9,30 @@ PhysicalResultSink
------------hashAgg[LOCAL]
--------------PhysicalProject
----------------filter((substring(ca_zip, 1, 5) IN ('80348', '81792', '83405', '85392', '85460', '85669', '86197', '86475', '88274') OR $c$1))
-------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF3 c_current_addr_sk->[ca_address_sk]
--------------------PhysicalProject
-----------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF3 i_item_sk->[ws_item_sk]
-------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalOlapScan[customer_address] apply RFs: RF3
+--------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalProject
+------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=() build RFs:RF2 ws_bill_customer_sk->[c_customer_sk]
--------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=() build RFs:RF2 c_customer_sk->[ws_bill_customer_sk]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
-----------------------------------PhysicalProject
-------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1 RF2 RF3
-----------------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------------PhysicalProject
---------------------------------------filter((date_dim.d_qoy = 1) and (date_dim.d_year = 2000))
-----------------------------------------PhysicalOlapScan[date_dim]
-------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------PhysicalOlapScan[customer] apply RFs: RF2
+--------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF0 i_item_sk->[ws_item_sk]
--------------------------------PhysicalProject
-----------------------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF0 ca_address_sk->[c_current_addr_sk]
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[customer] apply RFs: RF0
-------------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------PhysicalOlapScan[web_sales] apply RFs: RF0 RF1
+--------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------PhysicalProject
+------------------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
--------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[customer_address]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------PhysicalOlapScan[item]
---------------------PhysicalDistribute[DistributionSpecReplicated]
-----------------------PhysicalProject
-------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
---------------------------PhysicalOlapScan[item]
+----------------------------------------PhysicalOlapScan[item]
+--------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------------PhysicalProject
+------------------------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
+--------------------------------------------PhysicalOlapScan[item]
+------------------------------PhysicalDistribute[DistributionSpecReplicated]
+--------------------------------PhysicalProject
+----------------------------------filter((date_dim.d_qoy = 1) and (date_dim.d_year = 2000))
+------------------------------------PhysicalOlapScan[date_dim]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query10.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query10.out
index 845705681d5..f95b47f437d 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query10.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query10.out
@@ -11,12 +11,12 @@ PhysicalResultSink
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
--------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF5 ca_address_sk->[c_current_addr_sk]
-------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=()
---------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF5 ca_address_sk->[c_current_addr_sk]
+----------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=()
+------------------------------PhysicalDistribute[DistributionSpecHash]
+--------------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
----------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF3 c_customer_sk->[ss_customer_sk]
------------------------------------PhysicalDistribute[DistributionSpecHash]
--------------------------------------PhysicalProject
@@ -30,21 +30,21 @@ PhysicalResultSink
------------------------------------PhysicalDistribute[DistributionSpecHash]
--------------------------------------PhysicalProject
----------------------------------------PhysicalOlapScan[customer] apply RFs: RF5
---------------------------------PhysicalDistribute[DistributionSpecHash]
-----------------------------------PhysicalOlapScan[customer_demographics]
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter(ca_county IN ('Cochran County', 'Kandiyohi County', 'Marquette County', 'Storey County', 'Warren County'))
-------------------------------------PhysicalOlapScan[customer_address]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+----------------------------------------PhysicalProject
+------------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
+----------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------------------PhysicalProject
+--------------------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
+----------------------------------------------PhysicalOlapScan[date_dim]
+------------------------------PhysicalDistribute[DistributionSpecHash]
+--------------------------------PhysicalOlapScan[customer_demographics]
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------filter(ca_county IN ('Cochran County', 'Kandiyohi County', 'Marquette County', 'Storey County', 'Warren County'))
+----------------------------------PhysicalOlapScan[customer_address]
----------------------PhysicalDistribute[DistributionSpecHash]
------------------------PhysicalProject
--------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query35.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query35.out
index 97a42e09303..c6ccf6941a4 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query35.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query35.out
@@ -11,13 +11,13 @@ PhysicalResultSink
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
--------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=()
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=()
-----------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=()
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=()
+--------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=()
--------------------------------------PhysicalDistribute[DistributionSpecHash]
----------------------------------------PhysicalProject
@@ -31,21 +31,21 @@ PhysicalResultSink
--------------------------------------PhysicalDistribute[DistributionSpecHash]
----------------------------------------PhysicalProject
------------------------------------------PhysicalOlapScan[customer]
-----------------------------------PhysicalDistribute[DistributionSpecHash]
-------------------------------------PhysicalProject
---------------------------------------PhysicalOlapScan[customer_demographics]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_address]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+------------------------------------PhysicalDistribute[DistributionSpecHash]
+--------------------------------------PhysicalProject
+----------------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+------------------------------------------PhysicalProject
+--------------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
+------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+--------------------------------------------PhysicalProject
+----------------------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
+------------------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------PhysicalProject
+------------------------------------PhysicalOlapScan[customer_demographics]
+----------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------PhysicalOlapScan[customer_address]
----------------------PhysicalDistribute[DistributionSpecHash]
------------------------PhysicalProject
--------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query45.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query45.out
index 1e0fa30e89c..8cb112d5c99 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query45.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query45.out
@@ -9,33 +9,34 @@ PhysicalResultSink
------------hashAgg[LOCAL]
--------------PhysicalProject
----------------filter((substring(ca_zip, 1, 5) IN ('80348', '81792', '83405', '85392', '85460', '85669', '86197', '86475', '88274') OR $c$1))
-------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF3 i_item_sk->[ws_item_sk]
--------------------PhysicalProject
-----------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[ws_sold_date_sk]
-------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=()
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=()
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=()
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF3
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[customer]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_address]
---------------------------PhysicalDistribute[DistributionSpecReplicated]
-----------------------------PhysicalProject
-------------------------------PhysicalOlapScan[item]
+----------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF2 d_date_sk->[ws_sold_date_sk]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=()
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------PhysicalProject
+--------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=()
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF2 RF3
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------PhysicalOlapScan[customer]
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------PhysicalProject
+--------------------------------PhysicalOlapScan[customer_address]
------------------------PhysicalDistribute[DistributionSpecReplicated]
--------------------------PhysicalProject
----------------------------filter((date_dim.d_qoy = 2) and (date_dim.d_year = 2000))
------------------------------PhysicalOlapScan[date_dim]
--------------------PhysicalDistribute[DistributionSpecReplicated]
----------------------PhysicalProject
-------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
---------------------------PhysicalOlapScan[item]
+------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+--------------------------PhysicalProject
+----------------------------PhysicalOlapScan[item]
+--------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------PhysicalProject
+------------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
+--------------------------------PhysicalOlapScan[item]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query10.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query10.out
index 558e86d696e..b261aa1193b 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query10.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query10.out
@@ -11,12 +11,12 @@ PhysicalResultSink
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
--------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF5 ca_address_sk->[c_current_addr_sk]
-------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF4 cd_demo_sk->[c_current_cdemo_sk]
---------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF5 ca_address_sk->[c_current_addr_sk]
+----------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF4 cd_demo_sk->[c_current_cdemo_sk]
+------------------------------PhysicalDistribute[DistributionSpecHash]
+--------------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
----------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF3 c_customer_sk->[ss_customer_sk]
------------------------------------PhysicalDistribute[DistributionSpecHash]
--------------------------------------PhysicalProject
@@ -30,21 +30,21 @@ PhysicalResultSink
------------------------------------PhysicalDistribute[DistributionSpecHash]
--------------------------------------PhysicalProject
----------------------------------------PhysicalOlapScan[customer] apply RFs: RF4 RF5
---------------------------------PhysicalDistribute[DistributionSpecHash]
-----------------------------------PhysicalOlapScan[customer_demographics]
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter(ca_county IN ('Cochran County', 'Kandiyohi County', 'Marquette County', 'Storey County', 'Warren County'))
-------------------------------------PhysicalOlapScan[customer_address]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+----------------------------------------PhysicalProject
+------------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
+----------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------------------PhysicalProject
+--------------------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
+----------------------------------------------PhysicalOlapScan[date_dim]
+------------------------------PhysicalDistribute[DistributionSpecHash]
+--------------------------------PhysicalOlapScan[customer_demographics]
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------filter(ca_county IN ('Cochran County', 'Kandiyohi County', 'Marquette County', 'Storey County', 'Warren County'))
+----------------------------------PhysicalOlapScan[customer_address]
----------------------PhysicalDistribute[DistributionSpecHash]
------------------------PhysicalProject
--------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query35.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query35.out
index cc92a34b9e9..242014a9bf3 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query35.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query35.out
@@ -11,13 +11,13 @@ PhysicalResultSink
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
--------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF5 ca_address_sk->[c_current_addr_sk]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF4 cd_demo_sk->[c_current_cdemo_sk]
-----------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF5 ca_address_sk->[c_current_addr_sk]
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF4 cd_demo_sk->[c_current_cdemo_sk]
+--------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF3 c_customer_sk->[ss_customer_sk]
--------------------------------------PhysicalDistribute[DistributionSpecHash]
----------------------------------------PhysicalProject
@@ -31,21 +31,21 @@ PhysicalResultSink
--------------------------------------PhysicalDistribute[DistributionSpecHash]
----------------------------------------PhysicalProject
------------------------------------------PhysicalOlapScan[customer] apply RFs: RF4 RF5
-----------------------------------PhysicalDistribute[DistributionSpecHash]
-------------------------------------PhysicalProject
---------------------------------------PhysicalOlapScan[customer_demographics]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_address]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+------------------------------------PhysicalDistribute[DistributionSpecHash]
+--------------------------------------PhysicalProject
+----------------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+------------------------------------------PhysicalProject
+--------------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
+------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+--------------------------------------------PhysicalProject
+----------------------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
+------------------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------PhysicalProject
+------------------------------------PhysicalOlapScan[customer_demographics]
+----------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------PhysicalOlapScan[customer_address]
----------------------PhysicalDistribute[DistributionSpecHash]
------------------------PhysicalProject
--------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query45.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query45.out
index 3617dbf4b5a..2c2c1ff2157 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query45.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query45.out
@@ -9,33 +9,34 @@ PhysicalResultSink
------------hashAgg[LOCAL]
--------------PhysicalProject
----------------filter((substring(ca_zip, 1, 5) IN ('80348', '81792', '83405', '85392', '85460', '85669', '86197', '86475', '88274') OR $c$1))
-------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF3 i_item_sk->[ws_item_sk]
--------------------PhysicalProject
-----------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[ws_sold_date_sk]
-------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF2 i_item_sk->[ws_item_sk]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF1 ca_address_sk->[c_current_addr_sk]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=() build RFs:RF0 c_customer_sk->[ws_bill_customer_sk]
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF0 RF2 RF3
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[customer] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_address]
---------------------------PhysicalDistribute[DistributionSpecReplicated]
-----------------------------PhysicalProject
-------------------------------PhysicalOlapScan[item]
+----------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF2 d_date_sk->[ws_sold_date_sk]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF1 ca_address_sk->[c_current_addr_sk]
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------PhysicalProject
+--------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=() build RFs:RF0 c_customer_sk->[ws_bill_customer_sk]
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF0 RF2 RF3
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------PhysicalOlapScan[customer] apply RFs: RF1
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------PhysicalProject
+--------------------------------PhysicalOlapScan[customer_address]
------------------------PhysicalDistribute[DistributionSpecReplicated]
--------------------------PhysicalProject
----------------------------filter((date_dim.d_qoy = 2) and (date_dim.d_year = 2000))
------------------------------PhysicalOlapScan[date_dim]
--------------------PhysicalDistribute[DistributionSpecReplicated]
----------------------PhysicalProject
-------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
---------------------------PhysicalOlapScan[item]
+------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+--------------------------PhysicalProject
+----------------------------PhysicalOlapScan[item]
+--------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------PhysicalProject
+------------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
+--------------------------------PhysicalOlapScan[item]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query10.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query10.out
index 923159c5f63..68e345b2c85 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query10.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query10.out
@@ -10,46 +10,46 @@ PhysicalResultSink
--------------hashAgg[LOCAL]
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
---------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
+--------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
+----------------------PhysicalDistribute[DistributionSpecHash]
------------------------PhysicalProject
---------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF5 c_customer_sk->[ss_customer_sk]
+--------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF5 d_date_sk->[ws_sold_date_sk]
+----------------------------PhysicalProject
+------------------------------PhysicalOlapScan[web_sales] apply RFs: RF5
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------PhysicalProject
+--------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
+----------------------------------PhysicalOlapScan[date_dim]
+----------------------PhysicalProject
+------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF4 c_customer_sk->[ss_customer_sk]
+--------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------PhysicalProject
+------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[ss_sold_date_sk]
+--------------------------------PhysicalProject
+----------------------------------PhysicalOlapScan[store_sales] apply RFs: RF3 RF4
+--------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------PhysicalProject
+------------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
+--------------------------------------PhysicalOlapScan[date_dim]
+--------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
----------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------PhysicalProject
---------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF4 d_date_sk->[ss_sold_date_sk]
+--------------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF2 d_date_sk->[cs_sold_date_sk]
----------------------------------PhysicalProject
-------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF4 RF5
+------------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF2
----------------------------------PhysicalDistribute[DistributionSpecReplicated]
------------------------------------PhysicalProject
--------------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
----------------------------------------PhysicalOlapScan[date_dim]
----------------------------PhysicalDistribute[DistributionSpecHash]
-------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF3 c_current_cdemo_sk->[cd_demo_sk]
---------------------------------PhysicalOlapScan[customer_demographics] apply RFs: RF3
+------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF1 c_current_cdemo_sk->[cd_demo_sk]
+--------------------------------PhysicalOlapScan[customer_demographics] apply RFs: RF1
--------------------------------PhysicalDistribute[DistributionSpecReplicated]
-----------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF2 ca_address_sk->[c_current_addr_sk]
+----------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF0 ca_address_sk->[c_current_addr_sk]
------------------------------------PhysicalProject
---------------------------------------PhysicalOlapScan[customer] apply RFs: RF2
+--------------------------------------PhysicalOlapScan[customer] apply RFs: RF0
------------------------------------PhysicalDistribute[DistributionSpecReplicated]
--------------------------------------PhysicalProject
----------------------------------------filter(ca_county IN ('Cochran County', 'Kandiyohi County', 'Marquette County', 'Storey County', 'Warren County'))
------------------------------------------PhysicalOlapScan[customer_address]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
-------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
-----------------------PhysicalDistribute[DistributionSpecHash]
-------------------------PhysicalProject
---------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
-----------------------------PhysicalProject
-------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF0
-----------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------PhysicalProject
---------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
-----------------------------------PhysicalOlapScan[date_dim]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query35.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query35.out
index 1ea14729a92..fa1fb12ebd0 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query35.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query35.out
@@ -10,21 +10,39 @@ PhysicalResultSink
--------------hashAgg[LOCAL]
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
---------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=()
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=()
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=()
+--------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[cs_sold_date_sk]
+----------------------------PhysicalProject
+------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF3
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------PhysicalProject
+--------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
+----------------------------------PhysicalOlapScan[date_dim]
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=()
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------PhysicalProject
+--------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=()
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF2 c_customer_sk->[ss_customer_sk]
+--------------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------------PhysicalProject
+------------------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ss_sold_date_sk]
+--------------------------------------------PhysicalProject
+----------------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF1 RF2
+--------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------------------PhysicalProject
+------------------------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
+--------------------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
----------------------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------------------PhysicalProject
---------------------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF2 d_date_sk->[ss_sold_date_sk]
+--------------------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[ws_sold_date_sk]
----------------------------------------------PhysicalProject
-------------------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF2
+------------------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF0
----------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
------------------------------------------------PhysicalProject
--------------------------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
@@ -32,28 +50,10 @@ PhysicalResultSink
----------------------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------------------PhysicalProject
--------------------------------------------PhysicalOlapScan[customer]
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[customer_address]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_demographics]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------PhysicalOlapScan[customer_address]
+----------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
-----------------------PhysicalDistribute[DistributionSpecHash]
-------------------------PhysicalProject
---------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
-----------------------------PhysicalProject
-------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF0
-----------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------PhysicalProject
---------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
-----------------------------------PhysicalOlapScan[date_dim]
+--------------------------------PhysicalOlapScan[customer_demographics]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query45.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query45.out
index c73e08c0fc6..b3b09807b2d 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query45.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/rf_prune/query45.out
@@ -9,34 +9,30 @@ PhysicalResultSink
------------hashAgg[LOCAL]
--------------PhysicalProject
----------------filter((substring(ca_zip, 1, 5) IN ('80348', '81792', '83405', '85392', '85460', '85669', '86197', '86475', '88274') OR $c$1))
-------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF3 c_current_addr_sk->[ca_address_sk]
--------------------PhysicalProject
-----------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=()
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
-----------------------------------PhysicalProject
-------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-----------------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------------PhysicalProject
---------------------------------------filter((date_dim.d_qoy = 2) and (date_dim.d_year = 2000))
-----------------------------------------PhysicalOlapScan[date_dim]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[item]
-------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalOlapScan[customer_address] apply RFs: RF3
+--------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalProject
+------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=() build RFs:RF2 ws_bill_customer_sk->[c_customer_sk]
--------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=()
-------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------PhysicalOlapScan[customer] apply RFs: RF2
+--------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF0 i_item_sk->[ws_item_sk]
--------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer]
-------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------PhysicalOlapScan[web_sales] apply RFs: RF0 RF1
+--------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------PhysicalProject
+------------------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+--------------------------------------PhysicalProject
+----------------------------------------PhysicalOlapScan[item]
+--------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------------PhysicalProject
+------------------------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
+--------------------------------------------PhysicalOlapScan[item]
+------------------------------PhysicalDistribute[DistributionSpecReplicated]
--------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_address]
---------------------PhysicalDistribute[DistributionSpecReplicated]
-----------------------PhysicalProject
-------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
---------------------------PhysicalOlapScan[item]
+----------------------------------filter((date_dim.d_qoy = 2) and (date_dim.d_year = 2000))
+------------------------------------PhysicalOlapScan[date_dim]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query10.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query10.out
index 923159c5f63..68e345b2c85 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query10.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query10.out
@@ -10,46 +10,46 @@ PhysicalResultSink
--------------hashAgg[LOCAL]
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
---------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
+--------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
+----------------------PhysicalDistribute[DistributionSpecHash]
------------------------PhysicalProject
---------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF5 c_customer_sk->[ss_customer_sk]
+--------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF5 d_date_sk->[ws_sold_date_sk]
+----------------------------PhysicalProject
+------------------------------PhysicalOlapScan[web_sales] apply RFs: RF5
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------PhysicalProject
+--------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
+----------------------------------PhysicalOlapScan[date_dim]
+----------------------PhysicalProject
+------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF4 c_customer_sk->[ss_customer_sk]
+--------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------PhysicalProject
+------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[ss_sold_date_sk]
+--------------------------------PhysicalProject
+----------------------------------PhysicalOlapScan[store_sales] apply RFs: RF3 RF4
+--------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------PhysicalProject
+------------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
+--------------------------------------PhysicalOlapScan[date_dim]
+--------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
----------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------PhysicalProject
---------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF4 d_date_sk->[ss_sold_date_sk]
+--------------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF2 d_date_sk->[cs_sold_date_sk]
----------------------------------PhysicalProject
-------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF4 RF5
+------------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF2
----------------------------------PhysicalDistribute[DistributionSpecReplicated]
------------------------------------PhysicalProject
--------------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
----------------------------------------PhysicalOlapScan[date_dim]
----------------------------PhysicalDistribute[DistributionSpecHash]
-------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF3 c_current_cdemo_sk->[cd_demo_sk]
---------------------------------PhysicalOlapScan[customer_demographics] apply RFs: RF3
+------------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF1 c_current_cdemo_sk->[cd_demo_sk]
+--------------------------------PhysicalOlapScan[customer_demographics] apply RFs: RF1
--------------------------------PhysicalDistribute[DistributionSpecReplicated]
-----------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF2 ca_address_sk->[c_current_addr_sk]
+----------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF0 ca_address_sk->[c_current_addr_sk]
------------------------------------PhysicalProject
---------------------------------------PhysicalOlapScan[customer] apply RFs: RF2
+--------------------------------------PhysicalOlapScan[customer] apply RFs: RF0
------------------------------------PhysicalDistribute[DistributionSpecReplicated]
--------------------------------------PhysicalProject
----------------------------------------filter(ca_county IN ('Cochran County', 'Kandiyohi County', 'Marquette County', 'Storey County', 'Warren County'))
------------------------------------------PhysicalOlapScan[customer_address]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
-------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
-----------------------PhysicalDistribute[DistributionSpecHash]
-------------------------PhysicalProject
---------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
-----------------------------PhysicalProject
-------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF0
-----------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------PhysicalProject
---------------------------------filter((date_dim.d_moy <= 4) and (date_dim.d_moy >= 1) and (date_dim.d_year = 2001))
-----------------------------------PhysicalOlapScan[date_dim]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query35.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query35.out
index 0874f2dd35c..fa1fb12ebd0 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query35.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query35.out
@@ -10,50 +10,50 @@ PhysicalResultSink
--------------hashAgg[LOCAL]
----------------PhysicalProject
------------------filter((ifnull($c$1, FALSE) OR ifnull($c$2, FALSE)))
---------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
-----------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=() build RFs:RF5 cd_demo_sk->[c_current_cdemo_sk]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=() build RFs:RF4 ca_address_sk->[c_current_addr_sk]
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF3 c_customer_sk->[ss_customer_sk]
+--------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = catalog_sales.cs_ship_customer_sk)) otherCondition=()
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[cs_sold_date_sk]
+----------------------------PhysicalProject
+------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF3
+----------------------------PhysicalDistribute[DistributionSpecReplicated]
+------------------------------PhysicalProject
+--------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
+----------------------------------PhysicalOlapScan[date_dim]
+----------------------PhysicalDistribute[DistributionSpecHash]
+------------------------PhysicalProject
+--------------------------hashJoin[INNER_JOIN] hashCondition=((customer_demographics.cd_demo_sk = c.c_current_cdemo_sk)) otherCondition=()
+----------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------PhysicalProject
+--------------------------------hashJoin[INNER_JOIN] hashCondition=((c.c_current_addr_sk = ca.ca_address_sk)) otherCondition=()
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = store_sales.ss_customer_sk)) otherCondition=() build RFs:RF2 c_customer_sk->[ss_customer_sk]
+--------------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------------PhysicalProject
+------------------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ss_sold_date_sk]
+--------------------------------------------PhysicalProject
+----------------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF1 RF2
+--------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------------------PhysicalProject
+------------------------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
+--------------------------------------------------PhysicalOlapScan[date_dim]
+--------------------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((c.c_customer_sk = web_sales.ws_bill_customer_sk)) otherCondition=()
----------------------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------------------PhysicalProject
---------------------------------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF2 d_date_sk->[ss_sold_date_sk]
+--------------------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[ws_sold_date_sk]
----------------------------------------------PhysicalProject
-------------------------------------------------PhysicalOlapScan[store_sales] apply RFs: RF2 RF3
+------------------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF0
----------------------------------------------PhysicalDistribute[DistributionSpecReplicated]
------------------------------------------------PhysicalProject
--------------------------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
----------------------------------------------------PhysicalOlapScan[date_dim]
----------------------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------------------PhysicalProject
---------------------------------------------PhysicalOlapScan[customer] apply RFs: RF4 RF5
-------------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------PhysicalProject
-----------------------------------------PhysicalOlapScan[customer_address]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_demographics]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+--------------------------------------------PhysicalOlapScan[customer]
+----------------------------------PhysicalDistribute[DistributionSpecHash]
+------------------------------------PhysicalProject
+--------------------------------------PhysicalOlapScan[customer_address]
+----------------------------PhysicalDistribute[DistributionSpecHash]
------------------------------PhysicalProject
---------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1
-------------------------------PhysicalDistribute[DistributionSpecReplicated]
---------------------------------PhysicalProject
-----------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
-------------------------------------PhysicalOlapScan[date_dim]
-----------------------PhysicalDistribute[DistributionSpecHash]
-------------------------PhysicalProject
---------------------------hashJoin[INNER_JOIN] hashCondition=((catalog_sales.cs_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF0 d_date_sk->[cs_sold_date_sk]
-----------------------------PhysicalProject
-------------------------------PhysicalOlapScan[catalog_sales] apply RFs: RF0
-----------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------PhysicalProject
---------------------------------filter((date_dim.d_qoy < 4) and (date_dim.d_year = 2001))
-----------------------------------PhysicalOlapScan[date_dim]
+--------------------------------PhysicalOlapScan[customer_demographics]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query45.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query45.out
index e8d2715c4d6..b3b09807b2d 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query45.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query45.out
@@ -9,34 +9,30 @@ PhysicalResultSink
------------hashAgg[LOCAL]
--------------PhysicalProject
----------------filter((substring(ca_zip, 1, 5) IN ('80348', '81792', '83405', '85392', '85460', '85669', '86197', '86475', '88274') OR $c$1))
-------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF3 c_current_addr_sk->[ca_address_sk]
--------------------PhysicalProject
-----------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=() build RFs:RF3 c_customer_sk->[ws_bill_customer_sk]
-------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF2 i_item_sk->[ws_item_sk]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
-----------------------------------PhysicalProject
-------------------------------------PhysicalOlapScan[web_sales] apply RFs: RF1 RF2 RF3
-----------------------------------PhysicalDistribute[DistributionSpecReplicated]
-------------------------------------PhysicalProject
---------------------------------------filter((date_dim.d_qoy = 2) and (date_dim.d_year = 2000))
-----------------------------------------PhysicalOlapScan[date_dim]
-------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[item]
-------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalOlapScan[customer_address] apply RFs: RF3
+--------------------PhysicalDistribute[DistributionSpecHash]
+----------------------PhysicalProject
+------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_bill_customer_sk = customer.c_customer_sk)) otherCondition=() build RFs:RF2 ws_bill_customer_sk->[c_customer_sk]
--------------------------PhysicalProject
-----------------------------hashJoin[INNER_JOIN] hashCondition=((customer.c_current_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF0 ca_address_sk->[c_current_addr_sk]
-------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------PhysicalOlapScan[customer] apply RFs: RF2
+--------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_sold_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF1 d_date_sk->[ws_sold_date_sk]
+------------------------------hashJoin[INNER_JOIN] hashCondition=((web_sales.ws_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF0 i_item_sk->[ws_item_sk]
--------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer] apply RFs: RF0
-------------------------------PhysicalDistribute[DistributionSpecHash]
+----------------------------------PhysicalOlapScan[web_sales] apply RFs: RF0 RF1
+--------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------PhysicalProject
+------------------------------------hashJoin[LEFT_SEMI_JOIN] hashCondition=((item.i_item_id = item.i_item_id)) otherCondition=()
+--------------------------------------PhysicalProject
+----------------------------------------PhysicalOlapScan[item]
+--------------------------------------PhysicalDistribute[DistributionSpecReplicated]
+----------------------------------------PhysicalProject
+------------------------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
+--------------------------------------------PhysicalOlapScan[item]
+------------------------------PhysicalDistribute[DistributionSpecReplicated]
--------------------------------PhysicalProject
-----------------------------------PhysicalOlapScan[customer_address]
---------------------PhysicalDistribute[DistributionSpecReplicated]
-----------------------PhysicalProject
-------------------------filter(i_item_sk IN (11, 13, 17, 19, 2, 23, 29, 3, 5, 7))
---------------------------PhysicalOlapScan[item]
+----------------------------------filter((date_dim.d_qoy = 2) and (date_dim.d_year = 2000))
+------------------------------------PhysicalOlapScan[date_dim]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 15/38: [regression test] Test the unique model by modify a value type from TINYINT to other type (#31682)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1d9e9fc88439538ee27a960962873d31a7aaa09d
Author: kkop <45...@users.noreply.github.com>
AuthorDate: Wed Mar 6 20:55:40 2024 +0800
[regression test] Test the unique model by modify a value type from TINYINT to other type (#31682)
---
.../test_unique_model_schema_value_change.out | 80 +++++++++++
.../test_unique_model_schema_value_change.groovy | 155 +++++++++++++++++++++
2 files changed, 235 insertions(+)
diff --git a/regression-test/data/schema_change_p0/test_unique_model_schema_value_change.out b/regression-test/data/schema_change_p0/test_unique_model_schema_value_change.out
index 7a089233c2f..c8d4de184ea 100644
--- a/regression-test/data/schema_change_p0/test_unique_model_schema_value_change.out
+++ b/regression-test/data/schema_change_p0/test_unique_model_schema_value_change.out
@@ -139,3 +139,83 @@
789012345 Grace \N Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
123456689 Alice {"k1":"v31","k2":300} Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+-- ! --
+123456789 Alice 1 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 1 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 1 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 1 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 1 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1.0 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1.0 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1.0 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1.0 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0.0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0.0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1.0 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 1.2 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1.0 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1.0 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1.0 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1.0 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0.0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0.0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1.0 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice 1.23 Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice asd Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
+-- ! --
+123456789 Alice 1 Beijing 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+234567890 Bob 1 Shanghai 30 1 13998765432 No. 456 Street, Shanghai 2022-02-02T12:00
+345678901 Carol 1 Guangzhou 28 0 13724681357 No. 789 Street, Guangzhou 2022-03-03T14:00
+456789012 Dave 1 Shenzhen 35 1 13680864279 No. 987 Street, Shenzhen 2022-04-04T16:00
+567890123 Eve 0 Chengdu 27 0 13572468091 No. 654 Street, Chengdu 2022-05-05T18:00
+678901234 Frank 0 Hangzhou 32 1 13467985213 No. 321 Street, Hangzhou 2022-06-06T20:00
+789012345 Grace 1 Xian 29 0 13333333333 No. 222 Street, Xian 2022-07-07T22:00
+123456689 Alice asd Yaan 25 0 13812345678 No. 123 Street, Beijing 2022-01-01T10:00
+
diff --git a/regression-test/suites/schema_change_p0/test_unique_model_schema_value_change.groovy b/regression-test/suites/schema_change_p0/test_unique_model_schema_value_change.groovy
index a8ef1265ab0..c573624b26f 100644
--- a/regression-test/suites/schema_change_p0/test_unique_model_schema_value_change.groovy
+++ b/regression-test/suites/schema_change_p0/test_unique_model_schema_value_change.groovy
@@ -411,4 +411,159 @@ suite("test_unique_model_schema_value_change","p0") {
sql getTableStatusSql
time 60
}, insertSql, true,"${tbName}")*/
+
+
+
+ /**
+ * Test the unique model by modify a value type from TINYINT to other type
+ */
+
+ initTable = " CREATE TABLE IF NOT EXISTS ${tbName}\n" +
+ " (\n" +
+ " `user_id` LARGEINT NOT NULL COMMENT \"用户id\",\n" +
+ " `username` VARCHAR(50) NOT NULL COMMENT \"用户昵称\",\n" +
+ " `is_student` TINYINT COMMENT \"是否是学生\",\n" +
+ " `city` VARCHAR(20) COMMENT \"用户所在城市\",\n" +
+ " `age` SMALLINT COMMENT \"用户年龄\",\n" +
+ " `sex` TINYINT COMMENT \"用户性别\",\n" +
+ " `phone` LARGEINT COMMENT \"用户电话\",\n" +
+ " `address` VARCHAR(500) COMMENT \"用户地址\",\n" +
+ " `register_time` DATETIME COMMENT \"用户注册时间\"\n" +
+ " )\n" +
+ " UNIQUE KEY(`user_id`, `username`)\n" +
+ " DISTRIBUTED BY HASH(`user_id`) BUCKETS 1\n" +
+ " PROPERTIES (\n" +
+ " \"replication_allocation\" = \"tag.location.default: 1\",\n" +
+ " \"enable_unique_key_merge_on_write\" = \"true\"\n" +
+ " );"
+
+ initTableData = "insert into ${tbName} values(123456789, 'Alice', 1, 'Beijing', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00')," +
+ " (234567890, 'Bob', 1, 'Shanghai', 30, 1, 13998765432, 'No. 456 Street, Shanghai', '2022-02-02 12:00:00')," +
+ " (345678901, 'Carol', 1, 'Guangzhou', 28, 0, 13724681357, 'No. 789 Street, Guangzhou', '2022-03-03 14:00:00')," +
+ " (456789012, 'Dave', 1, 'Shenzhen', 35, 1, 13680864279, 'No. 987 Street, Shenzhen', '2022-04-04 16:00:00')," +
+ " (567890123, 'Eve', 0, 'Chengdu', 27, 0, 13572468091, 'No. 654 Street, Chengdu', '2022-05-05 18:00:00')," +
+ " (678901234, 'Frank', 0, 'Hangzhou', 32, 1, 13467985213, 'No. 321 Street, Hangzhou', '2022-06-06 20:00:00')," +
+ " (789012345, 'Grace', 1, 'Xian', 29, 0, 13333333333, 'No. 222 Street, Xian', '2022-07-07 22:00:00');"
+
+ //Test the unique model by modify a value type from TINYINT to BOOLEAN
+ //TODO java.sql.SQLException: errCode = 2, detailMessage = Can not change TINYINT to BOOLEAN
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student BOOLEAN DEFAULT "true" """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', false, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+
+
+ //Test the unique model by modify a value type from TINYINT to SMALLINT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student SMALLINT """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 1, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+ //Test the unique model by modify a value type from TINYINT to INT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student INT """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 1, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+ //Test the unique model by modify a value type from TINYINT to BIGINT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student BIGINT """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 1, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+ //Test the unique model by modify a value type from TINYINT to LARGEINT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student LARGEINT """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 1, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+ //Test the unique model by modify a value type from TINYINT to FLOAT
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student FLOAT """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 1.2, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+ //Test the unique model by modify a value type from TINYINT to DOUBLE
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student DOUBLE """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 1.23, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+ //TODO Test the unique model by modify a value type from TINYINT to DECIMAL32
+ //java.sql.SQLException: errCode = 2, detailMessage = Can not change TINYINT to DECIMAL32
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student DECIMAL32 """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 1.23, 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+ //TODO Test the unique model by modify a value type from TINYINT to CHAR
+ //java.sql.SQLException: errCode = 2, detailMessage = Can not change TINYINT to CHAR
+/* sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student CHAR(15) """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 'asd', 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")*/
+
+
+
+ //Test the unique model by modify a value type from TINYINT to VARCHAR
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student VARCHAR(100) """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 'asd', 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
+
+
+ //Test the unique model by modify a value type from TINYINT to STRING
+ sql initTable
+ sql initTableData
+ sql """ alter table ${tbName} MODIFY column is_student STRING """
+ insertSql = "insert into ${tbName} values(123456689, 'Alice', 'asd', 'Yaan', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01 10:00:00'); "
+ waitForSchemaChangeDone({
+ sql getTableStatusSql
+ time 60
+ }, insertSql, true,"${tbName}")
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 06/38: [docs](flink-connector) Fix Flink Doris Connector typo (#31830)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f897db666c2aaa047e228f10cf530261e8203ace
Author: KassieZ <13...@users.noreply.github.com>
AuthorDate: Wed Mar 6 14:26:18 2024 +0800
[docs](flink-connector) Fix Flink Doris Connector typo (#31830)
---
docs/zh-CN/docs/ecosystem/flink-doris-connector.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md
index 5c51da5f0a7..578c81c8176 100644
--- a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md
+++ b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md
@@ -59,7 +59,7 @@ under the License.
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>flink-doris-connector-1.16</artifactId>
- <version>1.4.0</version>
+ <version>1.5.2</version>
</dependency>
```
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 21/38: [improvement](jdbc catalog) opt get db2 schema list & xml type mapping (#31856)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b3db6b2eae1cc5721b7e686d77af3bae404a02c8
Author: zy-kkk <zh...@gmail.com>
AuthorDate: Wed Mar 6 22:42:15 2024 +0800
[improvement](jdbc catalog) opt get db2 schema list & xml type mapping (#31856)
1. Trim Schema Names: Adapted the system to remove trailing spaces from DB2 schema names, ensuring compatibility without affecting query operations.
2. XML Mapping: Implemented a feature to directly map XML types to String.
---
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 1 +
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 1 +
.../datasource/jdbc/client/JdbcDB2Client.java | 36 ++++++++++++++++++++++
.../datasource/jdbc/client/JdbcOracleClient.java | 6 ++--
.../jdbc/test_db2_jdbc_catalog.out | 20 ++++++++++++
.../jdbc/test_db2_jdbc_catalog.groovy | 24 +++++++++++++++
6 files changed, 85 insertions(+), 3 deletions(-)
diff --git a/docs/en/docs/lakehouse/multi-catalog/jdbc.md b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
index 64f598aec19..e0585cd13e1 100644
--- a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
@@ -890,6 +890,7 @@ When mapping DB2, Doris's Database corresponds to a Schema under the specified D
| LONG VARGRAPHIC | STRING | |
| TIME | STRING | |
| CLOB | STRING | |
+| XML | STRING | |
| OTHER | UNSUPPORTED | |
## JDBC Drivers
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
index ec54699227c..ae28cc6ed59 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
@@ -894,6 +894,7 @@ CREATE CATALOG `jdbc_db2` PROPERTIES (
| LONG VARGRAPHIC | STRING | |
| TIME | STRING | |
| CLOB | STRING | |
+| XML | STRING | |
| OTHER | UNSUPPORTED | |
## JDBC Driver 列表
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java
index 5cdcbf1ca4f..e53ff9165ec 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java
@@ -21,17 +21,52 @@ import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
+import com.google.common.collect.Lists;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.List;
+
public class JdbcDB2Client extends JdbcClient {
protected JdbcDB2Client(JdbcClientConfig jdbcClientConfig) {
super(jdbcClientConfig);
}
+ @Override
+ public List<String> getDatabaseNameList() {
+ Connection conn = getConnection();
+ Statement stmt = null;
+ ResultSet rs = null;
+ if (isOnlySpecifiedDatabase && includeDatabaseMap.isEmpty() && excludeDatabaseMap.isEmpty()) {
+ return getSpecifiedDatabase(conn);
+ }
+ List<String> remoteDatabaseNames = Lists.newArrayList();
+ try {
+ rs = conn.getMetaData().getSchemas(conn.getCatalog(), null);
+ while (rs.next()) {
+ remoteDatabaseNames.add(rs.getString("TABLE_SCHEM").trim());
+ }
+ } catch (SQLException e) {
+ throw new JdbcClientException("failed to get database name list from jdbc", e);
+ } finally {
+ close(rs, stmt, conn);
+ }
+ return filterDatabaseNames(remoteDatabaseNames);
+ }
+
@Override
protected String getDatabaseQuery() {
return "SELECT schemaname FROM syscat.schemata WHERE DEFINER = CURRENT USER;";
}
+ @Override
+ protected String getCatalogName(Connection conn) throws SQLException {
+ return conn.getCatalog();
+ }
+
@Override
protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) {
String db2Type = fieldSchema.getDataTypeName();
@@ -75,6 +110,7 @@ public class JdbcDB2Client extends JdbcClient {
case "CLOB":
case "VARGRAPHIC":
case "LONG VARGRAPHIC":
+ case "XML":
return ScalarType.createStringType();
default:
return Type.UNSUPPORTED;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java
index 514e7b81638..4d536a4ef3f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java
@@ -52,18 +52,18 @@ public class JdbcOracleClient extends JdbcClient {
if (isOnlySpecifiedDatabase && includeDatabaseMap.isEmpty() && excludeDatabaseMap.isEmpty()) {
return getSpecifiedDatabase(conn);
}
- List<String> databaseNames = Lists.newArrayList();
+ List<String> remoteDatabaseNames = Lists.newArrayList();
try {
rs = conn.getMetaData().getSchemas(conn.getCatalog(), null);
while (rs.next()) {
- databaseNames.add(rs.getString("TABLE_SCHEM"));
+ remoteDatabaseNames.add(rs.getString("TABLE_SCHEM"));
}
} catch (SQLException e) {
throw new JdbcClientException("failed to get database name list from jdbc", e);
} finally {
close(rs, conn);
}
- return filterDatabaseNames(databaseNames);
+ return filterDatabaseNames(remoteDatabaseNames);
}
@Override
diff --git a/regression-test/data/external_table_p0/jdbc/test_db2_jdbc_catalog.out b/regression-test/data/external_table_p0/jdbc/test_db2_jdbc_catalog.out
index b8dbcf1bfad..50cfbe42102 100644
--- a/regression-test/data/external_table_p0/jdbc/test_db2_jdbc_catalog.out
+++ b/regression-test/data/external_table_p0/jdbc/test_db2_jdbc_catalog.out
@@ -41,3 +41,23 @@ VARCHAR_COLUMN VARCHAR(255) Yes true \N
123 1234567890.1234567890 1 12345.6789 12345.679 1.2345678901234567E9 1.2345678901234567E9 123 12345 123456789012345 中文一 中文二 中文三 中文四 中文五 中文六 2024-01-24 2024-01-24T12:34:56.789 12:34:56 中文七
123 1234567890.1234567890 1 12345.6789 12345.679 1.2345678901234567E9 1.2345678901234567E9 123 12345 123456789012345 中文一 中文二 中文三 中文四 中文五 中文六 2024-01-24 2024-01-24T12:34:56.789 12:34:56 中文七
+-- !desc_db --
+DORIS_TEST
+NULLID
+SQLJ
+SYSCAT
+SYSFUN
+SYSIBM
+SYSIBMADM
+SYSIBMINTERNAL
+SYSIBMTS
+SYSPROC
+SYSPUBLIC
+SYSSTAT
+SYSTOOLS
+TEST
+information_schema
+
+-- !select_xml --
+1000 <catalog><book><author> Gambardella Matthew</author><title>XML Developers Guide</title><genre>Computer</genre><price>44.95</price><publish_date>2000-10-01</publish_date><description>An in-depth look at creating application\n with XML</description></book></catalog>
+
diff --git a/regression-test/suites/external_table_p0/jdbc/test_db2_jdbc_catalog.groovy b/regression-test/suites/external_table_p0/jdbc/test_db2_jdbc_catalog.groovy
index 68d3066c471..ddd812c7955 100644
--- a/regression-test/suites/external_table_p0/jdbc/test_db2_jdbc_catalog.groovy
+++ b/regression-test/suites/external_table_p0/jdbc/test_db2_jdbc_catalog.groovy
@@ -37,6 +37,7 @@ suite("test_db2_jdbc_catalog", "p0,external,db2,external_docker,external_docker_
try {
db2_docker "CREATE SCHEMA doris_test;"
+ db2_docker "CREATE SCHEMA test;"
db2_docker """CREATE TABLE doris_test.sample_table (
id_column INT GENERATED ALWAYS AS IDENTITY,
numeric_column NUMERIC,
@@ -239,12 +240,35 @@ suite("test_db2_jdbc_catalog", "p0,external,db2,external_docker,external_docker_
select * except(ID_COLUMN) from ${sample_table};
"""
+ db2_docker """create table test.books(id bigint not null
+ primary key, book XML);"""
+
+ db2_docker """insert into test.books values(1000, '<catalog>
+ <book>
+
+ <author> Gambardella Matthew</author>
+ <title>XML Developers Guide</title>
+ <genre>Computer</genre>
+ <price>44.95</price>
+ <publish_date>2000-10-01</publish_date>
+ <description>An in-depth look at creating application
+ with XML</description>
+ </book>
+
+ </catalog>');"""
+
order_qt_sample_table_insert """ select * except(ID_COLUMN) from ${sample_table} order by 1; """
+ order_qt_desc_db "show databases from ${catalog_name};"
+
+ order_qt_select_xml "select * from db2.TEST.BOOKS;"
+
sql """ drop catalog if exists ${catalog_name} """
db2_docker "DROP TABLE IF EXISTS doris_test.sample_table;"
db2_docker "DROP SCHEMA doris_test restrict;"
+ db2_docker "DROP TABLE IF EXISTS test.books;"
+ db2_docker "DROP SCHEMA test restrict;"
} catch (Exception e) {
e.printStackTrace()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 18/38: [fix](Nereids) fix group_concat(distinct) failed (#31873)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 561709451c5aa14e441261ef305448bcd9e090be
Author: 924060929 <92...@qq.com>
AuthorDate: Wed Mar 6 21:54:35 2024 +0800
[fix](Nereids) fix group_concat(distinct) failed (#31873)
---
.../rules/implementation/AggregateStrategies.java | 33 ++++++++--------
.../functions/agg/AggregateFunction.java | 3 ++
.../expressions/functions/agg/GroupConcat.java | 9 +++++
.../nereids/trees/plans/algebra/Aggregate.java | 2 +-
.../data/nereids_syntax_p0/group_concat.out | 8 ++++
.../suites/nereids_syntax_p0/group_concat.groovy | 45 +++++++++++++++++++++-
6 files changed, 81 insertions(+), 19 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java
index 254e014240b..10b21d0b979 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java
@@ -1108,7 +1108,7 @@ public class AggregateStrategies implements ImplementationRuleFactory {
* <p>
* single node aggregate:
* <p>
- * PhysicalHashAggregate(groupBy=[name], output=[name, count(distinct(id)], mode=BUFFER_TO_RESULT)
+ * PhysicalHashAggregate(groupBy=[name], output=[name, count(distinct(id))], mode=BUFFER_TO_RESULT)
* |
* PhysicalHashAggregate(groupBy=[name, id], output=[name, id], mode=INPUT_TO_BUFFER)
* |
@@ -1118,12 +1118,10 @@ public class AggregateStrategies implements ImplementationRuleFactory {
* <p>
* distribute node aggregate:
* <p>
- * PhysicalHashAggregate(groupBy=[name], output=[name, count(distinct(id)], mode=BUFFER_TO_RESULT)
+ * PhysicalHashAggregate(groupBy=[name], output=[name, count(distinct(id))], mode=BUFFER_TO_RESULT)
* |
* PhysicalHashAggregate(groupBy=[name, id], output=[name, id], mode=INPUT_TO_BUFFER)
* |
- * PhysicalDistribute(distributionSpec=HASH(name))
- * |
* LogicalOlapScan(table=tbl, **if distribute by name**)
*
*/
@@ -1175,8 +1173,9 @@ public class AggregateStrategies implements ImplementationRuleFactory {
if (outputChild instanceof AggregateFunction) {
AggregateFunction aggregateFunction = (AggregateFunction) outputChild;
if (aggregateFunction.isDistinct()) {
- Set<Expression> aggChild = Sets.newHashSet(aggregateFunction.children());
- Preconditions.checkArgument(aggChild.size() == 1,
+ Set<Expression> aggChild = Sets.newLinkedHashSet(aggregateFunction.children());
+ Preconditions.checkArgument(aggChild.size() == 1
+ || aggregateFunction.getDistinctArguments().size() == 1,
"cannot process more than one child in aggregate distinct function: "
+ aggregateFunction);
AggregateFunction nonDistinct = aggregateFunction
@@ -1236,7 +1235,7 @@ public class AggregateStrategies implements ImplementationRuleFactory {
* after:
* single node aggregate:
* <p>
- * PhysicalHashAggregate(groupBy=[name], output=[name, count(distinct(id)], mode=BUFFER_TO_RESULT)
+ * PhysicalHashAggregate(groupBy=[name], output=[name, count(distinct(id))], mode=BUFFER_TO_RESULT)
* |
* PhysicalHashAggregate(groupBy=[name, id], output=[name, id], mode=BUFFER_TO_BUFFER)
* |
@@ -1248,7 +1247,7 @@ public class AggregateStrategies implements ImplementationRuleFactory {
* <p>
* distribute node aggregate:
* <p>
- * PhysicalHashAggregate(groupBy=[name], output=[name, count(distinct(id)], mode=BUFFER_TO_RESULT)
+ * PhysicalHashAggregate(groupBy=[name], output=[name, count(distinct(id))], mode=BUFFER_TO_RESULT)
* |
* PhysicalHashAggregate(groupBy=[name, id], output=[name, id], mode=BUFFER_TO_BUFFER)
* |
@@ -1331,14 +1330,14 @@ public class AggregateStrategies implements ImplementationRuleFactory {
if (expr instanceof AggregateFunction) {
AggregateFunction aggregateFunction = (AggregateFunction) expr;
if (aggregateFunction.isDistinct()) {
- Set<Expression> aggChild = Sets.newHashSet(aggregateFunction.children());
- Preconditions.checkArgument(aggChild.size() == 1,
+ Set<Expression> aggChild = Sets.newLinkedHashSet(aggregateFunction.children());
+ Preconditions.checkArgument(aggChild.size() == 1
+ || aggregateFunction.getDistinctArguments().size() == 1,
"cannot process more than one child in aggregate distinct function: "
+ aggregateFunction);
AggregateFunction nonDistinct = aggregateFunction
.withDistinctAndChildren(false, ImmutableList.copyOf(aggChild));
- return new AggregateExpression(nonDistinct,
- bufferToResultParam, aggregateFunction.child(0));
+ return new AggregateExpression(nonDistinct, bufferToResultParam, aggregateFunction);
} else {
Alias alias = nonDistinctAggFunctionToAliasPhase2.get(expr);
return new AggregateExpression(aggregateFunction,
@@ -1727,8 +1726,9 @@ public class AggregateStrategies implements ImplementationRuleFactory {
if (expr instanceof AggregateFunction) {
AggregateFunction aggregateFunction = (AggregateFunction) expr;
if (aggregateFunction.isDistinct()) {
- Set<Expression> aggChild = Sets.newHashSet(aggregateFunction.children());
- Preconditions.checkArgument(aggChild.size() == 1,
+ Set<Expression> aggChild = Sets.newLinkedHashSet(aggregateFunction.children());
+ Preconditions.checkArgument(aggChild.size() == 1
+ || aggregateFunction.getDistinctArguments().size() == 1,
"cannot process more than one child in aggregate distinct function: "
+ aggregateFunction);
AggregateFunction nonDistinct = aggregateFunction
@@ -1767,8 +1767,9 @@ public class AggregateStrategies implements ImplementationRuleFactory {
if (expr instanceof AggregateFunction) {
AggregateFunction aggregateFunction = (AggregateFunction) expr;
if (aggregateFunction.isDistinct()) {
- Set<Expression> aggChild = Sets.newHashSet(aggregateFunction.children());
- Preconditions.checkArgument(aggChild.size() == 1,
+ Set<Expression> aggChild = Sets.newLinkedHashSet(aggregateFunction.children());
+ Preconditions.checkArgument(aggChild.size() == 1
+ || aggregateFunction.getDistinctArguments().size() == 1,
"cannot process more than one child in aggregate distinct function: "
+ aggregateFunction);
AggregateFunction nonDistinct = aggregateFunction
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AggregateFunction.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AggregateFunction.java
index a7e523dfdb5..4f53b383d24 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AggregateFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AggregateFunction.java
@@ -124,4 +124,7 @@ public abstract class AggregateFunction extends BoundFunction implements Expects
return getName() + "(" + (distinct ? "DISTINCT " : "") + args + ")";
}
+ public List<Expression> getDistinctArguments() {
+ return distinct ? getArguments() : ImmutableList.of();
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/GroupConcat.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/GroupConcat.java
index 0f2e7bcb03a..d8b6646cff7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/GroupConcat.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/GroupConcat.java
@@ -111,6 +111,15 @@ public class GroupConcat extends NullableAggregateFunction
.anyMatch(expression -> !(expression instanceof OrderExpression) && expression.nullable());
}
+ @Override
+ public List<Expression> getDistinctArguments() {
+ if (distinct) {
+ return ImmutableList.of(getArgument(0));
+ } else {
+ return ImmutableList.of();
+ }
+ }
+
@Override
public void checkLegalityBeforeTypeCoercion() {
DataType typeOrArg0 = getArgumentType(0);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Aggregate.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Aggregate.java
index 8361e230be7..15fd5bec868 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Aggregate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Aggregate.java
@@ -56,7 +56,7 @@ public interface Aggregate<CHILD_TYPE extends Plan> extends UnaryPlan<CHILD_TYPE
default Set<Expression> getDistinctArguments() {
return getAggregateFunctions().stream()
.filter(AggregateFunction::isDistinct)
- .flatMap(aggregateExpression -> aggregateExpression.getArguments().stream())
+ .flatMap(aggregateFunction -> aggregateFunction.getDistinctArguments().stream())
.collect(ImmutableSet.toImmutableSet());
}
}
diff --git a/regression-test/data/nereids_syntax_p0/group_concat.out b/regression-test/data/nereids_syntax_p0/group_concat.out
new file mode 100644
index 00000000000..6e3ab42329b
--- /dev/null
+++ b/regression-test/data/nereids_syntax_p0/group_concat.out
@@ -0,0 +1,8 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !group_by_distinct --
+1 \N
+2 a
+3 b
+4 c
+5 \N
+
diff --git a/regression-test/suites/nereids_syntax_p0/group_concat.groovy b/regression-test/suites/nereids_syntax_p0/group_concat.groovy
index 60f52c2ba06..b46091616ba 100644
--- a/regression-test/suites/nereids_syntax_p0/group_concat.groovy
+++ b/regression-test/suites/nereids_syntax_p0/group_concat.groovy
@@ -48,6 +48,47 @@ suite("group_concat") {
sql "select group_concat(cast(number as string), NULL) from numbers('number'='10')"
result([[null]])
}
-
-
+
+ def testGroupByDistinct = {
+ sql "drop table if exists test_group_concat_distinct_tbl1"
+ sql """create table test_group_concat_distinct_tbl1(
+ tbl1_id1 int
+ ) distributed by hash(tbl1_id1)
+ properties('replication_num'='1')
+ """
+
+ sql "insert into test_group_concat_distinct_tbl1 values(1), (2), (3), (4), (5)"
+
+
+ sql "drop table if exists test_group_concat_distinct_tbl2"
+ sql """create table test_group_concat_distinct_tbl2(
+ tbl2_id1 int,
+ tbl2_id2 int,
+ ) distributed by hash(tbl2_id1)
+ properties('replication_num'='1')
+ """
+ sql "insert into test_group_concat_distinct_tbl2 values(1, 11), (2, 22), (3, 33), (4, 44)"
+
+
+ sql "drop table if exists test_group_concat_distinct_tbl3"
+ sql """create table test_group_concat_distinct_tbl3(
+ tbl3_id2 int,
+ tbl3_name varchar(255)
+ ) distributed by hash(tbl3_id2)
+ properties('replication_num'='1')
+ """
+ sql "insert into test_group_concat_distinct_tbl3 values(22, 'a'), (33, 'b'), (44, 'c')"
+
+ sql "sync"
+
+ order_qt_group_by_distinct """
+ SELECT
+ tbl1.tbl1_id1,
+ group_concat(DISTINCT tbl3.tbl3_name, ',') AS `names`
+ FROM test_group_concat_distinct_tbl1 tbl1
+ LEFT OUTER JOIN test_group_concat_distinct_tbl2 tbl2 ON tbl2.tbl2_id1 = tbl1.tbl1_id1
+ LEFT OUTER JOIN test_group_concat_distinct_tbl3 tbl3 ON tbl3.tbl3_id2 = tbl2.tbl2_id2
+ GROUP BY tbl1.tbl1_id1
+ """
+ }()
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/38: [fix](build)Fix index_tool build error (#31834)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2e94512e0755767c2c6d82680facd5cf8c45ffa9
Author: qiye <ji...@gmail.com>
AuthorDate: Wed Mar 6 13:26:46 2024 +0800
[fix](build)Fix index_tool build error (#31834)
Co-authored-by: Luennng <lu...@gmail.com>
---
be/src/index-tools/index_tool.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/index-tools/index_tool.cpp b/be/src/index-tools/index_tool.cpp
index 91beb905ae2..f18ecdc3621 100644
--- a/be/src/index-tools/index_tool.cpp
+++ b/be/src/index-tools/index_tool.cpp
@@ -266,7 +266,7 @@ int main(int argc, char** argv) {
bool exists = false;
std::filesystem::path root_dir(FLAGS_directory);
doris::Status status = fs->list(root_dir, true, &files, &exists);
- if (!status.ok) {
+ if (!status.ok()) {
std::cerr << "can't search from directory's all files,err : " << status
<< std::endl;
return -1;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 29/38: [fix](money_format) fix money_format #31883
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 137666651493115c1730c40d65abf8af9d727b42
Author: zhiqiang <se...@163.com>
AuthorDate: Thu Mar 7 11:26:41 2024 +0800
[fix](money_format) fix money_format #31883
---
be/src/vec/functions/function.cpp | 2 +-
be/src/vec/functions/function_string.h | 30 +++++++++++++++-------
.../string_functions/test_string_function.out | 24 +++++++++++++++++
.../string_functions/test_string_function.groovy | 8 ++++++
4 files changed, 54 insertions(+), 10 deletions(-)
diff --git a/be/src/vec/functions/function.cpp b/be/src/vec/functions/function.cpp
index 6a033d52f03..b4b0e82a0aa 100644
--- a/be/src/vec/functions/function.cpp
+++ b/be/src/vec/functions/function.cpp
@@ -141,7 +141,7 @@ Status PreparedFunctionImpl::default_implementation_for_constant_arguments(
return Status::OK();
}
- // now all columns is const.
+ // now all columns are const.
Block temporary_block;
size_t arguments_size = args.size();
diff --git a/be/src/vec/functions/function_string.h b/be/src/vec/functions/function_string.h
index c789345743e..a1f5c9e1306 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -26,6 +26,7 @@
#include <array>
#include <boost/iterator/iterator_facade.hpp>
#include <cstddef>
+#include <cstdlib>
#include <iomanip>
#include <memory>
#include <ostream>
@@ -36,6 +37,7 @@
#include <vector>
#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/exception.h"
#include "common/status.h"
#include "gutil/strings/numbers.h"
#include "gutil/strings/substitute.h"
@@ -2941,10 +2943,15 @@ template <typename Impl>
class FunctionMoneyFormat : public IFunction {
public:
static constexpr auto name = "money_format";
- static FunctionPtr create() { return std::make_shared<FunctionMoneyFormat<Impl>>(); }
+ static FunctionPtr create() { return std::make_shared<FunctionMoneyFormat>(); }
String get_name() const override { return name; }
DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ if (arguments.size() != 1) {
+ throw doris::Exception(ErrorCode::INVALID_ARGUMENT,
+ "Function {} requires exactly 1 argument", name);
+ }
+
return std::make_shared<DataTypeString>();
}
DataTypes get_variadic_argument_types_impl() const override {
@@ -3017,9 +3024,11 @@ struct MoneyFormatDoubleImpl {
static void execute(FunctionContext* context, ColumnString* result_column,
const ColumnPtr col_ptr, size_t input_rows_count) {
const auto* data_column = assert_cast<const ColumnVector<Float64>*>(col_ptr.get());
+ // when scale is above 38, we will go here
for (size_t i = 0; i < input_rows_count; i++) {
+ // truncate to 2 decimal places, keep same with mysql
double value =
- MathFunctions::my_double_round(data_column->get_element(i), 2, false, false);
+ MathFunctions::my_double_round(data_column->get_element(i), 2, false, true);
StringRef str = MoneyFormat::do_money_format(context, fmt::format("{:.2f}", value));
result_column->insert_data(str.data, str.size);
}
@@ -3076,8 +3085,8 @@ struct MoneyFormatDecimalImpl {
} else if (auto* decimal32_column =
check_and_get_column<ColumnDecimal<Decimal32>>(*col_ptr)) {
const UInt32 scale = decimal32_column->get_scale();
- const auto multiplier =
- scale > 2 ? common::exp10_i32(scale - 2) : common::exp10_i32(2 - scale);
+ // scale is up to 9, so exp10_i32 is enough
+ const auto multiplier = common::exp10_i32(std::abs(static_cast<int>(scale - 2)));
for (size_t i = 0; i < input_rows_count; i++) {
Decimal32 frac_part = decimal32_column->get_fractional_part(i);
if (scale > 2) {
@@ -3095,8 +3104,8 @@ struct MoneyFormatDecimalImpl {
} else if (auto* decimal64_column =
check_and_get_column<ColumnDecimal<Decimal64>>(*col_ptr)) {
const UInt32 scale = decimal64_column->get_scale();
- const auto multiplier =
- scale > 2 ? common::exp10_i32(scale - 2) : common::exp10_i32(2 - scale);
+ // 9 < scale <= 18
+ const auto multiplier = common::exp10_i64(std::abs(static_cast<int>(scale - 2)));
for (size_t i = 0; i < input_rows_count; i++) {
Decimal64 frac_part = decimal64_column->get_fractional_part(i);
if (scale > 2) {
@@ -3114,8 +3123,8 @@ struct MoneyFormatDecimalImpl {
} else if (auto* decimal128_column =
check_and_get_column<ColumnDecimal<Decimal128V3>>(*col_ptr)) {
const UInt32 scale = decimal128_column->get_scale();
- const auto multiplier =
- scale > 2 ? common::exp10_i32(scale - 2) : common::exp10_i32(2 - scale);
+ // 18 < scale <= 38
+ const auto multiplier = common::exp10_i128(std::abs(static_cast<int>(scale - 2)));
for (size_t i = 0; i < input_rows_count; i++) {
Decimal128V3 frac_part = decimal128_column->get_fractional_part(i);
if (scale > 2) {
@@ -3130,6 +3139,9 @@ struct MoneyFormatDecimalImpl {
result_column->insert_data(str.data, str.size);
}
+ } else {
+ throw doris::Exception(ErrorCode::INVALID_ARGUMENT,
+ "Not supported input argument type {}", col_ptr->get_name());
}
// TODO: decimal256
/* else if (auto* decimal256_column =
@@ -3635,7 +3647,7 @@ public:
// +---------------------------------------------------------------------------------------------+
// | char(0xe5, 0xa4, 0x9a, 0xe7, 0x9d, 0xbf, 0xe4, 0xb8, 0x9d, 68, 111, 114, 105, 115 using utf8) |
// +---------------------------------------------------------------------------------------------+
-// | 多睿丝Doris |
+// | 多睿丝 Doris |
// +---------------------------------------------------------------------------------------------+
// mysql> select char(68, 111, 114, 0, 105, null, 115 using utf8);
// +--------------------------------------------------+
diff --git a/regression-test/data/nereids_p0/sql_functions/string_functions/test_string_function.out b/regression-test/data/nereids_p0/sql_functions/string_functions/test_string_function.out
index 7f2e30b5c14..090ec198a7b 100644
--- a/regression-test/data/nereids_p0/sql_functions/string_functions/test_string_function.out
+++ b/regression-test/data/nereids_p0/sql_functions/string_functions/test_string_function.out
@@ -170,6 +170,30 @@ ab d
-- !sql --
1,123.40
+-- !sql --
+1.12
+
+-- !sql_decimal32 --
+1.12
+
+-- !sql_decimal64 --
+1.12
+
+-- !sql_decimal64 --
+1.12
+
+-- !sql_decimal128 --
+1.12
+
+-- !sql_decimal128 --
+1.12
+
+-- !sql_float64 --
+1.12
+
+-- !sql_float64 --
+1.12
+
-- !sql --
true
diff --git a/regression-test/suites/nereids_p0/sql_functions/string_functions/test_string_function.groovy b/regression-test/suites/nereids_p0/sql_functions/string_functions/test_string_function.groovy
index 64fcc2e372a..20c8294b114 100644
--- a/regression-test/suites/nereids_p0/sql_functions/string_functions/test_string_function.groovy
+++ b/regression-test/suites/nereids_p0/sql_functions/string_functions/test_string_function.groovy
@@ -104,6 +104,14 @@ suite("test_string_function") {
qt_sql "select money_format(17014116);"
qt_sql "select money_format(1123.456);"
qt_sql "select money_format(1123.4);"
+ qt_sql "select money_format(1.1249);"
+ qt_sql_decimal32 "select money_format(cast(concat('1.124', repeat('9', 5)) as DECIMAL(9, 8)));"
+ qt_sql_decimal64 "select money_format(cast(concat('1.124', repeat('9', 6)) as DECIMAL(10, 9)));"
+ qt_sql_decimal64 "select money_format(cast(concat('1.124', repeat('9', 14)) as DECIMAL(18, 17)));"
+ qt_sql_decimal128 "select money_format(cast(concat('1.124', repeat('9', 15)) as DECIMAL(19, 18)));"
+ qt_sql_decimal128 "select money_format(cast(concat('1.124', repeat('9', 34)) as DECIMAL(38, 37)));"
+ qt_sql_float64 "select money_format(cast(concat('1.124', repeat('9', 35)) as DOUBLE));"
+ qt_sql_float64 "select money_format(cast(concat('1.124', repeat('9', 70)) as DOUBLE));"
qt_sql "select null_or_empty(null);"
qt_sql "select null_or_empty(\"\");"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 28/38: [fix](mtmv) support null value in partition for updating (#31843)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 73ebeb24a3c1f176d5ff9c4bc794d4d9522b39bb
Author: 谢健 <ji...@gmail.com>
AuthorDate: Thu Mar 7 11:18:37 2024 +0800
[fix](mtmv) support null value in partition for updating (#31843)
---
.../plans/commands/UpdateMvByPartitionCommand.java | 19 ++++++++++++++++++-
.../commands/UpdateMvByPartitionCommandTest.java | 21 +++++++++++++++++++++
2 files changed, 39 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/UpdateMvByPartitionCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/UpdateMvByPartitionCommand.java
index 0c976506a87..f6bd39ac838 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/UpdateMvByPartitionCommand.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/UpdateMvByPartitionCommand.java
@@ -31,10 +31,12 @@ import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
import org.apache.doris.nereids.trees.expressions.InPredicate;
+import org.apache.doris.nereids.trees.expressions.IsNull;
import org.apache.doris.nereids.trees.expressions.LessThan;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.algebra.Sink;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
@@ -54,6 +56,7 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.stream.Collectors;
/**
* Update mv by partition
@@ -123,7 +126,21 @@ public class UpdateMvByPartitionCommand extends InsertOverwriteTableCommand {
List<Expression> inValues = ((ListPartitionItem) item).getItems().stream()
.map(UpdateMvByPartitionCommand::convertPartitionKeyToLiteral)
.collect(ImmutableList.toImmutableList());
- return new InPredicate(col, inValues);
+ List<Expression> predicates = new ArrayList<>();
+ if (inValues.stream().anyMatch(NullLiteral.class::isInstance)) {
+ inValues = inValues.stream()
+ .filter(e -> !(e instanceof NullLiteral))
+ .collect(Collectors.toList());
+ Expression isNullPredicate = new IsNull(col);
+ predicates.add(isNullPredicate);
+ }
+ if (!inValues.isEmpty()) {
+ predicates.add(new InPredicate(col, inValues));
+ }
+ if (predicates.isEmpty()) {
+ return BooleanLiteral.of(true);
+ }
+ return ExpressionUtils.or(predicates);
} else {
Range<PartitionKey> range = item.getItems();
List<Expression> exprs = new ArrayList<>();
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/UpdateMvByPartitionCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/UpdateMvByPartitionCommandTest.java
index af00d8ae464..b7374565da1 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/UpdateMvByPartitionCommandTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/UpdateMvByPartitionCommandTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.nereids.trees.plans.commands;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ListPartitionItem;
import org.apache.doris.catalog.PartitionItem;
import org.apache.doris.catalog.PartitionKey;
import org.apache.doris.catalog.PrimitiveType;
@@ -26,6 +27,7 @@ import org.apache.doris.catalog.RangePartitionItem;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.GreaterThanEqual;
+import org.apache.doris.nereids.trees.expressions.IsNull;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.types.IntegerType;
@@ -53,4 +55,23 @@ class UpdateMvByPartitionCommandTest {
Expression expr = (Expression) m.invoke(null, rangePartitionItem, new SlotReference("s", IntegerType.INSTANCE));
Assertions.assertTrue(expr instanceof GreaterThanEqual);
}
+
+ @Test
+ void testNull() throws AnalysisException, NoSuchMethodException, InvocationTargetException,
+ IllegalAccessException {
+ Method m = UpdateMvByPartitionCommand.class.getDeclaredMethod("convertPartitionItemToPredicate", PartitionItem.class,
+ Slot.class);
+ m.setAccessible(true);
+ Column column = new Column("a", PrimitiveType.INT);
+ PartitionKey v = PartitionKey.createListPartitionKeyWithTypes(ImmutableList.of(new PartitionValue("NULL", true)), ImmutableList.of(column.getType()), false);
+ ListPartitionItem listPartitionItem = new ListPartitionItem(ImmutableList.of(v));
+ Expression expr = (Expression) m.invoke(null, listPartitionItem, new SlotReference("s", IntegerType.INSTANCE));
+ Assertions.assertTrue(expr instanceof IsNull);
+
+ PartitionKey v1 = PartitionKey.createListPartitionKeyWithTypes(ImmutableList.of(new PartitionValue("NULL", true)), ImmutableList.of(column.getType()), false);
+ PartitionKey v2 = PartitionKey.createListPartitionKeyWithTypes(ImmutableList.of(new PartitionValue("1", false)), ImmutableList.of(column.getType()), false);
+ listPartitionItem = new ListPartitionItem(ImmutableList.of(v1, v2));
+ expr = (Expression) m.invoke(null, listPartitionItem, new SlotReference("s", IntegerType.INSTANCE));
+ Assertions.assertEquals("(s IS NULL OR s IN (1))", expr.toSql());
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 10/38: [fix](Nereids) json_object can't be odd parameters, need even parameters (#31767)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c7d86c062ad39edf16c5ed251e098ba925f00bd9
Author: zzwwhh <30...@users.noreply.github.com>
AuthorDate: Wed Mar 6 20:37:48 2024 +0800
[fix](Nereids) json_object can't be odd parameters, need even parameters (#31767)
---
.../doris/nereids/trees/expressions/functions/scalar/JsonObject.java | 3 +++
1 file changed, 3 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/JsonObject.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/JsonObject.java
index e1e2c961ef2..6d3d1536eb8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/JsonObject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/JsonObject.java
@@ -49,6 +49,9 @@ public class JsonObject extends ScalarFunction
@Override
public void checkLegalityBeforeTypeCoercion() {
+ if ((arity() & 1) == 1) {
+ throw new AnalysisException("json_object can't be odd parameters, need even parameters: " + this.toSql());
+ }
for (int i = 0; i < arity(); i++) {
if ((i & 1) == 0 && getArgumentType(i).isNullType()) {
throw new AnalysisException("json_object key can't be NULL: " + this.toSql());
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 33/38: [cloud](point query) enable short circuit query in cloud (#31897)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 44f41054d8c22d4942d4b8be10b526562b44aaf9
Author: lihangyu <15...@163.com>
AuthorDate: Thu Mar 7 12:25:43 2024 +0800
[cloud](point query) enable short circuit query in cloud (#31897)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org