You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ki...@apache.org on 2026/05/27 06:49:40 UTC
(doris) branch 4.1-62196 created (now ad120c47d3f)
This is an automated email from the ASF dual-hosted git repository.
CalvinKirs pushed a change to branch 4.1-62196
in repository https://gitbox.apache.org/repos/asf/doris.git
at ad120c47d3f [feature](fe) Add partition filter sql block rule (#62196)
This branch includes the following new commits:
new ad120c47d3f [feature](fe) Add partition filter sql block rule (#62196)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/01: [feature](fe) Add partition filter sql block rule (#62196)
Posted by ki...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
CalvinKirs pushed a commit to branch 4.1-62196
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ad120c47d3f845ffa72b96f30718447c3e4db0f3
Author: Calvin Kirs <gu...@selectdb.com>
AuthorDate: Thu Apr 30 15:28:49 2026 +0800
[feature](fe) Add partition filter sql block rule (#62196)
Today SQL_BLOCK_RULE can block SQL by regex/sqlHash or by scan scale
thresholds such as
`partition_num`, `tablet_num`, and `cardinality`, but it cannot directly
reject queries that
scan a partitioned table without using any partition filter.
This PR adds a new scan-based SQL block rule option:
`require_partition_filter`.
When this option is enabled on a rule, Doris rejects scans on supported
partitioned tables if
the query does not contain any usable partition predicate. This helps
prevent accidental full
partition scans caused by missing partition filters.
The rule is intended for workloads where partition filters are mandatory
for safety or cost
control.
### User-visible behavior
Users can now create a SQL block rule like this:
```sql
CREATE SQL_BLOCK_RULE require_partition_filter_rule
PROPERTIES(
"require_partition_filter" = "true",
"global" = "true",
"enable" = "true"
);
Or bind it to specific users:
CREATE SQL_BLOCK_RULE require_partition_filter_rule
PROPERTIES(
"require_partition_filter" = "true",
"global" = "false",
"enable" = "true"
);
SET PROPERTY FOR 'test_user' 'sql_block_rules' =
'require_partition_filter_rule';
If a supported partitioned table is scanned without any partition
predicate, Doris returns an
error like:
sql hits sql block rule: require_partition_filter_rule, missing
partition filter
### Scope
This new rule currently applies to:
- Partitioned internal tables
- Partitioned Hive external tables
This rule does not apply to:
- Non-partitioned internal tables
- Non-partitioned Hive tables
- Iceberg tables
- Other external table types not wired into this rule yet
### Rule semantics
The new property is:
- require_partition_filter = true|false
Behavior:
- The rule only takes effect when require_partition_filter=true
- For supported partitioned tables, the scan is allowed if the query
hits any partition column
in partition pruning predicates
- Filters on non-partition columns do not count
- The rule applies to scan-producing statements, such as:
- SELECT
- INSERT INTO ... SELECT ...
- EXPLAIN is not blocked
Examples:
Blocked:
SELECT * FROM part_tbl;
SELECT * FROM part_tbl WHERE non_partition_col = 1;
INSERT INTO dst SELECT * FROM part_tbl;
Allowed:
SELECT * FROM part_tbl WHERE dt = '2026-04-09';
SELECT * FROM part_tbl WHERE dt = '2026-04-09' AND hh = '10';
INSERT INTO dst SELECT * FROM part_tbl WHERE dt = '2026-04-09';
### Compatibility and validation
require_partition_filter is treated as a scan-based SQL block condition.
It can be used together with existing scan-based limits such as:
- partition_num
- tablet_num
- cardinality
It cannot be mixed with text-based block conditions such as:
- sql
- sqlHash
---
.../schema_sql_block_rule_status_scanner.cpp | 32 ++-
.../org/apache/doris/blockrule/SqlBlockRule.java | 25 +-
.../apache/doris/blockrule/SqlBlockRuleMgr.java | 59 ++--
.../java/org/apache/doris/catalog/SchemaTable.java | 2 +
.../org/apache/doris/common/util/SqlBlockUtil.java | 87 +++---
.../java/org/apache/doris/common/util/Util.java | 26 ++
.../doris/datasource/hive/source/HiveScanNode.java | 7 +-
.../glue/translator/PhysicalPlanTranslator.java | 2 +
.../rules/expression/rules/PartitionPruner.java | 74 ++++-
.../LogicalOlapScanToPhysicalOlapScan.java | 1 +
.../rules/rewrite/PruneFileScanPartition.java | 10 +-
.../rules/rewrite/PruneOlapScanPartition.java | 52 ++--
.../plans/commands/AlterSqlBlockRuleCommand.java | 15 +-
.../plans/commands/CreateSqlBlockRuleCommand.java | 13 +-
.../plans/commands/ShowSqlBlockRuleCommand.java | 2 +
.../trees/plans/commands/SqlBlockRuleCommand.java | 13 +-
.../trees/plans/logical/LogicalFileScan.java | 26 +-
.../trees/plans/logical/LogicalOlapScan.java | 69 +++--
.../trees/plans/physical/PhysicalFileScan.java | 4 +
.../trees/plans/physical/PhysicalOlapScan.java | 44 ++-
.../org/apache/doris/planner/OlapScanNode.java | 3 +
.../java/org/apache/doris/planner/ScanNode.java | 30 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 15 +
.../doris/tablefunction/MetadataGenerator.java | 1 +
.../doris/blockrule/SqlBlockRuleMgrTest.java | 69 +++++
.../org/apache/doris/catalog/SchemaTableTest.java | 1 +
.../org/apache/doris/common/util/UtilTest.java | 27 ++
.../datasource/hive/source/HiveScanNodeTest.java | 44 +++
.../translator/PhysicalPlanTranslatorTest.java | 142 ++++++++--
.../nereids/mv/OptimizeGetAvailableMvsTest.java | 176 ++++++------
.../nereids/rules/rewrite/PartitionPrunerTest.java | 302 ++++++++++++++++++++-
.../plans/commands/SqlBlockRuleCommandTest.java | 124 +++++++++
.../org/apache/doris/planner/OlapScanNodeTest.java | 51 ++++
.../java/org/apache/doris/qe/StmtExecutorTest.java | 93 +++++--
.../apache/doris/utframe/TestWithFeService.java | 5 +-
.../hive/test_external_sql_block_rule.groovy | 93 ++++++-
.../schema_table/test_sql_block_rule_status.groovy | 12 +-
.../sql_block_rule_p0/test_sql_block_rule.groovy | 111 +++++++-
38 files changed, 1537 insertions(+), 325 deletions(-)
diff --git a/be/src/information_schema/schema_sql_block_rule_status_scanner.cpp b/be/src/information_schema/schema_sql_block_rule_status_scanner.cpp
index 065a610bf47..2a223c144ba 100644
--- a/be/src/information_schema/schema_sql_block_rule_status_scanner.cpp
+++ b/be/src/information_schema/schema_sql_block_rule_status_scanner.cpp
@@ -32,6 +32,11 @@
namespace doris {
+namespace {
+constexpr size_t LEGACY_SQL_BLOCK_RULE_STATUS_COLUMN_SIZE = 12;
+constexpr size_t REQUIRE_PARTITION_FILTER_INDEX = 8;
+} // namespace
+
std::vector<SchemaScanner::ColumnDesc>
SchemaSqlBlockRuleStatusScanner::_s_sql_block_rule_status_columns = {
{"NAME", TYPE_STRING, sizeof(StringRef), true},
@@ -42,6 +47,7 @@ std::vector<SchemaScanner::ColumnDesc>
{"CARDINALITY", TYPE_BIGINT, sizeof(int64_t), true},
{"GLOBAL", TYPE_BOOLEAN, sizeof(bool), true},
{"ENABLE", TYPE_BOOLEAN, sizeof(bool), true},
+ {"REQUIRE_PARTITION_FILTER", TYPE_BOOLEAN, sizeof(bool), true},
{"BLOCKS", TYPE_BIGINT, sizeof(int64_t), true},
{"AVERAGE_DURATION", TYPE_BIGINT, sizeof(int64_t), true},
{"LONGEST_DURATION", TYPE_BIGINT, sizeof(int64_t), true},
@@ -65,11 +71,6 @@ Status SchemaSqlBlockRuleStatusScanner::_get_sql_block_rule_status_block_from_fe
}
TSchemaTableRequestParams schema_table_request_params;
- for (int i = 0; i < _s_sql_block_rule_status_columns.size(); i++) {
- schema_table_request_params.__isset.columns_name = true;
- schema_table_request_params.columns_name.emplace_back(
- _s_sql_block_rule_status_columns[i].name);
- }
schema_table_request_params.__set_current_user_ident(*_param->common_param->current_user_ident);
schema_table_request_params.__set_frontend_conjuncts(*_param->common_param->frontend_conjuncts);
@@ -112,16 +113,31 @@ Status SchemaSqlBlockRuleStatusScanner::_get_sql_block_rule_status_block_from_fe
if (result_data.size() > 0) {
auto col_size = result_data[0].column_value.size();
- if (col_size != _s_sql_block_rule_status_columns.size()) {
+ if (col_size != LEGACY_SQL_BLOCK_RULE_STATUS_COLUMN_SIZE &&
+ col_size != _s_sql_block_rule_status_columns.size()) {
return Status::InternalError("col size not equal");
}
}
// Add rows from this FE to the result block
for (int i = 0; i < result_data.size(); i++) {
- TRow row = result_data[i];
+ const TRow& row = result_data[i];
+ auto col_size = row.column_value.size();
for (int j = 0; j < _s_sql_block_rule_status_columns.size(); j++) {
- RETURN_IF_ERROR(insert_block_column(row.column_value[j], j,
+ if (col_size == LEGACY_SQL_BLOCK_RULE_STATUS_COLUMN_SIZE &&
+ j == REQUIRE_PARTITION_FILTER_INDEX) {
+ TCell default_cell;
+ default_cell.__set_boolVal(false);
+ RETURN_IF_ERROR(insert_block_column(default_cell, j,
+ _sql_block_rule_status_block.get(),
+ _s_sql_block_rule_status_columns[j].type));
+ continue;
+ }
+ int row_index = col_size == LEGACY_SQL_BLOCK_RULE_STATUS_COLUMN_SIZE &&
+ j > REQUIRE_PARTITION_FILTER_INDEX
+ ? j - 1
+ : j;
+ RETURN_IF_ERROR(insert_block_column(row.column_value[row_index], j,
_sql_block_rule_status_block.get(),
_s_sql_block_rule_status_columns[j].type));
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRule.java b/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRule.java
index da962d9a3f7..210eabcbd6b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRule.java
@@ -75,6 +75,10 @@ public class SqlBlockRule implements Writable, GsonPostProcessable {
@SerializedName(value = "enable")
private Boolean enable;
+ // whether partitioned table scans require partition filters
+ @SerializedName(value = "requirePartitionFilter")
+ private Boolean requirePartitionFilter;
+
private Pattern sqlPattern;
private Histogram tryBlockHistogram;
private LongCounterMetric blockCount;
@@ -83,13 +87,14 @@ public class SqlBlockRule implements Writable, GsonPostProcessable {
* Create SqlBlockRule.
**/
public SqlBlockRule(String name, String sql, String sqlHash, Long partitionNum, Long tabletNum, Long cardinality,
- Boolean global, Boolean enable) {
+ Boolean requirePartitionFilter, Boolean global, Boolean enable) {
this.name = name;
this.sql = sql;
this.sqlHash = sqlHash;
this.partitionNum = partitionNum;
this.tabletNum = tabletNum;
this.cardinality = cardinality;
+ this.requirePartitionFilter = requirePartitionFilter;
this.global = global;
this.enable = enable;
if (StringUtils.isNotEmpty(sql)) {
@@ -103,6 +108,7 @@ public class SqlBlockRule implements Writable, GsonPostProcessable {
public SqlBlockRule() {
this.tryBlockHistogram = new Histogram(new SlidingWindowReservoir(1000));
this.blockCount = new LongCounterMetric("blocks", MetricUnit.ROWS, "");
+ this.requirePartitionFilter = false;
}
public String getName() {
@@ -141,6 +147,10 @@ public class SqlBlockRule implements Writable, GsonPostProcessable {
return enable;
}
+ public Boolean getRequirePartitionFilter() {
+ return requirePartitionFilter;
+ }
+
public void setSql(String sql) {
this.sql = sql;
}
@@ -173,6 +183,10 @@ public class SqlBlockRule implements Writable, GsonPostProcessable {
this.enable = enable;
}
+ public void setRequirePartitionFilter(Boolean requirePartitionFilter) {
+ this.requirePartitionFilter = requirePartitionFilter;
+ }
+
/**
* Show SqlBlockRule info.
**/
@@ -181,7 +195,11 @@ public class SqlBlockRule implements Writable, GsonPostProcessable {
this.partitionNum == null ? "0" : Long.toString(this.partitionNum),
this.tabletNum == null ? "0" : Long.toString(this.tabletNum),
this.cardinality == null ? "0" : Long.toString(this.cardinality), String.valueOf(this.global),
- String.valueOf(this.enable));
+ String.valueOf(this.enable), toShowBoolean(this.requirePartitionFilter));
+ }
+
+ private static String toShowBoolean(Boolean value) {
+ return Boolean.TRUE.equals(value) ? "1" : "0";
}
public Histogram getTryBlockHistogram() {
@@ -211,5 +229,8 @@ public class SqlBlockRule implements Writable, GsonPostProcessable {
this.getSql())) {
this.setSqlPattern(Pattern.compile(this.getSql()));
}
+ if (this.getRequirePartitionFilter() == null) {
+ this.setRequirePartitionFilter(false);
+ }
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java b/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
index c6145a638e7..f68307a3d2a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
@@ -161,6 +161,9 @@ public class SqlBlockRuleMgr implements Writable {
if (sqlBlockRule.getEnable() == null) {
sqlBlockRule.setEnable(originRule.getEnable());
}
+ if (sqlBlockRule.getRequirePartitionFilter() == null) {
+ sqlBlockRule.setRequirePartitionFilter(originRule.getRequirePartitionFilter());
+ }
sqlBlockRule.setSqlPattern(Pattern.compile(sqlBlockRule.getSql()));
verifyLimitations(sqlBlockRule);
SqlBlockUtil.checkAlterValidate(sqlBlockRule);
@@ -269,13 +272,22 @@ public class SqlBlockRuleMgr implements Writable {
**/
public void checkLimitations(Long partitionNum, Long tabletNum, Long cardinality, String user)
throws AnalysisException {
+ checkLimitations(partitionNum, tabletNum, cardinality, false, false, user);
+ }
+
+ /**
+ * Check scan limitations whether legal by user.
+ **/
+ public void checkLimitations(Long partitionNum, Long tabletNum, Long cardinality,
+ boolean isPartitionedTable, boolean hasPartitionPredicate, String user) throws AnalysisException {
if (ConnectContext.get().getState().isInternal()) {
return;
}
// match global rule
for (SqlBlockRule rule : nameToSqlBlockRuleMap.values()) {
if (rule.getGlobal()) {
- checkLimitations(rule, partitionNum, tabletNum, cardinality);
+ checkLimitations(rule, partitionNum, tabletNum, cardinality,
+ isPartitionedTable, hasPartitionPredicate);
}
}
// match user rule
@@ -285,33 +297,42 @@ public class SqlBlockRuleMgr implements Writable {
if (rule == null) {
continue;
}
- checkLimitations(rule, partitionNum, tabletNum, cardinality);
+ checkLimitations(rule, partitionNum, tabletNum, cardinality,
+ isPartitionedTable, hasPartitionPredicate);
}
}
/**
* Check number whether legal by SqlBlockRule.
**/
- private void checkLimitations(SqlBlockRule rule, Long partitionNum, Long tabletNum, Long cardinality)
+ @VisibleForTesting
+ void checkLimitations(SqlBlockRule rule, Long partitionNum, Long tabletNum, Long cardinality,
+ boolean isPartitionedTable, boolean hasPartitionPredicate)
throws AnalysisException {
+ if (!rule.getEnable()) {
+ return;
+ }
+ if (Boolean.TRUE.equals(rule.getRequirePartitionFilter()) && isPartitionedTable && !hasPartitionPredicate) {
+ MetricRepo.COUNTER_HIT_SQL_BLOCK_RULE.increase(1L);
+ throw new AnalysisException("sql hits sql block rule: " + rule.getName() + ", missing partition filter");
+ }
if (rule.getPartitionNum() == 0 && rule.getTabletNum() == 0 && rule.getCardinality() == 0) {
return;
- } else if (rule.getEnable()) {
- if ((rule.getPartitionNum() != 0 && rule.getPartitionNum() < partitionNum) || (rule.getTabletNum() != 0
- && rule.getTabletNum() < tabletNum) || (rule.getCardinality() != 0
- && rule.getCardinality() < cardinality)) {
- MetricRepo.COUNTER_HIT_SQL_BLOCK_RULE.increase(1L);
- if (rule.getPartitionNum() < partitionNum && rule.getPartitionNum() != 0) {
- throw new AnalysisException(
- "sql hits sql block rule: " + rule.getName() + ", reach partition_num : "
- + rule.getPartitionNum());
- } else if (rule.getTabletNum() < tabletNum && rule.getTabletNum() != 0) {
- throw new AnalysisException("sql hits sql block rule: " + rule.getName() + ", reach tablet_num : "
- + rule.getTabletNum());
- } else if (rule.getCardinality() < cardinality && rule.getCardinality() != 0) {
- throw new AnalysisException("sql hits sql block rule: " + rule.getName() + ", reach cardinality : "
- + rule.getCardinality());
- }
+ }
+ if ((rule.getPartitionNum() != 0 && rule.getPartitionNum() < partitionNum) || (rule.getTabletNum() != 0
+ && rule.getTabletNum() < tabletNum) || (rule.getCardinality() != 0
+ && rule.getCardinality() < cardinality)) {
+ MetricRepo.COUNTER_HIT_SQL_BLOCK_RULE.increase(1L);
+ if (rule.getPartitionNum() < partitionNum && rule.getPartitionNum() != 0) {
+ throw new AnalysisException(
+ "sql hits sql block rule: " + rule.getName() + ", reach partition_num : "
+ + rule.getPartitionNum());
+ } else if (rule.getTabletNum() < tabletNum && rule.getTabletNum() != 0) {
+ throw new AnalysisException("sql hits sql block rule: " + rule.getName() + ", reach tablet_num : "
+ + rule.getTabletNum());
+ } else if (rule.getCardinality() < cardinality && rule.getCardinality() != 0) {
+ throw new AnalysisException("sql hits sql block rule: " + rule.getName() + ", reach cardinality : "
+ + rule.getCardinality());
}
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
index a54c707e4e4..bccafe64010 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
@@ -792,6 +792,8 @@ public class SchemaTable extends Table {
.column("CARDINALITY", ScalarType.createType(PrimitiveType.BIGINT))
.column("GLOBAL", ScalarType.createType(PrimitiveType.BOOLEAN))
.column("ENABLE", ScalarType.createType(PrimitiveType.BOOLEAN))
+ .column("REQUIRE_PARTITION_FILTER",
+ ScalarType.createType(PrimitiveType.BOOLEAN))
.column("BLOCKS", ScalarType.createType(PrimitiveType.BIGINT),
SchemaTableAggregateType.SUM, false)
.column("AVERAGE_DURATION", ScalarType.createType(PrimitiveType.BIGINT),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/SqlBlockUtil.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/SqlBlockUtil.java
index 7fb2a7d8a8a..07fe00fe562 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/SqlBlockUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/SqlBlockUtil.java
@@ -33,16 +33,17 @@ public class SqlBlockUtil {
public static void checkSqlAndSqlHashSetBoth(String sql, String sqlHash) throws AnalysisException {
- if (!STRING_DEFAULT.equals(sql) && !STRING_DEFAULT.equals(sqlHash)) {
+ if (isSqlConditionConfigured(sql) && isSqlConditionConfigured(sqlHash)) {
throw new AnalysisException("Only sql or sqlHash can be configured");
}
}
// check (sql or sqlHash) and (limitations: partitioNum, tabletNum, cardinality) are not set both
public static void checkSqlAndLimitationsSetBoth(String sql, String sqlHash,
- String partitionNumString, String tabletNumString, String cardinalityString) throws AnalysisException {
- if ((!STRING_DEFAULT.equals(sql) || !STRING_DEFAULT.equals(sqlHash))
- && !isSqlBlockLimitationsEmpty(partitionNumString, tabletNumString, cardinalityString)) {
+ String partitionNumString, String tabletNumString, String cardinalityString,
+ boolean requirePartitionFilter) throws AnalysisException {
+ if (hasSqlCondition(sql, sqlHash)
+ && hasScanCondition(partitionNumString, tabletNumString, cardinalityString, requirePartitionFilter)) {
ErrorReport.reportAnalysisException(ErrorCode.ERROR_SQL_AND_LIMITATIONS_SET_IN_ONE_RULE);
}
}
@@ -50,57 +51,65 @@ public class SqlBlockUtil {
// 1. check (sql or sqlHash) and (limitations: partitioNum, tabletNum, cardinality) are not set both
// 2. check any of limitations is set while sql or sqlHash is not set
public static void checkPropertiesValidate(String sql, String sqlHash,
- String partitionNumString, String tabletNumString, String cardinalityString) throws AnalysisException {
- if (((!STRING_DEFAULT.equals(sql) || !STRING_DEFAULT.equals(sqlHash))
- && !isSqlBlockLimitationsEmpty(partitionNumString, tabletNumString, cardinalityString))
- || ((STRING_DEFAULT.equals(sql) && STRING_DEFAULT.equals(sqlHash))
- && isSqlBlockLimitationsEmpty(partitionNumString, tabletNumString, cardinalityString))) {
+ String partitionNumString, String tabletNumString, String cardinalityString,
+ boolean requirePartitionFilter) throws AnalysisException {
+ boolean hasSqlCondition = hasSqlCondition(sql, sqlHash);
+ boolean hasScanCondition = hasScanCondition(partitionNumString, tabletNumString,
+ cardinalityString, requirePartitionFilter);
+ if ((hasSqlCondition && hasScanCondition) || (!hasSqlCondition && !hasScanCondition)) {
ErrorReport.reportAnalysisException(ErrorCode.ERROR_SQL_AND_LIMITATIONS_SET_IN_ONE_RULE);
}
}
// check at least one of the (limitations: partitioNum, tabletNum, cardinality) is not empty
public static Boolean isSqlBlockLimitationsEmpty(String partitionNumString,
- String tabletNumString, String cardinalityString) {
+ String tabletNumString, String cardinalityString, boolean requirePartitionFilter) {
return StringUtils.isEmpty(partitionNumString)
- && StringUtils.isEmpty(tabletNumString) && StringUtils.isEmpty(cardinalityString);
+ && StringUtils.isEmpty(tabletNumString)
+ && StringUtils.isEmpty(cardinalityString)
+ && !requirePartitionFilter;
}
- public static Boolean isSqlBlockLimitationsDefault(Long partitionNum, Long tabletNum, Long cardinality) {
- return partitionNum == LONG_ZERO && tabletNum == LONG_ZERO && cardinality == LONG_ZERO;
+ public static Boolean isSqlBlockLimitationsDefault(Long partitionNum, Long tabletNum, Long cardinality,
+ Boolean requirePartitionFilter) {
+ return partitionNum == LONG_ZERO && tabletNum == LONG_ZERO && cardinality == LONG_ZERO
+ && !Boolean.TRUE.equals(requirePartitionFilter);
}
- public static Boolean isSqlBlockLimitationsNull(Long partitionNum, Long tabletNum, Long cardinality) {
- return partitionNum == null && tabletNum == null && cardinality == null;
+ public static Boolean isSqlBlockLimitationsNull(Long partitionNum, Long tabletNum, Long cardinality,
+ Boolean requirePartitionFilter) {
+ return partitionNum == null && tabletNum == null && cardinality == null && requirePartitionFilter == null;
}
// alter operation not allowed to change other properties that not set
public static void checkAlterValidate(SqlBlockRule sqlBlockRule) throws AnalysisException {
- if (!STRING_DEFAULT.equals(sqlBlockRule.getSql())) {
- if (!STRING_DEFAULT.equals(sqlBlockRule.getSqlHash())
- && StringUtils.isNotEmpty(sqlBlockRule.getSqlHash())) {
- throw new AnalysisException("Only sql or sqlHash can be configured");
- } else if (!isSqlBlockLimitationsDefault(sqlBlockRule.getPartitionNum(),
- sqlBlockRule.getTabletNum(), sqlBlockRule.getCardinality())
- && !isSqlBlockLimitationsNull(sqlBlockRule.getPartitionNum(),
- sqlBlockRule.getTabletNum(), sqlBlockRule.getCardinality())) {
- ErrorReport.reportAnalysisException(ErrorCode.ERROR_SQL_AND_LIMITATIONS_SET_IN_ONE_RULE);
- }
- } else if (!STRING_DEFAULT.equals(sqlBlockRule.getSqlHash())) {
- if (!STRING_DEFAULT.equals(sqlBlockRule.getSql()) && StringUtils.isNotEmpty(sqlBlockRule.getSql())) {
- throw new AnalysisException("Only sql or sqlHash can be configured");
- } else if (!isSqlBlockLimitationsDefault(sqlBlockRule.getPartitionNum(),
- sqlBlockRule.getTabletNum(), sqlBlockRule.getCardinality())
- && !isSqlBlockLimitationsNull(sqlBlockRule.getPartitionNum(),
- sqlBlockRule.getTabletNum(), sqlBlockRule.getCardinality())) {
- ErrorReport.reportAnalysisException(ErrorCode.ERROR_SQL_AND_LIMITATIONS_SET_IN_ONE_RULE);
- }
- } else if (!isSqlBlockLimitationsDefault(sqlBlockRule.getPartitionNum(),
- sqlBlockRule.getTabletNum(), sqlBlockRule.getCardinality())) {
- if (!STRING_DEFAULT.equals(sqlBlockRule.getSql()) || !STRING_DEFAULT.equals(sqlBlockRule.getSqlHash())) {
- ErrorReport.reportAnalysisException(ErrorCode.ERROR_SQL_AND_LIMITATIONS_SET_IN_ONE_RULE);
- }
+ checkSqlAndSqlHashSetBoth(sqlBlockRule.getSql(), sqlBlockRule.getSqlHash());
+ boolean hasSqlCondition = hasSqlCondition(sqlBlockRule.getSql(), sqlBlockRule.getSqlHash());
+ boolean hasScanCondition = hasScanCondition(sqlBlockRule.getPartitionNum(), sqlBlockRule.getTabletNum(),
+ sqlBlockRule.getCardinality(), sqlBlockRule.getRequirePartitionFilter());
+ if ((hasSqlCondition && hasScanCondition) || (!hasSqlCondition && !hasScanCondition)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERROR_SQL_AND_LIMITATIONS_SET_IN_ONE_RULE);
}
}
+ private static boolean hasSqlCondition(String sql, String sqlHash) {
+ return isSqlConditionConfigured(sql) || isSqlConditionConfigured(sqlHash);
+ }
+
+ private static boolean hasScanCondition(String partitionNumString, String tabletNumString, String cardinalityString,
+ boolean requirePartitionFilter) {
+ return !isSqlBlockLimitationsEmpty(partitionNumString, tabletNumString, cardinalityString,
+ requirePartitionFilter);
+ }
+
+ private static boolean hasScanCondition(Long partitionNum, Long tabletNum, Long cardinality,
+ Boolean requirePartitionFilter) {
+ return !isSqlBlockLimitationsDefault(partitionNum, tabletNum, cardinality, requirePartitionFilter)
+ && !isSqlBlockLimitationsNull(partitionNum, tabletNum, cardinality, requirePartitionFilter);
+ }
+
+ private static boolean isSqlConditionConfigured(String value) {
+ return StringUtils.isNotEmpty(value) && !STRING_DEFAULT.equals(value);
+ }
+
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/Util.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/Util.java
index ae75e4dd68f..f3822c85af7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/Util.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/Util.java
@@ -426,6 +426,32 @@ public class Util {
}
}
+ public static boolean getBooleanPropertyOrDefault(Map<String, String> properties,
+ String propertyName, boolean defaultValue) throws AnalysisException {
+ if (!properties.containsKey(propertyName)) {
+ return defaultValue;
+ }
+ return parseBooleanProperty(properties.get(propertyName), propertyName);
+ }
+
+ public static Boolean getOptionalBooleanProperty(Map<String, String> properties,
+ String propertyName) throws AnalysisException {
+ if (!properties.containsKey(propertyName)) {
+ return null;
+ }
+ return parseBooleanProperty(properties.get(propertyName), propertyName);
+ }
+
+ public static boolean parseBooleanProperty(String value, String propertyName) throws AnalysisException {
+ if ("true".equalsIgnoreCase(value)) {
+ return true;
+ }
+ if ("false".equalsIgnoreCase(value)) {
+ return false;
+ }
+ throw new AnalysisException(propertyName + " should be a boolean");
+ }
+
// not support encode negative value now
public static void encodeVarint64(long source, DataOutput out) throws IOException {
assert source >= 0;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java
index bed07db31bd..86e8cb968f9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java
@@ -66,7 +66,6 @@ import org.apache.doris.thrift.TTransactionalHiveDesc;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
-import lombok.Setter;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.metastore.api.Table;
import org.apache.logging.log4j.LogManager;
@@ -93,7 +92,6 @@ public class HiveScanNode extends FileQueryScanNode {
private HiveTransaction hiveTransaction = null;
// will only be set in Nereids, for lagency planner, it should be null
- @Setter
protected SelectedPartitions selectedPartitions = null;
private DirectoryLister directoryLister;
@@ -127,6 +125,11 @@ public class HiveScanNode extends FileQueryScanNode {
this.directoryLister = directoryLister;
}
+ public void setSelectedPartitions(SelectedPartitions selectedPartitions) {
+ this.selectedPartitions = selectedPartitions;
+ setHasPartitionPredicate(selectedPartitions != null && selectedPartitions.hasPartitionPredicate);
+ }
+
@Override
protected void doInitialize() throws UserException {
super.doInitialize();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 7c21dc288a7..f54c25db0d9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -877,6 +877,7 @@ public class PhysicalPlanTranslator extends DefaultPlanVisitor<PlanFragment, Pla
scanNode.setNereidsId(fileScan.getId());
context.getNereidsIdToPlanNodeIdMap().put(fileScan.getId(), scanNode.getId());
scanNode.setPushDownAggNoGrouping(context.getRelationPushAggOp(fileScan.getRelationId()));
+ scanNode.setHasPartitionPredicate(fileScan.hasPartitionPredicate());
TableNameInfo tableNameInfo = new TableNameInfo(null, "", "");
TableRefInfo ref = new TableRefInfo(tableNameInfo, null, null);
@@ -1033,6 +1034,7 @@ public class PhysicalPlanTranslator extends DefaultPlanVisitor<PlanFragment, Pla
BaseTableRefInfo tableRefInfo = new BaseTableRefInfo(ref, tableName, olapTable);
tupleDescriptor.setRef(tableRefInfo);
olapScanNode.setSelectedPartitionIds(olapScan.getSelectedPartitionIds());
+ olapScanNode.setHasPartitionPredicate(olapScan.hasPartitionPredicate());
olapScanNode.setNereidsPrunedTabletIds(new LinkedHashSet<>(olapScan.getSelectedTabletIds()));
if (olapScan.getTableSample().isPresent()) {
olapScanNode.setTableSample(new TableSample(olapScan.getTableSample().get().isPercent,
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java
index eea4aeff433..e95e9497920 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java
@@ -36,7 +36,11 @@ import org.apache.doris.nereids.trees.expressions.literal.DateLiteral;
import org.apache.doris.nereids.trees.expressions.literal.DateTimeLiteral;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
import org.apache.doris.nereids.types.DateTimeType;
+import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.ImmutableList;
@@ -47,6 +51,7 @@ import com.google.common.collect.Range;
import com.google.common.collect.RangeSet;
import com.google.common.collect.Sets;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
@@ -67,6 +72,20 @@ public class PartitionPruner extends DefaultExpressionRewriter<Void> {
EXTERNAL
}
+ /** Result of partition prune. */
+ public static final class PartitionPruneResult<K extends Comparable<K>> {
+ public final List<K> partitions;
+ public final Optional<Expression> prunedPartitionPredicate;
+ public final boolean hasPartitionPredicate;
+
+ public PartitionPruneResult(List<K> partitions, Optional<Expression> prunedPartitionPredicate,
+ boolean hasPartitionPredicate) {
+ this.partitions = partitions;
+ this.prunedPartitionPredicate = prunedPartitionPredicate;
+ this.hasPartitionPredicate = hasPartitionPredicate;
+ }
+ }
+
private PartitionPruner(List<OnePartitionEvaluator<?>> partitions, Expression partitionPredicate) {
this.partitions = Objects.requireNonNull(partitions, "partitions cannot be null");
this.partitionPredicate = Objects.requireNonNull(partitionPredicate.accept(this, null),
@@ -128,8 +147,9 @@ public class PartitionPruner extends DefaultExpressionRewriter<Void> {
Expression partitionPredicate,
Map<K, PartitionItem> idToPartitions, CascadesContext cascadesContext,
PartitionTableType partitionTableType) {
- return prune(partitionSlots, partitionPredicate, idToPartitions,
+ PartitionPruneResult<K> result = pruneWithResult(partitionSlots, partitionPredicate, idToPartitions,
cascadesContext, partitionTableType, Optional.empty());
+ return Pair.of(result.partitions, result.prunedPartitionPredicate);
}
/**
@@ -139,6 +159,16 @@ public class PartitionPruner extends DefaultExpressionRewriter<Void> {
Expression partitionPredicate,
Map<K, PartitionItem> idToPartitions, CascadesContext cascadesContext,
PartitionTableType partitionTableType, Optional<SortedPartitionRanges<K>> sortedPartitionRanges) {
+ PartitionPruneResult<K> result = pruneWithResult(partitionSlots, partitionPredicate, idToPartitions,
+ cascadesContext, partitionTableType, sortedPartitionRanges);
+ return Pair.of(result.partitions, result.prunedPartitionPredicate);
+ }
+
+ /** prune partition and return partition predicate info. */
+ public static <K extends Comparable<K>> PartitionPruneResult<K> pruneWithResult(List<Slot> partitionSlots,
+ Expression partitionPredicate,
+ Map<K, PartitionItem> idToPartitions, CascadesContext cascadesContext,
+ PartitionTableType partitionTableType, Optional<SortedPartitionRanges<K>> sortedPartitionRanges) {
long startAt = System.currentTimeMillis();
try {
return pruneInternal(partitionSlots, partitionPredicate, idToPartitions, cascadesContext,
@@ -152,7 +182,7 @@ public class PartitionPruner extends DefaultExpressionRewriter<Void> {
}
}
- private static <K extends Comparable<K>> Pair<List<K>, Optional<Expression>> pruneInternal(
+ private static <K extends Comparable<K>> PartitionPruneResult<K> pruneInternal(
List<Slot> partitionSlots,
Expression partitionPredicate,
Map<K, PartitionItem> idToPartitions, CascadesContext cascadesContext,
@@ -169,10 +199,11 @@ public class PartitionPruner extends DefaultExpressionRewriter<Void> {
partitionPredicate, new ExpressionRewriteContext(cascadesContext));
if (BooleanLiteral.TRUE.equals(partitionPredicate)) {
// The partition column predicate is always true and can be deleted, the partition cannot be pruned
- return Pair.of(Utils.fastToImmutableList(idToPartitions.keySet()), Optional.of(originalPartitionPredicate));
+ return new PartitionPruneResult<>(Utils.fastToImmutableList(idToPartitions.keySet()),
+ Optional.of(originalPartitionPredicate), false);
} else if (BooleanLiteral.FALSE.equals(partitionPredicate) || partitionPredicate.isNullLiteral()) {
// The partition column predicate is always false, and all partitions can be pruned.
- return Pair.of(ImmutableList.of(), Optional.empty());
+ return new PartitionPruneResult<>(ImmutableList.<K>of(), Optional.empty(), true);
}
if (sortedPartitionRanges.isPresent()) {
@@ -183,10 +214,12 @@ public class PartitionPruner extends DefaultExpressionRewriter<Void> {
sortedPartitionRanges.get(), partitionSlots, partitionPredicate, cascadesContext,
expandThreshold, predicateRanges
);
+ boolean hasPartitionPredicate = hasEffectivePartitionPredicate(res.first, idToPartitions.size());
if (res.second) {
- return Pair.of(res.first, Optional.of(originalPartitionPredicate));
+ return new PartitionPruneResult<>(res.first, Optional.of(originalPartitionPredicate),
+ hasPartitionPredicate);
} else {
- return Pair.of(res.first, Optional.empty());
+ return new PartitionPruneResult<>(res.first, Optional.empty(), hasPartitionPredicate);
}
}
}
@@ -194,13 +227,20 @@ public class PartitionPruner extends DefaultExpressionRewriter<Void> {
Pair<List<K>, Boolean> res = sequentialFiltering(
idToPartitions, partitionSlots, partitionPredicate, cascadesContext, expandThreshold
);
+ boolean hasPartitionPredicate = hasEffectivePartitionPredicate(res.first, idToPartitions.size());
if (res.second) {
- return Pair.of(res.first, Optional.of(originalPartitionPredicate));
+ return new PartitionPruneResult<>(res.first, Optional.of(originalPartitionPredicate),
+ hasPartitionPredicate);
} else {
- return Pair.of(res.first, Optional.empty());
+ return new PartitionPruneResult<>(res.first, Optional.empty(), hasPartitionPredicate);
}
}
+ private static <K extends Comparable<K>> boolean hasEffectivePartitionPredicate(
+ List<K> selectedPartitions, int totalPartitions) {
+ return selectedPartitions.size() != totalPartitions;
+ }
+
/**
* convert partition item to partition evaluator
*/
@@ -350,4 +390,22 @@ public class PartitionPruner extends DefaultExpressionRewriter<Void> {
return Pair.of(true, false);
}
}
+
+ /** remove predicates that are always true */
+ public static Plan prunePredicate(boolean skipPrunePredicate, Optional<Expression> prunedPredicates,
+ LogicalFilter<? extends Plan> filter, LogicalRelation scan) {
+ if (!skipPrunePredicate && prunedPredicates.isPresent()) {
+ Set<Expression> conjuncts = new LinkedHashSet<>(filter.getConjuncts());
+ Expression deletedPredicate = prunedPredicates.get();
+ Set<Expression> deletedPredicateSet = ExpressionUtils.extractConjunctionToSet(deletedPredicate);
+ conjuncts.removeAll(deletedPredicateSet);
+ if (conjuncts.isEmpty()) {
+ return scan;
+ } else {
+ return filter.withConjunctsAndChild(conjuncts, scan);
+ }
+ } else {
+ return filter.withChildren(ImmutableList.of(scan));
+ }
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalOlapScanToPhysicalOlapScan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalOlapScanToPhysicalOlapScan.java
index cd69b5ec42e..59a8f41bb64 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalOlapScanToPhysicalOlapScan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalOlapScanToPhysicalOlapScan.java
@@ -54,6 +54,7 @@ public class LogicalOlapScanToPhysicalOlapScan extends OneImplementationRuleFact
olapScan.getSelectedIndexId(),
olapScan.getSelectedTabletIds(),
olapScan.getSelectedPartitionIds(),
+ olapScan.hasPartitionPredicate(),
convertDistribution(olapScan),
olapScan.getPreAggStatus(),
olapScan.getOutputByIndex(olapScan.getTable().getBaseIndexId()),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java
index 35a1105c916..f3822215e8c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java
@@ -18,15 +18,14 @@
package org.apache.doris.nereids.rules.rewrite;
import org.apache.doris.catalog.PartitionItem;
-import org.apache.doris.common.Pair;
import org.apache.doris.datasource.ExternalTable;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.rules.expression.rules.PartitionPruner;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruner.PartitionPruneResult;
import org.apache.doris.nereids.rules.expression.rules.PartitionPruner.PartitionTableType;
import org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges;
-import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions;
@@ -98,14 +97,15 @@ public class PruneFileScanPartition extends OneRewriteRuleFactory {
if (enableBinarySearch) {
sortedPartitionRanges = (Optional) externalTable.getSortedPartitionRanges(scan);
}
- Pair<List<String>, Optional<Expression>> res = PartitionPruner.prune(
+ PartitionPruneResult<String> result = PartitionPruner.pruneWithResult(
partitionSlots, filter.getPredicate(), nameToPartitionItem, ctx,
PartitionTableType.EXTERNAL, sortedPartitionRanges);
- List<String> prunedPartitions = new ArrayList<>(res.first);
+ List<String> prunedPartitions = new ArrayList<>(result.partitions);
for (String name : prunedPartitions) {
selectedPartitionItems.put(name, nameToPartitionItem.get(name));
}
- return new SelectedPartitions(nameToPartitionItem.size(), selectedPartitionItems, true);
+ return new SelectedPartitions(nameToPartitionItem.size(), selectedPartitionItems, true,
+ result.hasPartitionPredicate);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java
index 9d55e1c0c3b..611d7fc78c8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java
@@ -31,6 +31,7 @@ import org.apache.doris.nereids.pattern.MatchingContext;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.rules.expression.rules.PartitionPruner;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruner.PartitionPruneResult;
import org.apache.doris.nereids.rules.expression.rules.PartitionPruner.PartitionTableType;
import org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges;
import org.apache.doris.nereids.trees.expressions.Expression;
@@ -68,11 +69,13 @@ public class PruneOlapScanPartition implements RewriteRuleFactory {
return ImmutableList.of(
logicalOlapScan()
.when(scan -> !scan.isPartitionPruned()
- && !scan.getManuallySpecifiedTabletIds().isEmpty()
+ && (!scan.getManuallySpecifiedTabletIds().isEmpty()
+ || !scan.getManuallySpecifiedPartitions().isEmpty())
&& scan.getTable().isPartitionedTable()
)
.thenApply(ctx -> {
- // Case1: sql without filter condition, e.g. SELECT * FROM tbl (${tabletID})
+ // Case1: sql without filter condition, e.g. SELECT * FROM tbl PARTITION(p1)
+ // or SELECT * FROM tbl TABLET(${tabletID})
LogicalOlapScan scan = ctx.root;
OlapTable table = scan.getTable();
return prunePartition(scan, table, null, ctx).first;
@@ -123,9 +126,9 @@ public class PruneOlapScanPartition implements RewriteRuleFactory {
OlapTable table,
LogicalFilter filter,
MatchingContext ctx) {
- Pair<List<Long>, Optional<Expression>> prunedPartitionsByFilters =
+ PartitionPruneResult<Long> prunedPartitionsByFilters =
prunePartitionByFilters(scan, table, filter, ctx);
- List<Long> prunedPartitions = prunePartitionByTabletIds(scan, table, prunedPartitionsByFilters.first);
+ List<Long> prunedPartitions = prunePartitionByTabletIds(scan, table, prunedPartitionsByFilters.partitions);
if (prunedPartitions == null) {
return Pair.of(null, Optional.empty());
}
@@ -134,36 +137,27 @@ public class PruneOlapScanPartition implements RewriteRuleFactory {
ConnectContext.get().getStatementContext().getNextRelationId(),
ctx.root.getOutput()), Optional.empty());
}
- return Pair.of(scan.withSelectedPartitionIds(prunedPartitions), prunedPartitionsByFilters.second);
+ boolean hasPartitionPredicate = prunedPartitionsByFilters.hasPartitionPredicate
+ || !scan.getManuallySpecifiedPartitions().isEmpty()
+ || !scan.getManuallySpecifiedTabletIds().isEmpty();
+ return Pair.of(scan.withSelectedPartitionIds(prunedPartitions,
+ hasPartitionPredicate),
+ prunedPartitionsByFilters.prunedPartitionPredicate);
}
- private Pair<List<Long>, Optional<Expression>> prunePartitionByFilters(LogicalOlapScan scan,
+ private PartitionPruneResult<Long> prunePartitionByFilters(LogicalOlapScan scan,
OlapTable table,
LogicalFilter filter,
MatchingContext ctx) {
Set<String> partitionColumnNameSet = Utils.execWithReturnVal(table::getPartitionColumnNames);
if (partitionColumnNameSet.isEmpty()) {
- return Pair.of(null, Optional.empty());
+ return new PartitionPruneResult<>(null, Optional.empty(), false);
}
- List<Slot> output = scan.getOutput();
- PartitionInfo partitionInfo = table.getPartitionInfo();
- List<Column> partitionColumns = partitionInfo.getPartitionColumns();
- List<Slot> partitionSlots = new ArrayList<>(partitionColumns.size());
- for (Column column : partitionColumns) {
- Slot partitionSlot = null;
- // loop search is faster than build a map
- for (Slot slot : output) {
- if (slot.getName().equalsIgnoreCase(column.getName())) {
- partitionSlot = slot;
- break;
- }
- }
- if (partitionSlot == null) {
- return Pair.of(null, Optional.empty());
- } else {
- partitionSlots.add(partitionSlot);
- }
+ List<Slot> partitionSlots = getPartitionSlots(scan, table);
+ if (partitionSlots == null) {
+ return new PartitionPruneResult<>(null, Optional.empty(), false);
}
+ PartitionInfo partitionInfo = table.getPartitionInfo();
NereidsSortedPartitionsCacheManager sortedPartitionsCacheManager = Env.getCurrentEnv()
.getSortedPartitionsCacheManager();
List<Long> manuallySpecifiedPartitions = scan.getManuallySpecifiedPartitions();
@@ -183,14 +177,14 @@ public class PruneOlapScanPartition implements RewriteRuleFactory {
.collect(Collectors.toMap(Function.identity(), allPartitions::get));
}
if (filter != null) {
- Pair<List<Long>, Optional<Expression>> prunedPartitions = PartitionPruner.prune(
+ return PartitionPruner.pruneWithResult(
partitionSlots, filter.getPredicate(), idToPartitions, ctx.cascadesContext,
PartitionTableType.OLAP, sortedPartitionRanges);
- return prunedPartitions;
} else if (!manuallySpecifiedPartitions.isEmpty()) {
- return Pair.of(Utils.fastToImmutableList(idToPartitions.keySet()), Optional.empty());
+ return new PartitionPruneResult<>(Utils.fastToImmutableList(idToPartitions.keySet()),
+ Optional.empty(), true);
} else {
- return Pair.of(null, Optional.empty());
+ return new PartitionPruneResult<>(null, Optional.empty(), false);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterSqlBlockRuleCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterSqlBlockRuleCommand.java
index 8357a44d347..3b3635dc9b7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterSqlBlockRuleCommand.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterSqlBlockRuleCommand.java
@@ -28,8 +28,6 @@ import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.StmtExecutor;
-import org.apache.commons.lang3.StringUtils;
-
import java.util.Map;
/**
@@ -50,7 +48,7 @@ public class AlterSqlBlockRuleCommand extends SqlBlockRuleCommand {
public void doRun(ConnectContext ctx, StmtExecutor executor) throws Exception {
Env.getCurrentEnv().getSqlBlockRuleMgr().alterSqlBlockRule(new SqlBlockRule(ruleName,
sql, sqlHash, partitionNum,
- tabletNum, cardinality, global, enable));
+ tabletNum, cardinality, requirePartitionFilter, global, enable));
}
@Override
@@ -70,10 +68,13 @@ public class AlterSqlBlockRuleCommand extends SqlBlockRuleCommand {
String partitionNumString = properties.get(SCANNED_PARTITION_NUM);
String tabletNumString = properties.get(SCANNED_TABLET_NUM);
String cardinalityString = properties.get(SCANNED_CARDINALITY);
+ this.requirePartitionFilter = Util.getOptionalBooleanProperty(
+ properties, REQUIRE_PARTITION_FILTER_PROPERTY);
SqlBlockUtil.checkSqlAndSqlHashSetBoth(sql, sqlHash);
SqlBlockUtil.checkSqlAndLimitationsSetBoth(sql, sqlHash,
- partitionNumString, tabletNumString, cardinalityString);
+ partitionNumString, tabletNumString, cardinalityString,
+ Boolean.TRUE.equals(requirePartitionFilter));
this.partitionNum = Util.getLongPropertyOrDefault(partitionNumString, SqlBlockRuleCommand.LONG_NOT_SET, null,
SCANNED_PARTITION_NUM + " should be a long");
this.tabletNum = Util.getLongPropertyOrDefault(tabletNumString, SqlBlockRuleCommand.LONG_NOT_SET, null,
@@ -81,9 +82,7 @@ public class AlterSqlBlockRuleCommand extends SqlBlockRuleCommand {
this.cardinality = Util.getLongPropertyOrDefault(cardinalityString, SqlBlockRuleCommand.LONG_NOT_SET, null,
SCANNED_CARDINALITY + " should be a long");
// allow null, represents no modification
- String globalStr = properties.get(GLOBAL_PROPERTY);
- this.global = StringUtils.isNotEmpty(globalStr) ? Boolean.parseBoolean(globalStr) : null;
- String enableStr = properties.get(ENABLE_PROPERTY);
- this.enable = StringUtils.isNotEmpty(enableStr) ? Boolean.parseBoolean(enableStr) : null;
+ this.global = Util.getOptionalBooleanProperty(properties, GLOBAL_PROPERTY);
+ this.enable = Util.getOptionalBooleanProperty(properties, ENABLE_PROPERTY);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CreateSqlBlockRuleCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CreateSqlBlockRuleCommand.java
index d43fa694f29..47e83d64ebd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CreateSqlBlockRuleCommand.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CreateSqlBlockRuleCommand.java
@@ -62,7 +62,7 @@ public class CreateSqlBlockRuleCommand extends SqlBlockRuleCommand {
}
Env.getCurrentEnv().getSqlBlockRuleMgr().createSqlBlockRule(new SqlBlockRule(ruleName, sql,
sqlHash, partitionNum,
- tabletNum, cardinality, global, enable), ifNotExists);
+ tabletNum, cardinality, requirePartitionFilter, global, enable), ifNotExists);
}
@Override
@@ -72,9 +72,12 @@ public class CreateSqlBlockRuleCommand extends SqlBlockRuleCommand {
String partitionNumString = properties.get(SCANNED_PARTITION_NUM);
String tabletNumString = properties.get(SCANNED_TABLET_NUM);
String cardinalityString = properties.get(SCANNED_CARDINALITY);
+ this.requirePartitionFilter = Util.getBooleanPropertyOrDefault(
+ properties, REQUIRE_PARTITION_FILTER_PROPERTY, false);
SqlBlockUtil.checkSqlAndSqlHashSetBoth(sql, sqlHash);
- SqlBlockUtil.checkPropertiesValidate(sql, sqlHash, partitionNumString, tabletNumString, cardinalityString);
+ SqlBlockUtil.checkPropertiesValidate(sql, sqlHash, partitionNumString, tabletNumString,
+ cardinalityString, requirePartitionFilter);
this.partitionNum = Util.getLongPropertyOrDefault(partitionNumString, 0L, null,
SCANNED_PARTITION_NUM + " should be a long");
@@ -83,10 +86,8 @@ public class CreateSqlBlockRuleCommand extends SqlBlockRuleCommand {
this.cardinality = Util.getLongPropertyOrDefault(cardinalityString, 0L, null,
SCANNED_CARDINALITY + " should be a long");
- this.global = Util.getBooleanPropertyOrDefault(properties.get(GLOBAL_PROPERTY), false,
- GLOBAL_PROPERTY + " should be a boolean");
- this.enable = Util.getBooleanPropertyOrDefault(properties.get(ENABLE_PROPERTY), true,
- ENABLE_PROPERTY + " should be a boolean");
+ this.global = Util.getBooleanPropertyOrDefault(properties, GLOBAL_PROPERTY, false);
+ this.enable = Util.getBooleanPropertyOrDefault(properties, ENABLE_PROPERTY, true);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowSqlBlockRuleCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowSqlBlockRuleCommand.java
index 03c7d55dffa..9d6b75678c8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowSqlBlockRuleCommand.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowSqlBlockRuleCommand.java
@@ -20,6 +20,7 @@ package org.apache.doris.nereids.trees.plans.commands;
import org.apache.doris.blockrule.SqlBlockRule;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
@@ -49,6 +50,7 @@ public class ShowSqlBlockRuleCommand extends ShowCommand {
.addColumn(new Column("Cardinality", ScalarType.createVarchar(20)))
.addColumn(new Column("Global", ScalarType.createVarchar(4)))
.addColumn(new Column("Enable", ScalarType.createVarchar(4)))
+ .addColumn(new Column("RequirePartitionFilter", ScalarType.createType(PrimitiveType.BOOLEAN)))
.build();
private final String ruleName; // optional
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/SqlBlockRuleCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/SqlBlockRuleCommand.java
index 00804faede0..e6a4f2ff7f9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/SqlBlockRuleCommand.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/SqlBlockRuleCommand.java
@@ -51,13 +51,16 @@ public abstract class SqlBlockRuleCommand extends Command implements ForwardWith
public static final String ENABLE_PROPERTY = "enable";
+ public static final String REQUIRE_PARTITION_FILTER_PROPERTY = "require_partition_filter";
+
public static final Long LONG_NOT_SET = SqlBlockUtil.LONG_MINUS_ONE;
public static final String STRING_NOT_SET = SqlBlockUtil.STRING_DEFAULT;
private static final ImmutableSet<String> PROPERTIES_SET = new ImmutableSet.Builder<String>().add(SQL_PROPERTY)
.add(SQL_HASH_PROPERTY).add(GLOBAL_PROPERTY).add(ENABLE_PROPERTY).add(SCANNED_PARTITION_NUM)
- .add(SCANNED_TABLET_NUM).add(SCANNED_CARDINALITY).build();
+ .add(SCANNED_TABLET_NUM).add(SCANNED_CARDINALITY)
+ .add(REQUIRE_PARTITION_FILTER_PROPERTY).build();
protected final String ruleName;
@@ -77,6 +80,9 @@ public abstract class SqlBlockRuleCommand extends Command implements ForwardWith
// whether to use the rule, default is true
protected Boolean enable;
+ // whether partitioned table scans require partition filters
+ protected Boolean requirePartitionFilter;
+
protected final Map<String, String> properties;
/**
@@ -145,5 +151,8 @@ public abstract class SqlBlockRuleCommand extends Command implements ForwardWith
public Boolean getEnable() {
return enable;
}
-}
+ public Boolean getRequirePartitionFilter() {
+ return requirePartitionFilter;
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java
index f2b1a9b226f..3039f991376 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java
@@ -100,6 +100,10 @@ public class LogicalFileScan extends LogicalCatalogRelation implements SupportPr
return selectedPartitions;
}
+ public boolean hasPartitionPredicate() {
+ return selectedPartitions.hasPartitionPredicate;
+ }
+
public Optional<TableSample> getTableSample() {
return tableSample;
}
@@ -238,7 +242,7 @@ public class LogicalFileScan extends LogicalCatalogRelation implements SupportPr
// NOT_PRUNED means the Nereids planner does not handle the partition pruning.
// This can be treated as the initial value of SelectedPartitions.
// Or used to indicate that the partition pruning is not processed.
- public static SelectedPartitions NOT_PRUNED = new SelectedPartitions(0, ImmutableMap.of(), false);
+ public static SelectedPartitions NOT_PRUNED = new SelectedPartitions(0, ImmutableMap.of(), false, false);
/**
* total partition number
*/
@@ -253,15 +257,29 @@ public class LogicalFileScan extends LogicalCatalogRelation implements SupportPr
*/
public final boolean isPruned;
+ /**
+ * true means the pruning logic found a usable partition predicate.
+ */
+ public final boolean hasPartitionPredicate;
+
/**
* Constructor for SelectedPartitions.
*/
public SelectedPartitions(long totalPartitionNum, Map<String, PartitionItem> selectedPartitions,
boolean isPruned) {
+ this(totalPartitionNum, selectedPartitions, isPruned, false);
+ }
+
+ /**
+ * Constructor for SelectedPartitions.
+ */
+ public SelectedPartitions(long totalPartitionNum, Map<String, PartitionItem> selectedPartitions,
+ boolean isPruned, boolean hasPartitionPredicate) {
this.totalPartitionNum = totalPartitionNum;
this.selectedPartitions = ImmutableMap.copyOf(Objects.requireNonNull(selectedPartitions,
"selectedPartitions is null"));
this.isPruned = isPruned;
+ this.hasPartitionPredicate = hasPartitionPredicate;
}
@Override
@@ -273,13 +291,15 @@ public class LogicalFileScan extends LogicalCatalogRelation implements SupportPr
return false;
}
SelectedPartitions that = (SelectedPartitions) o;
- return isPruned == that.isPruned && Objects.equals(
+ return isPruned == that.isPruned
+ && hasPartitionPredicate == that.hasPartitionPredicate
+ && Objects.equals(
selectedPartitions.keySet(), that.selectedPartitions.keySet());
}
@Override
public int hashCode() {
- return Objects.hash(selectedPartitions, isPruned);
+ return Objects.hash(selectedPartitions, isPruned, hasPartitionPredicate);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java
index 6836ac78b4c..50341f62410 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java
@@ -136,6 +136,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
* todo: should be pulled up to base class?
*/
private final boolean partitionPruned;
+ private final boolean hasPartitionPredicate;
private final List<Long> manuallySpecifiedPartitions;
private final List<Long> selectedPartitionIds;
@@ -179,12 +180,13 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
*/
public LogicalOlapScan(RelationId id, OlapTable table, List<String> qualifier) {
this(id, table, qualifier, Optional.empty(), Optional.empty(),
- table.getPartitionIds(), false,
+ table.getPartitionIds(), false, false,
ImmutableList.of(),
-1, false, PreAggStatus.unset(), ImmutableList.of(), ImmutableList.of(),
Maps.newHashMap(), Optional.empty(), Optional.empty(), false, ImmutableMap.of(),
ImmutableList.of(), ImmutableList.of(), ImmutableList.of(),
- ImmutableList.of(), Optional.empty(), Optional.empty(), ImmutableList.of(), Optional.empty());
+ ImmutableList.of(), Optional.empty(), Optional.empty(), ImmutableList.of(), Optional.empty(),
+ Optional.empty());
}
/**
@@ -193,11 +195,11 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public LogicalOlapScan(RelationId id, OlapTable table, List<String> qualifier, List<Long> tabletIds,
List<String> hints, Optional<TableSample> tableSample, Collection<Slot> operativeSlots) {
this(id, table, qualifier, Optional.empty(), Optional.empty(),
- table.getPartitionIds(), false, tabletIds,
+ table.getPartitionIds(), false, false, tabletIds,
-1, false, PreAggStatus.unset(), ImmutableList.of(), hints, Maps.newHashMap(), Optional.empty(),
tableSample, false, ImmutableMap.of(), ImmutableList.of(), operativeSlots,
ImmutableList.of(), ImmutableList.of(), Optional.empty(), Optional.empty(),
- ImmutableList.of(), Optional.empty());
+ ImmutableList.of(), Optional.empty(), Optional.empty());
}
/**
@@ -207,11 +209,11 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
List<Long> tabletIds, List<String> hints, Optional<TableSample> tableSample, List<Slot> operativeSlots) {
this(id, table, qualifier, Optional.empty(), Optional.empty(),
// must use specifiedPartitions here for prune partition by sql like 'select * from t partition p1'
- specifiedPartitions, false, tabletIds,
+ specifiedPartitions, false, false, tabletIds,
-1, false, PreAggStatus.unset(), specifiedPartitions, hints, Maps.newHashMap(), Optional.empty(),
tableSample, false, ImmutableMap.of(), ImmutableList.of(), operativeSlots,
ImmutableList.of(), ImmutableList.of(), Optional.empty(), Optional.empty(),
- ImmutableList.of(), Optional.empty());
+ ImmutableList.of(), Optional.empty(), Optional.empty());
}
/**
@@ -222,12 +224,12 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
List<Long> specifiedPartitions, List<String> hints, Optional<TableSample> tableSample,
Collection<Slot> operativeSlots) {
this(id, table, qualifier, Optional.empty(), Optional.empty(),
- selectedPartitionIds, false, tabletIds,
+ selectedPartitionIds, false, false, tabletIds,
selectedIndexId, true, preAggStatus,
specifiedPartitions, hints, Maps.newHashMap(), Optional.empty(), tableSample, true, ImmutableMap.of(),
ImmutableList.of(), operativeSlots, ImmutableList.of(), ImmutableList.of(),
Optional.empty(), Optional.empty(),
- ImmutableList.of(), Optional.empty());
+ ImmutableList.of(), Optional.empty(), Optional.empty());
}
/**
@@ -245,7 +247,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
List<OrderKey> scoreOrderKeys, Optional<Long> scoreLimit, Optional<ScoreRangeInfo> scoreRangeInfo,
List<OrderKey> annOrderKeys, Optional<Long> annLimit) {
this(id, table, qualifier, groupExpression, logicalProperties, selectedPartitionIds, partitionPruned,
- selectedTabletIds, selectedIndexId, indexSelected, preAggStatus,
+ false, selectedTabletIds, selectedIndexId, indexSelected, preAggStatus,
specifiedPartitions, hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, specifiedTabletIds, operativeSlots, virtualColumns,
scoreOrderKeys, scoreLimit, scoreRangeInfo, annOrderKeys, annLimit,
@@ -257,7 +259,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
*/
public LogicalOlapScan(RelationId id, Table table, List<String> qualifier,
Optional<GroupExpression> groupExpression, Optional<LogicalProperties> logicalProperties,
- List<Long> selectedPartitionIds, boolean partitionPruned,
+ List<Long> selectedPartitionIds, boolean partitionPruned, boolean hasPartitionPredicate,
List<Long> selectedTabletIds, long selectedIndexId, boolean indexSelected,
PreAggStatus preAggStatus, List<Long> specifiedPartitions,
List<String> hints, Map<Pair<Long, String>, Slot> cacheSlotWithSlotName,
@@ -273,6 +275,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
"selectedPartitionIds can not be null");
this.selectedTabletIds = Utils.fastToImmutableList(selectedTabletIds);
this.partitionPruned = partitionPruned;
+ this.hasPartitionPredicate = hasPartitionPredicate;
this.selectedIndexId = selectedIndexId <= 0 ? getTable().getBaseIndexId() : selectedIndexId;
this.indexSelected = indexSelected;
this.preAggStatus = preAggStatus;
@@ -327,7 +330,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
Optional<PartitionPrunablePredicate> partitionPrunablePredicates) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
groupExpression, Optional.of(getLogicalProperties()),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, manuallySpecifiedTabletIds, operativeSlots, virtualColumns,
@@ -335,6 +338,10 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
partitionPrunablePredicates);
}
+ public boolean hasPartitionPredicate() {
+ return hasPartitionPredicate;
+ }
+
@Override
public String getFingerprint() {
String partitions = "";
@@ -388,7 +395,9 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
}
LogicalOlapScan that = (LogicalOlapScan) o;
return selectedIndexId == that.selectedIndexId && indexSelected == that.indexSelected
- && partitionPruned == that.partitionPruned && Objects.equals(preAggStatus, that.preAggStatus)
+ && partitionPruned == that.partitionPruned
+ && hasPartitionPredicate == that.hasPartitionPredicate
+ && Objects.equals(preAggStatus, that.preAggStatus)
&& Objects.equals(selectedTabletIds, that.selectedTabletIds)
&& Objects.equals(manuallySpecifiedPartitions, that.manuallySpecifiedPartitions)
&& Objects.equals(manuallySpecifiedTabletIds, that.manuallySpecifiedTabletIds)
@@ -412,7 +421,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public LogicalOlapScan withGroupExpression(Optional<GroupExpression> groupExpression) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
groupExpression, Optional.of(getLogicalProperties()),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, manuallySpecifiedTabletIds, operativeSlots, virtualColumns,
@@ -423,7 +432,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public Plan withGroupExprLogicalPropChildren(Optional<GroupExpression> groupExpression,
Optional<LogicalProperties> logicalProperties, List<Plan> children) {
return new LogicalOlapScan(relationId, (Table) table, qualifier, groupExpression, logicalProperties,
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, manuallySpecifiedTabletIds, operativeSlots, virtualColumns,
@@ -434,9 +443,17 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
* withSelectedPartitionIds
*/
public LogicalOlapScan withSelectedPartitionIds(List<Long> selectedPartitionIds) {
+ return withSelectedPartitionIds(selectedPartitionIds, false);
+ }
+
+ /**
+ * withSelectedPartitionIds
+ */
+ public LogicalOlapScan withSelectedPartitionIds(List<Long> selectedPartitionIds,
+ boolean hasPartitionPredicate) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
Optional.empty(), Optional.of(getLogicalProperties()),
- selectedPartitionIds, true, selectedTabletIds,
+ selectedPartitionIds, true, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, manuallySpecifiedTabletIds, operativeSlots, virtualColumns,
@@ -451,7 +468,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public LogicalOlapScan withMaterializedIndexSelected(long indexId) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
Optional.empty(), Optional.of(getLogicalProperties()),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
indexId, true, PreAggStatus.unset(), manuallySpecifiedPartitions, hints, cacheSlotWithSlotName,
cachedOutput, tableSample, directMvScan, colToSubPathsMap, manuallySpecifiedTabletIds,
operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit, scoreRangeInfo,
@@ -464,7 +481,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public LogicalOlapScan withSelectedTabletIds(List<Long> selectedTabletIds) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
Optional.empty(), Optional.of(getLogicalProperties()),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, manuallySpecifiedTabletIds, operativeSlots, virtualColumns, scoreOrderKeys,
@@ -477,7 +494,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public LogicalOlapScan withPreAggStatus(PreAggStatus preAggStatus) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
Optional.empty(), Optional.of(getLogicalProperties()),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, manuallySpecifiedTabletIds, operativeSlots, virtualColumns,
@@ -490,7 +507,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public LogicalOlapScan withColToSubPathsMap(Map<String, Set<List<String>>> colToSubPathsMap) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
Optional.empty(), Optional.empty(),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, manuallySpecifiedTabletIds, operativeSlots, virtualColumns,
@@ -503,7 +520,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public LogicalOlapScan withManuallySpecifiedTabletIds(List<Long> manuallySpecifiedTabletIds) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
Optional.empty(), Optional.of(getLogicalProperties()),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan,
colToSubPathsMap, manuallySpecifiedTabletIds, operativeSlots, virtualColumns,
@@ -515,7 +532,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
// we have to set partitionPruned to false, so that mtmv rewrite can prevent deadlock when rewriting union
return new LogicalOlapScan(relationId, (Table) table, qualifier,
Optional.empty(), Optional.empty(),
- selectedPartitionIds, false, selectedTabletIds,
+ selectedPartitionIds, false, false, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, Maps.newHashMap(), Optional.empty(), tableSample, directMvScan,
colToSubPathsMap, selectedTabletIds, operativeSlots, virtualColumns, scoreOrderKeys,
@@ -535,7 +552,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
logicalProperties = new LogicalProperties(() -> output, this::computeDataTrait);
return new LogicalOlapScan(relationId, (Table) table, qualifier,
groupExpression, Optional.of(logicalProperties),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan, colToSubPathsMap,
manuallySpecifiedTabletIds, operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit,
@@ -558,7 +575,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
.build();
return new LogicalOlapScan(relationId, (Table) table, qualifier,
groupExpression, Optional.of(logicalProperties),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan, colToSubPathsMap,
manuallySpecifiedTabletIds, operativeSlots, mergedVirtualColumns, scoreOrderKeys, scoreLimit,
@@ -587,7 +604,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
.build();
return new LogicalOlapScan(relationId, (Table) table, qualifier,
groupExpression, Optional.of(logicalProperties),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan, colToSubPathsMap,
manuallySpecifiedTabletIds, operativeSlots, mergedVirtualColumns, scoreOrderKeys, scoreLimit,
@@ -933,7 +950,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public CatalogRelation withOperativeSlots(Collection<Slot> operativeSlots) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
groupExpression, Optional.of(getLogicalProperties()),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, cachedOutput, tableSample, directMvScan, colToSubPathsMap,
manuallySpecifiedTabletIds, operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit,
@@ -1014,7 +1031,7 @@ public class LogicalOlapScan extends LogicalCatalogRelation implements OlapScan,
public LogicalOlapScan withCachedOutput(List<Slot> outputSlots) {
return new LogicalOlapScan(relationId, (Table) table, qualifier,
groupExpression, Optional.empty(),
- selectedPartitionIds, partitionPruned, selectedTabletIds,
+ selectedPartitionIds, partitionPruned, hasPartitionPredicate, selectedTabletIds,
selectedIndexId, indexSelected, preAggStatus, manuallySpecifiedPartitions,
hints, cacheSlotWithSlotName, Optional.of(outputSlots), tableSample, directMvScan, colToSubPathsMap,
manuallySpecifiedTabletIds, operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit,
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalFileScan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalFileScan.java
index e592051e18c..570a22a6b21 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalFileScan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalFileScan.java
@@ -118,6 +118,10 @@ public class PhysicalFileScan extends PhysicalCatalogRelation {
return selectedPartitions;
}
+ public boolean hasPartitionPredicate() {
+ return selectedPartitions.hasPartitionPredicate;
+ }
+
public Optional<TableSample> getTableSample() {
return tableSample;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapScan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapScan.java
index 5644ca25e5b..4ffc49eb627 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapScan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapScan.java
@@ -62,6 +62,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
private final long selectedIndexId;
private final ImmutableList<Long> selectedTabletIds;
private final ImmutableList<Long> selectedPartitionIds;
+ private final boolean hasPartitionPredicate;
private final PreAggStatus preAggStatus;
private final List<Slot> baseOutputs;
private final Optional<TableSample> tableSample;
@@ -97,11 +98,30 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
List<OrderKey> scoreOrderKeys, Optional<Long> scoreLimit, Optional<ScoreRangeInfo> scoreRangeInfo,
List<OrderKey> annOrderKeys, Optional<Long> annLimit) {
this(id, olapTable, qualifier,
- selectedIndexId, selectedTabletIds, selectedPartitionIds, distributionSpec,
+ selectedIndexId, selectedTabletIds, selectedPartitionIds, false, distributionSpec,
preAggStatus, baseOutputs,
groupExpression, logicalProperties, null,
null, tableSample, operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit,
- scoreRangeInfo, annOrderKeys, annLimit);
+ scoreRangeInfo, annOrderKeys, annLimit, Optional.empty());
+ }
+
+ /**
+ * Constructor for PhysicalOlapScan.
+ */
+ public PhysicalOlapScan(RelationId id, OlapTable olapTable, List<String> qualifier, long selectedIndexId,
+ List<Long> selectedTabletIds, List<Long> selectedPartitionIds, boolean hasPartitionPredicate,
+ DistributionSpec distributionSpec, PreAggStatus preAggStatus, List<Slot> baseOutputs,
+ Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ Optional<TableSample> tableSample, List<Slot> operativeSlots, List<NamedExpression> virtualColumns,
+ List<OrderKey> scoreOrderKeys, Optional<Long> scoreLimit, Optional<ScoreRangeInfo> scoreRangeInfo,
+ List<OrderKey> annOrderKeys, Optional<Long> annLimit,
+ Optional<PartitionPrunablePredicate> partitionPrunablePredicates) {
+ this(id, olapTable, qualifier,
+ selectedIndexId, selectedTabletIds, selectedPartitionIds, hasPartitionPredicate, distributionSpec,
+ preAggStatus, baseOutputs,
+ groupExpression, logicalProperties, null,
+ null, tableSample, operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit,
+ scoreRangeInfo, annOrderKeys, annLimit, partitionPrunablePredicates);
}
/**
@@ -116,7 +136,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
List<OrderKey> annOrderKeys, Optional<Long> annLimit,
Optional<PartitionPrunablePredicate> partitionPrunablePredicates) {
this(id, olapTable, qualifier,
- selectedIndexId, selectedTabletIds, selectedPartitionIds, distributionSpec,
+ selectedIndexId, selectedTabletIds, selectedPartitionIds, false, distributionSpec,
preAggStatus, baseOutputs,
groupExpression, logicalProperties, null,
null, tableSample, operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit,
@@ -135,7 +155,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
Collection<Slot> operativeSlots, List<NamedExpression> virtualColumns,
List<OrderKey> scoreOrderKeys, Optional<Long> scoreLimit, Optional<ScoreRangeInfo> scoreRangeInfo,
List<OrderKey> annOrderKeys, Optional<Long> annLimit) {
- this(id, olapTable, qualifier, selectedIndexId, selectedTabletIds, selectedPartitionIds,
+ this(id, olapTable, qualifier, selectedIndexId, selectedTabletIds, selectedPartitionIds, false,
distributionSpec, preAggStatus, baseOutputs, groupExpression,
logicalProperties, physicalProperties, statistics, tableSample, operativeSlots, virtualColumns,
scoreOrderKeys, scoreLimit, scoreRangeInfo, annOrderKeys, annLimit,
@@ -146,7 +166,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
* Ultimate constructor for PhysicalOlapScan.
*/
public PhysicalOlapScan(RelationId id, OlapTable olapTable, List<String> qualifier, long selectedIndexId,
- List<Long> selectedTabletIds, List<Long> selectedPartitionIds,
+ List<Long> selectedTabletIds, List<Long> selectedPartitionIds, boolean hasPartitionPredicate,
DistributionSpec distributionSpec, PreAggStatus preAggStatus, List<Slot> baseOutputs,
Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
PhysicalProperties physicalProperties, Statistics statistics,
@@ -160,6 +180,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
this.selectedIndexId = selectedIndexId;
this.selectedTabletIds = ImmutableList.copyOf(selectedTabletIds);
this.selectedPartitionIds = ImmutableList.copyOf(selectedPartitionIds);
+ this.hasPartitionPredicate = hasPartitionPredicate;
this.distributionSpec = distributionSpec;
this.preAggStatus = preAggStatus;
this.baseOutputs = ImmutableList.copyOf(baseOutputs);
@@ -194,6 +215,10 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
return partitionPrunablePredicates;
}
+ public boolean hasPartitionPredicate() {
+ return hasPartitionPredicate;
+ }
+
@Override
public OlapTable getTable() {
return (OlapTable) table;
@@ -310,6 +335,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
}
PhysicalOlapScan olapScan = (PhysicalOlapScan) o;
return selectedIndexId == olapScan.selectedIndexId
+ && hasPartitionPredicate == olapScan.hasPartitionPredicate
&& Objects.equals(distributionSpec, olapScan.distributionSpec)
&& Objects.equals(selectedTabletIds, olapScan.selectedTabletIds)
&& Objects.equals(selectedPartitionIds, olapScan.selectedPartitionIds)
@@ -339,7 +365,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
@Override
public PhysicalOlapScan withGroupExpression(Optional<GroupExpression> groupExpression) {
return new PhysicalOlapScan(relationId, getTable(), qualifier,
- selectedIndexId, selectedTabletIds, selectedPartitionIds,
+ selectedIndexId, selectedTabletIds, selectedPartitionIds, hasPartitionPredicate,
distributionSpec, preAggStatus, baseOutputs, groupExpression, getLogicalProperties(), null, null,
tableSample, operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit, scoreRangeInfo,
annOrderKeys, annLimit, partitionPrunablePredicates);
@@ -349,7 +375,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
public Plan withGroupExprLogicalPropChildren(Optional<GroupExpression> groupExpression,
Optional<LogicalProperties> logicalProperties, List<Plan> children) {
return new PhysicalOlapScan(relationId, getTable(), qualifier,
- selectedIndexId, selectedTabletIds, selectedPartitionIds,
+ selectedIndexId, selectedTabletIds, selectedPartitionIds, hasPartitionPredicate,
distributionSpec, preAggStatus, baseOutputs, groupExpression, logicalProperties.get(), null, null,
tableSample, operativeSlots, virtualColumns, scoreOrderKeys, scoreLimit, scoreRangeInfo,
annOrderKeys, annLimit, partitionPrunablePredicates);
@@ -359,7 +385,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
public PhysicalOlapScan withPhysicalPropertiesAndStats(
PhysicalProperties physicalProperties, Statistics statistics) {
return new PhysicalOlapScan(relationId, getTable(), qualifier,
- selectedIndexId, selectedTabletIds, selectedPartitionIds,
+ selectedIndexId, selectedTabletIds, selectedPartitionIds, hasPartitionPredicate,
distributionSpec, preAggStatus, baseOutputs, groupExpression, getLogicalProperties(),
physicalProperties, statistics, tableSample, operativeSlots, virtualColumns, scoreOrderKeys,
scoreLimit, scoreRangeInfo, annOrderKeys, annLimit, partitionPrunablePredicates);
@@ -386,7 +412,7 @@ public class PhysicalOlapScan extends PhysicalCatalogRelation implements OlapSca
@Override
public CatalogRelation withOperativeSlots(Collection<Slot> operativeSlots) {
return new PhysicalOlapScan(relationId, (OlapTable) table, qualifier,
- selectedIndexId, selectedTabletIds, selectedPartitionIds,
+ selectedIndexId, selectedTabletIds, selectedPartitionIds, hasPartitionPredicate,
distributionSpec, preAggStatus, baseOutputs, groupExpression, getLogicalProperties(),
getPhysicalProperties(), statistics, tableSample, operativeSlots, virtualColumns, scoreOrderKeys,
scoreLimit,
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
index 3f62ccdd3bd..5e776bd6e85 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
@@ -732,8 +732,11 @@ public class OlapScanNode extends ScanNode {
PartitionNamesInfo partitionNames = desc.getRef().getPartitionNamesInfo();
PartitionInfo partitionInfo = olapTable.getPartitionInfo();
if (partitionInfo.getType() == PartitionType.RANGE || partitionInfo.getType() == PartitionType.LIST) {
+ setHasPartitionPredicate(ScanNode.containsPartitionPredicate(
+ partitionInfo.getPartitionColumns(), desc, conjuncts, partitionInfo));
selectedPartitionIds = partitionPrune(partitionInfo, partitionNames);
} else {
+ setHasPartitionPredicate(false);
selectedPartitionIds = olapTable.getPartitionIds();
}
selectedPartitionIds = olapTable.selectNonEmptyPartitionIds(selectedPartitionIds);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
index 9ecdb14e90b..858ab311e84 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
@@ -99,6 +99,7 @@ public abstract class ScanNode extends PlanNode implements SplitGenerator {
protected long selectedPartitionNum = 0;
protected int selectedSplitNum = 0;
+ private boolean hasPartitionPredicate = false;
// create a mapping between output slot's id and project expr
Map<SlotId, Expr> outputSlotToProjectExpr = new HashMap<>();
@@ -206,6 +207,33 @@ public abstract class ScanNode extends PlanNode implements SplitGenerator {
return desc.getTable();
}
+ public boolean isPartitionedTable() {
+ return getTableIf() != null && getTableIf().isPartitionedTable();
+ }
+
+ public boolean hasPartitionPredicate() {
+ return hasPartitionPredicate;
+ }
+
+ public void setHasPartitionPredicate(boolean hasPartitionPredicate) {
+ this.hasPartitionPredicate = hasPartitionPredicate;
+ }
+
+ static boolean containsPartitionPredicate(List<Column> partitionColumns, TupleDescriptor tupleDescriptor,
+ List<Expr> conjuncts, PartitionInfo partitionInfo) {
+ for (Column partitionColumn : partitionColumns) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getColumnSlot(partitionColumn.getName());
+ if (slotDescriptor == null) {
+ continue;
+ }
+ if (createPartitionFilter(slotDescriptor, conjuncts, partitionInfo) != null
+ || createColumnRange(slotDescriptor, conjuncts, partitionInfo).hasFilter()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
public static ColumnRange createColumnRange(SlotDescriptor desc,
List<Expr> conjuncts, PartitionInfo partitionsInfo) {
ColumnRange result = ColumnRange.create();
@@ -353,7 +381,7 @@ public abstract class ScanNode extends PlanNode implements SplitGenerator {
}
}
- private PartitionColumnFilter createPartitionFilter(SlotDescriptor desc, List<Expr> conjuncts,
+ protected static PartitionColumnFilter createPartitionFilter(SlotDescriptor desc, List<Expr> conjuncts,
PartitionInfo partitionsInfo) {
PartitionColumnFilter partitionColumnFilter = null;
for (Expr expr : conjuncts) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 332af6ed9b2..965979a240f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -609,6 +609,8 @@ public class StmtExecutor {
scanNode.getSelectedPartitionNum(),
scanNode.getSelectedSplitNum(),
scanNode.getCardinality(),
+ scanNode.isPartitionedTable(),
+ scanNode.hasPartitionPredicate(),
context.getQualifiedUser());
}
@@ -1799,6 +1801,9 @@ public class StmtExecutor {
if (item != null && !item.equals(FeConstants.null_string)) {
Column col = metaData.getColumn(i);
switch (col.getType().getPrimitiveType()) {
+ case BOOLEAN:
+ serializer.writeInt1(parseBooleanResultValue(item));
+ break;
case INT:
serializer.writeInt4(Integer.parseInt(item));
break;
@@ -1832,6 +1837,16 @@ public class StmtExecutor {
}
}
+ private static int parseBooleanResultValue(String item) {
+ if ("1".equals(item) || "true".equalsIgnoreCase(item)) {
+ return 1;
+ }
+ if ("0".equals(item) || "false".equalsIgnoreCase(item)) {
+ return 0;
+ }
+ throw new IllegalArgumentException("Invalid boolean result value: " + item);
+ }
+
public void handleExplainPlanProcessStmt(List<PlanProcess> result) throws IOException {
ShowResultSetMetaData metaData = ShowResultSetMetaData.builder()
.addColumn(new Column("Rule", ScalarType.createVarchar(-1)))
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
index e2a0530b027..530aca66c3e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
@@ -696,6 +696,7 @@ public class MetadataGenerator {
trow.addToColumnValue(new TCell().setLongVal(sqlBlockRule.getCardinality()));
trow.addToColumnValue(new TCell().setBoolVal(sqlBlockRule.getGlobal()));
trow.addToColumnValue(new TCell().setBoolVal(sqlBlockRule.getEnable()));
+ trow.addToColumnValue(new TCell().setBoolVal(sqlBlockRule.getRequirePartitionFilter()));
trow.addToColumnValue(new TCell().setLongVal(sqlBlockRule.getBlockCount().getValue()));
Snapshot snapshot = sqlBlockRule.getTryBlockHistogram().getSnapshot();
trow.addToColumnValue(new TCell().setLongVal((long) snapshot.getMean()));
diff --git a/fe/fe-core/src/test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java b/fe/fe-core/src/test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java
index 06e2fef9f43..a87fcf1a84b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java
@@ -17,15 +17,24 @@
package org.apache.doris.blockrule;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.metric.MetricRepo;
import org.apache.doris.persist.gson.GsonUtils;
import org.junit.Assert;
+import org.junit.BeforeClass;
import org.junit.Test;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public class SqlBlockRuleMgrTest {
+ @BeforeClass
+ public static void setUp() {
+ MetricRepo.init();
+ }
+
@Test
public void testToInfoString() {
SqlBlockRuleMgr mgr = new SqlBlockRuleMgr();
@@ -39,4 +48,64 @@ public class SqlBlockRuleMgrTest {
Assert.assertTrue(nameToSqlBlockRuleMap.containsKey("r1"));
}
+ @Test
+ public void testRuleSerializeRequirePartitionFilter() {
+ SqlBlockRule rule = new SqlBlockRule("r1", "NULL", "NULL", 0L, 0L, 0L,
+ true, true, true);
+ String json = GsonUtils.GSON.toJson(rule);
+ SqlBlockRule roundTrip = GsonUtils.GSON.fromJson(json, SqlBlockRule.class);
+ Assert.assertTrue(roundTrip.getRequirePartitionFilter());
+ }
+
+ @Test
+ public void testShowInfoUseNumericBooleanForRequirePartitionFilter() {
+ SqlBlockRule enabledRule = new SqlBlockRule("r1", "NULL", "NULL", 0L, 0L, 0L,
+ true, true, true);
+ List<String> enabledShowInfo = enabledRule.getShowInfo();
+ Assert.assertEquals(9, enabledShowInfo.size());
+ Assert.assertEquals("1", enabledShowInfo.get(8));
+
+ SqlBlockRule disabledRule = new SqlBlockRule("r2", "NULL", "NULL", 0L, 0L, 0L,
+ false, true, true);
+ List<String> disabledShowInfo = disabledRule.getShowInfo();
+ Assert.assertEquals("0", disabledShowInfo.get(8));
+ }
+
+ @Test
+ public void testConstructorPlaceRequirePartitionFilterBeforeGlobal() {
+ SqlBlockRule rule = new SqlBlockRule("r1", "NULL", "NULL", 0L, 0L, 0L,
+ true, false, true);
+ Assert.assertTrue(rule.getRequirePartitionFilter());
+ Assert.assertFalse(rule.getGlobal());
+ Assert.assertTrue(rule.getEnable());
+ }
+
+ @Test
+ public void testRequirePartitionFilterBlocksPartitionedScanWithoutFilter() {
+ SqlBlockRuleMgr mgr = new SqlBlockRuleMgr();
+ SqlBlockRule rule = new SqlBlockRule("r1", "NULL", "NULL", 0L, 0L, 0L,
+ true, true, true);
+
+ AnalysisException exception = Assert.assertThrows(AnalysisException.class,
+ () -> mgr.checkLimitations(rule, 2L, 3L, 4L, true, false));
+ Assert.assertTrue(exception.getMessage().contains("sql hits sql block rule: r1, missing partition filter"));
+ }
+
+ @Test
+ public void testRequirePartitionFilterAllowsPartitionedScanWithFilter() throws AnalysisException {
+ SqlBlockRuleMgr mgr = new SqlBlockRuleMgr();
+ SqlBlockRule rule = new SqlBlockRule("r1", "NULL", "NULL", 0L, 0L, 0L,
+ true, true, true);
+
+ mgr.checkLimitations(rule, 2L, 3L, 4L, true, true);
+ }
+
+ @Test
+ public void testRequirePartitionFilterAllowsUnpartitionedScan() throws AnalysisException {
+ SqlBlockRuleMgr mgr = new SqlBlockRuleMgr();
+ SqlBlockRule rule = new SqlBlockRule("r1", "NULL", "NULL", 0L, 0L, 0L,
+ true, true, true);
+
+ mgr.checkLimitations(rule, 0L, 3L, 4L, false, false);
+ }
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/SchemaTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/SchemaTableTest.java
index 9970d50a37a..c23d2ba7455 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/SchemaTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/SchemaTableTest.java
@@ -84,6 +84,7 @@ public class SchemaTableTest {
SchemaTable sqlBlockRuleStatus = (SchemaTable) SchemaTable.TABLE_MAP.get("sql_block_rule_status");
Assertions.assertTrue(sqlBlockRuleStatus.shouldFetchAllFe());
Assertions.assertTrue(sqlBlockRuleStatus.shouldAddAgg());
+ Assertions.assertEquals("REQUIRE_PARTITION_FILTER", sqlBlockRuleStatus.getFullSchema().get(8).getName());
SchemaTable processlist = (SchemaTable) SchemaTable.TABLE_MAP.get("processlist");
Assertions.assertTrue(processlist.shouldFetchAllFe());
diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/util/UtilTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/util/UtilTest.java
index 88403effa31..db4f7f028f4 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/common/util/UtilTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/common/util/UtilTest.java
@@ -17,9 +17,14 @@
package org.apache.doris.common.util;
+import org.apache.doris.common.AnalysisException;
+
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.util.HashMap;
+import java.util.Map;
+
public class UtilTest {
@Test
@@ -102,4 +107,26 @@ public class UtilTest {
Assertions.assertEquals(result, Util.sha256long(testStr),
"Same input should produce same output");
}
+
+ @Test
+ public void strictBooleanPropertyParsesAndDefaults() throws AnalysisException {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("flag_true", "TRUE");
+ properties.put("flag_false", "false");
+
+ Assertions.assertTrue(Util.getBooleanPropertyOrDefault(properties, "flag_true", false));
+ Assertions.assertFalse(Util.getBooleanPropertyOrDefault(properties, "flag_false", true));
+ Assertions.assertTrue(Util.getBooleanPropertyOrDefault(properties, "missing_flag", true));
+ Assertions.assertNull(Util.getOptionalBooleanProperty(properties, "missing_optional_flag"));
+ }
+
+ @Test
+ public void strictBooleanPropertyRejectsInvalidValue() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("flag", "not_a_bool");
+
+ AnalysisException exception = Assertions.assertThrows(AnalysisException.class,
+ () -> Util.getBooleanPropertyOrDefault(properties, "flag", false));
+ Assertions.assertEquals("flag should be a boolean", exception.getDetailMessage());
+ }
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/source/HiveScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/source/HiveScanNodeTest.java
index abb48bcae0b..197d8c1fc61 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/source/HiveScanNodeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/source/HiveScanNodeTest.java
@@ -22,10 +22,12 @@ import org.apache.doris.analysis.TupleId;
import org.apache.doris.datasource.hive.HMSExternalCatalog;
import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.datasource.hive.HiveExternalMetaCache;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions;
import org.apache.doris.planner.PlanNodeId;
import org.apache.doris.planner.ScanContext;
import org.apache.doris.qe.SessionVariable;
+import com.google.common.collect.ImmutableMap;
import org.junit.Assert;
import org.junit.Test;
import org.mockito.Mockito;
@@ -86,4 +88,46 @@ public class HiveScanNodeTest {
long target = (long) method.invoke(node, caches, false);
Assert.assertEquals(32 * MB, target);
}
+
+ @Test
+ public void testSelectedPartitionsCarryPartitionPredicateFlag() {
+ SelectedPartitions selectedPartitions = new SelectedPartitions(3, ImmutableMap.of(), true, true);
+ Assert.assertTrue(selectedPartitions.hasPartitionPredicate);
+ }
+
+ @Test
+ public void testHiveScanNodeExposePartitionPredicateFlag() {
+ HiveScanNode node = createHiveScanNode();
+ node.setSelectedPartitions(new SelectedPartitions(3, ImmutableMap.of(), true, true));
+ Assert.assertTrue(node.hasPartitionPredicate());
+ }
+
+ @Test
+ public void testHiveScanNodeExposePartitionedTableFlag() {
+ HiveScanNode node = createHiveScanNode(true);
+ Assert.assertTrue(node.isPartitionedTable());
+ }
+
+ @Test
+ public void testHiveScanNodeExposeMissingPartitionPredicateFlag() {
+ HiveScanNode node = createHiveScanNode();
+ node.setSelectedPartitions(new SelectedPartitions(3, ImmutableMap.of(), true, false));
+ Assert.assertFalse(node.hasPartitionPredicate());
+ }
+
+ private HiveScanNode createHiveScanNode() {
+ return createHiveScanNode(false);
+ }
+
+ private HiveScanNode createHiveScanNode(boolean partitioned) {
+ SessionVariable sv = new SessionVariable();
+ TupleDescriptor desc = new TupleDescriptor(new TupleId(0));
+ HMSExternalTable table = Mockito.mock(HMSExternalTable.class);
+ HMSExternalCatalog catalog = Mockito.mock(HMSExternalCatalog.class);
+ Mockito.when(table.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.bindBrokerName()).thenReturn("");
+ Mockito.when(table.isPartitionedTable()).thenReturn(partitioned);
+ desc.setTable(table);
+ return new HiveScanNode(new PlanNodeId(0), desc, false, sv, null, ScanContext.EMPTY);
+ }
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorTest.java
index 288c5f65a2b..4cbbba08f17 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorTest.java
@@ -21,9 +21,12 @@ import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.GroupingInfo;
import org.apache.doris.analysis.SlotRef;
import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.analysis.TupleId;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.common.UserException;
+import org.apache.doris.nereids.NereidsPlanner;
import org.apache.doris.nereids.properties.DataTrait;
import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.trees.expressions.Expression;
@@ -35,6 +38,8 @@ import org.apache.doris.nereids.trees.expressions.StatementScopeIdGenerator;
import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.plans.PreAggStatus;
+import org.apache.doris.nereids.trees.plans.RelationId;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalFileScan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalFilter;
import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
@@ -45,8 +50,12 @@ import org.apache.doris.planner.AggregationNode;
import org.apache.doris.planner.OlapScanNode;
import org.apache.doris.planner.PlanFragment;
import org.apache.doris.planner.PlanNode;
+import org.apache.doris.planner.PlanNodeId;
import org.apache.doris.planner.Planner;
import org.apache.doris.planner.RepeatNode;
+import org.apache.doris.planner.ScanContext;
+import org.apache.doris.planner.ScanNode;
+import org.apache.doris.thrift.TScanRangeLocations;
import org.apache.doris.utframe.TestWithFeService;
import com.google.common.collect.ImmutableList;
@@ -55,13 +64,17 @@ import com.google.common.collect.Sets;
import mockit.Injectable;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
import java.lang.reflect.Field;
+import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
import java.util.Set;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
public class PhysicalPlanTranslatorTest extends TestWithFeService {
@@ -70,6 +83,15 @@ public class PhysicalPlanTranslatorTest extends TestWithFeService {
createDatabase("test_db");
createTable("create table test_db.t(a int, b int) distributed by hash(a) buckets 3 "
+ "properties('replication_num' = '1');");
+ createTable("create table test_db.partitioned_t(k1 int, p1 int)\n"
+ + "duplicate key(k1, p1)\n"
+ + "partition by range(`p1`)\n"
+ + "(\n"
+ + "partition p1 values less than(\"10\"),\n"
+ + "partition p2 values less than(\"20\")\n"
+ + ")\n"
+ + "distributed by hash(k1) buckets 3\n"
+ + "properties('replication_num' = '1');");
connectContext.getSessionVariable().setDisableNereidsRules("prune_empty_partition");
}
@@ -85,7 +107,17 @@ public class PhysicalPlanTranslatorTest extends TestWithFeService {
t1Output.add(col1);
t1Output.add(col2);
t1Output.add(col3);
- LogicalProperties t1Properties = new LogicalProperties(() -> t1Output, () -> DataTrait.EMPTY_TRAIT);
+ LogicalProperties t1Properties = new LogicalProperties(new Supplier<List<Slot>>() {
+ @Override
+ public List<Slot> get() {
+ return t1Output;
+ }
+ }, new Supplier<DataTrait>() {
+ @Override
+ public DataTrait get() {
+ return DataTrait.EMPTY_TRAIT;
+ }
+ });
PhysicalOlapScan scan = new PhysicalOlapScan(StatementScopeIdGenerator.newRelationId(), t1, qualifier, t1.getBaseIndexId(),
Collections.emptyList(), Collections.emptyList(), null, PreAggStatus.on(),
ImmutableList.of(), Optional.empty(), t1Properties, Optional.empty(),
@@ -104,7 +136,7 @@ public class PhysicalPlanTranslatorTest extends TestWithFeService {
PlanFragment fragment = translator.visitPhysicalProject(project, planTranslatorContext);
PlanNode planNode = fragment.getPlanRoot();
List<OlapScanNode> scanNodeList = new ArrayList<>();
- planNode.collect(OlapScanNode.class::isInstance, scanNodeList);
+ planNode.collect(OlapScanNode.class, scanNodeList);
Assertions.assertEquals(2, scanNodeList.get(0).getTupleDesc().getSlots().size());
}
@@ -122,7 +154,7 @@ public class PhysicalPlanTranslatorTest extends TestWithFeService {
for (PlanFragment fragment : fragments) {
PlanNode root = fragment.getPlanRoot();
if (root != null) {
- root.collect(AggregationNode.class::isInstance, aggNodes);
+ root.collect(AggregationNode.class, aggNodes);
}
}
Assertions.assertEquals(2, aggNodes.size());
@@ -139,30 +171,98 @@ public class PhysicalPlanTranslatorTest extends TestWithFeService {
"upper AggregationNode needsFinalize should be true");
}
+ @Test
+ public void testNereidsOlapScanCarryPartitionPredicateSignal() throws Exception {
+ OlapScanNode filteredScanNode = getFirstOlapScanNode(
+ "select * from test_db.partitioned_t where p1 = 1");
+ Assertions.assertTrue(filteredScanNode.hasPartitionPredicate());
+
+ OlapScanNode manuallyPartitionedScanNode = getFirstOlapScanNode(
+ "select * from test_db.partitioned_t partition(p1)");
+ Assertions.assertTrue(manuallyPartitionedScanNode.hasPartitionPredicate());
+
+ OlapScanNode nonPartitionFilteredScanNode = getFirstOlapScanNode(
+ "select * from test_db.partitioned_t where k1 = 1");
+ Assertions.assertFalse(nonPartitionFilteredScanNode.hasPartitionPredicate());
+ }
+
+ @Test
+ public void testNereidsFileScanCarryPartitionPredicateSignal() throws Exception {
+ PhysicalFileScan fileScan = Mockito.mock(PhysicalFileScan.class);
+ PlanTranslatorContext context = new PlanTranslatorContext();
+ PhysicalPlanTranslator translator = new PhysicalPlanTranslator(context, null);
+ TestScanNode scanNode = new TestScanNode();
+
+ Mockito.when(fileScan.getId()).thenReturn(1);
+ Mockito.when(fileScan.getRelationId()).thenReturn(RelationId.createGenerator().getNextId());
+ Mockito.when(fileScan.hasPartitionPredicate()).thenReturn(true);
+ Mockito.when(fileScan.getStats()).thenReturn(null);
+
+ Method method = PhysicalPlanTranslator.class.getDeclaredMethod("getPlanFragmentForPhysicalFileScan",
+ PhysicalFileScan.class, PlanTranslatorContext.class, ScanNode.class);
+ method.setAccessible(true);
+ method.invoke(translator, fileScan, context, scanNode);
+
+ Assertions.assertTrue(scanNode.hasPartitionPredicate());
+ }
+
@Test
public void testRepeatInputOutputOrder() throws Exception {
String sql = "select grouping(a), grouping(b), grouping_id(a, b), sum(a + 2 * b), sum(a + 3 * b) + grouping_id(b, a, b), b, a, b, a"
+ " from test_db.t"
+ " group by grouping sets((a, b), (), (b), (a, b), (a + b), (a * b))";
- PlanChecker.from(connectContext).checkPlannerResult(sql,
- planner -> {
- Set<RepeatNode> repeatNodes = Sets.newHashSet();
- planner.getFragments().stream()
- .map(PlanFragment::getPlanRoot)
- .forEach(plan -> plan.collect(RepeatNode.class, repeatNodes));
- Assertions.assertEquals(1, repeatNodes.size());
- RepeatNode repeatNode = repeatNodes.iterator().next();
- GroupingInfo groupingInfo = repeatNode.getGroupingInfo();
- List<Expr> preRepeatExprs = groupingInfo.getPreRepeatExprs();
- TupleDescriptor outputs = groupingInfo.getOutputTupleDesc();
- for (int i = 0; i < preRepeatExprs.size(); i++) {
- Expr inputExpr = preRepeatExprs.get(i);
- Assertions.assertInstanceOf(SlotRef.class, inputExpr);
- Column inputColumn = ((SlotRef) inputExpr).getColumn();
- Column outputColumn = outputs.getSlots().get(i).getColumn();
- Assertions.assertEquals(inputColumn, outputColumn);
+ PlanChecker.from(connectContext).checkPlannerResult(sql, new Consumer<NereidsPlanner>() {
+ @Override
+ public void accept(NereidsPlanner planner) {
+ Set<RepeatNode> repeatNodes = Sets.newHashSet();
+ for (PlanFragment fragment : planner.getFragments()) {
+ PlanNode plan = fragment.getPlanRoot();
+ if (plan != null) {
+ plan.collect(RepeatNode.class, repeatNodes);
}
}
- );
+ Assertions.assertEquals(1, repeatNodes.size());
+ RepeatNode repeatNode = repeatNodes.iterator().next();
+ GroupingInfo groupingInfo = repeatNode.getGroupingInfo();
+ List<Expr> preRepeatExprs = groupingInfo.getPreRepeatExprs();
+ TupleDescriptor outputs = groupingInfo.getOutputTupleDesc();
+ for (int i = 0; i < preRepeatExprs.size(); i++) {
+ Expr inputExpr = preRepeatExprs.get(i);
+ Assertions.assertInstanceOf(SlotRef.class, inputExpr);
+ Column inputColumn = ((SlotRef) inputExpr).getColumn();
+ Column outputColumn = outputs.getSlots().get(i).getColumn();
+ Assertions.assertEquals(inputColumn, outputColumn);
+ }
+ }
+ });
+ }
+
+ private OlapScanNode getFirstOlapScanNode(String sql) throws Exception {
+ Planner planner = getSQLPlanner(sql);
+ Assertions.assertNotNull(planner);
+ List<OlapScanNode> scanNodes = new ArrayList<>();
+ for (PlanFragment fragment : planner.getFragments()) {
+ PlanNode root = fragment.getPlanRoot();
+ if (root != null) {
+ root.collect(OlapScanNode.class, scanNodes);
+ }
+ }
+ Assertions.assertFalse(scanNodes.isEmpty());
+ return scanNodes.get(0);
+ }
+
+ private static final class TestScanNode extends ScanNode {
+ private TestScanNode() {
+ super(new PlanNodeId(0), new TupleDescriptor(new TupleId(0)), "TEST_SCAN_NODE", ScanContext.EMPTY);
+ }
+
+ @Override
+ protected void createScanRangeLocations() throws UserException {
+ }
+
+ @Override
+ public List<TScanRangeLocations> getScanRangeLocations(long maxScanRangeLength) {
+ return Collections.emptyList();
+ }
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/mv/OptimizeGetAvailableMvsTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/mv/OptimizeGetAvailableMvsTest.java
index 0bd07bddc32..e47f3a3ef30 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/mv/OptimizeGetAvailableMvsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/mv/OptimizeGetAvailableMvsTest.java
@@ -28,6 +28,7 @@ import org.apache.doris.common.Pair;
import org.apache.doris.mtmv.BaseTableInfo;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.rules.expression.rules.PartitionPruner;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruner.PartitionPruneResult;
import org.apache.doris.nereids.rules.expression.rules.PartitionPruner.PartitionTableType;
import org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges;
import org.apache.doris.nereids.sqltest.SqlTestBase;
@@ -123,50 +124,52 @@ public class OptimizeGetAvailableMvsTest extends SqlTestBase {
connectContext.getSessionVariable().enableMaterializedViewRewrite = true;
connectContext.getSessionVariable().enableMaterializedViewNestRewrite = true;
- createMvByNereids("create materialized view mv1 "
- + " BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL\n"
- + " PARTITION BY (id)\n"
- + " DISTRIBUTED BY RANDOM BUCKETS 1\n"
- + " PROPERTIES ('replication_num' = '1') \n"
- + " as "
- + " select T4.id from T4 inner join T2 "
- + " on T4.id = T2.id;");
- CascadesContext c1 = createCascadesContext(
- "select T4.id "
- + "from T4 "
- + "inner join T2 on T4.id = T2.id "
- + "inner join T3 on T4.id = T3.id",
- connectContext
- );
- PlanChecker.from(c1)
- .setIsQuery()
- .analyze()
- .rewrite()
- .preMvRewrite()
- .optimize()
- .printlnBestPlanTree();
- Multimap<List<String>, Pair<RelationId, Set<String>>> tableUsedPartitionNameMap = c1.getStatementContext()
- .getTableUsedPartitionNameMap();
- Assertions.assertFalse(tableUsedPartitionNameMap.isEmpty());
+ try {
+ createMvByNereids("create materialized view mv1 "
+ + " BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL\n"
+ + " PARTITION BY (id)\n"
+ + " DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + " PROPERTIES ('replication_num' = '1') \n"
+ + " as "
+ + " select T4.id from T4 inner join T2 "
+ + " on T4.id = T2.id;");
+ CascadesContext c1 = createCascadesContext(
+ "select T4.id "
+ + "from T4 "
+ + "inner join T2 on T4.id = T2.id "
+ + "inner join T3 on T4.id = T3.id",
+ connectContext
+ );
+ PlanChecker.from(c1)
+ .setIsQuery()
+ .analyze()
+ .rewrite()
+ .preMvRewrite()
+ .optimize()
+ .printlnBestPlanTree();
+ Multimap<List<String>, Pair<RelationId, Set<String>>> tableUsedPartitionNameMap = c1.getStatementContext()
+ .getTableUsedPartitionNameMap();
+ Assertions.assertFalse(tableUsedPartitionNameMap.isEmpty());
- for (Map.Entry<List<String>, Pair<RelationId, Set<String>>> tableInfoEntry
- : tableUsedPartitionNameMap.entries()) {
- if (tableInfoEntry.getKey().contains("T2")) {
- Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
- } else if (tableInfoEntry.getKey().contains("T3")) {
- Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
- } else if (tableInfoEntry.getKey().contains("T4")) {
- Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
+ for (Map.Entry<List<String>, Pair<RelationId, Set<String>>> tableInfoEntry
+ : tableUsedPartitionNameMap.entries()) {
+ if (tableInfoEntry.getKey().contains("T2")) {
+ Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
+ } else if (tableInfoEntry.getKey().contains("T3")) {
+ Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
+ } else if (tableInfoEntry.getKey().contains("T4")) {
+ Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
+ }
}
- }
-
- Map<BaseTableInfo, Collection<Partition>> mvCanRewritePartitionsMap = c1.getStatementContext()
- .getMvCanRewritePartitionsMap();
- Assertions.assertEquals(1, mvCanRewritePartitionsMap.size());
- Assertions.assertTrue(mvCanRewritePartitionsMap.keySet().iterator().next().getTableName()
- .equalsIgnoreCase("mv1"));
- dropMvByNereids("drop materialized view mv1");
+ Map<BaseTableInfo, Collection<Partition>> mvCanRewritePartitionsMap = c1.getStatementContext()
+ .getMvCanRewritePartitionsMap();
+ Assertions.assertEquals(1, mvCanRewritePartitionsMap.size());
+ Assertions.assertTrue(mvCanRewritePartitionsMap.keySet().iterator().next().getTableName()
+ .equalsIgnoreCase("mv1"));
+ } finally {
+ dropMvByNereids("drop materialized view if exists mv1");
+ }
}
@Test
@@ -182,10 +185,11 @@ public class OptimizeGetAvailableMvsTest extends SqlTestBase {
new MockUp<PartitionPruner>() {
@Mock
- public <K extends Comparable<K>> Pair<List<K>, Optional<Expression>> prune(List<Slot> partitionSlots, Expression partitionPredicate,
+ public <K extends Comparable<K>> PartitionPruneResult<K> pruneWithResult(List<Slot> partitionSlots,
+ Expression partitionPredicate,
Map<K, PartitionItem> idToPartitions, CascadesContext cascadesContext,
PartitionTableType partitionTableType, Optional<SortedPartitionRanges<K>> sortedPartitionRanges) {
- return Pair.of((List) Lists.newArrayList(1L), Optional.empty());
+ return new PartitionPruneResult<>((List<K>) Lists.newArrayList(1L), Optional.empty(), true);
}
};
@@ -235,50 +239,52 @@ public class OptimizeGetAvailableMvsTest extends SqlTestBase {
connectContext.getSessionVariable().enableMaterializedViewRewrite = true;
connectContext.getSessionVariable().enableMaterializedViewNestRewrite = true;
- createMvByNereids("create materialized view mv2 "
- + " BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL\n"
- + " PARTITION BY (id)\n"
- + " DISTRIBUTED BY RANDOM BUCKETS 1\n"
- + " PROPERTIES ('replication_num' = '1') \n"
- + " as "
- + " select T4.id from T4 inner join T2 "
- + " on T4.id = T2.id;");
- CascadesContext c1 = createCascadesContext(
- "select T4.id "
- + "from T4 "
- + "inner join T2 on T4.id = T2.id "
- + "inner join T3 on T4.id = T3.id "
- + "where T4.id > 0",
- connectContext
- );
- PlanChecker.from(c1)
- .setIsQuery()
- .analyze()
- .rewrite()
- .preMvRewrite()
- .optimize()
- .printlnBestPlanTree();
- Multimap<List<String>, Pair<RelationId, Set<String>>> tableUsedPartitionNameMap = c1.getStatementContext()
- .getTableUsedPartitionNameMap();
- Map<BaseTableInfo, Collection<Partition>> mvCanRewritePartitionsMap = c1.getStatementContext()
- .getMvCanRewritePartitionsMap();
- Assertions.assertFalse(tableUsedPartitionNameMap.isEmpty());
+ try {
+ createMvByNereids("create materialized view mv2 "
+ + " BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL\n"
+ + " PARTITION BY (id)\n"
+ + " DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + " PROPERTIES ('replication_num' = '1') \n"
+ + " as "
+ + " select T4.id from T4 inner join T2 "
+ + " on T4.id = T2.id;");
+ CascadesContext c1 = createCascadesContext(
+ "select T4.id "
+ + "from T4 "
+ + "inner join T2 on T4.id = T2.id "
+ + "inner join T3 on T4.id = T3.id "
+ + "where T4.id > 0",
+ connectContext
+ );
+ PlanChecker.from(c1)
+ .setIsQuery()
+ .analyze()
+ .rewrite()
+ .preMvRewrite()
+ .optimize()
+ .printlnBestPlanTree();
+ Multimap<List<String>, Pair<RelationId, Set<String>>> tableUsedPartitionNameMap = c1.getStatementContext()
+ .getTableUsedPartitionNameMap();
+ Map<BaseTableInfo, Collection<Partition>> mvCanRewritePartitionsMap = c1.getStatementContext()
+ .getMvCanRewritePartitionsMap();
+ Assertions.assertFalse(tableUsedPartitionNameMap.isEmpty());
- for (Map.Entry<List<String>, Pair<RelationId, Set<String>>> tableInfoEntry
- : tableUsedPartitionNameMap.entries()) {
- if (tableInfoEntry.getKey().contains("T2")) {
- Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
- } else if (tableInfoEntry.getKey().contains("T3")) {
- Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
- } else if (tableInfoEntry.getKey().contains("T4")) {
- Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
+ for (Map.Entry<List<String>, Pair<RelationId, Set<String>>> tableInfoEntry
+ : tableUsedPartitionNameMap.entries()) {
+ if (tableInfoEntry.getKey().contains("T2")) {
+ Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
+ } else if (tableInfoEntry.getKey().contains("T3")) {
+ Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
+ } else if (tableInfoEntry.getKey().contains("T4")) {
+ Assertions.assertEquals(tableInfoEntry.getValue().value(), Sets.newHashSet("mock_partition"));
+ }
}
- }
-
- Assertions.assertEquals(1, mvCanRewritePartitionsMap.size());
- Assertions.assertTrue(mvCanRewritePartitionsMap.keySet().iterator().next().getTableName()
- .equalsIgnoreCase("mv2"));
- dropMvByNereids("drop materialized view mv2");
+ Assertions.assertEquals(1, mvCanRewritePartitionsMap.size());
+ Assertions.assertTrue(mvCanRewritePartitionsMap.keySet().iterator().next().getTableName()
+ .equalsIgnoreCase("mv2"));
+ } finally {
+ dropMvByNereids("drop materialized view if exists mv2");
+ }
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PartitionPrunerTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PartitionPrunerTest.java
index 116d2a89272..6d38a17fca1 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PartitionPrunerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PartitionPrunerTest.java
@@ -20,6 +20,8 @@ package org.apache.doris.nereids.rules.rewrite;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.ListPartitionItem;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.PartitionItem;
import org.apache.doris.catalog.PartitionKey;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.AnalysisException;
@@ -28,27 +30,40 @@ import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.rules.expression.rules.OneListPartitionEvaluator;
import org.apache.doris.nereids.rules.expression.rules.OnePartitionEvaluator;
import org.apache.doris.nereids.rules.expression.rules.PartitionPruner;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruner.PartitionPruneResult;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruner.PartitionTableType;
import org.apache.doris.nereids.trees.expressions.And;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.GreaterThan;
import org.apache.doris.nereids.trees.expressions.InPredicate;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.LessThan;
import org.apache.doris.nereids.trees.expressions.Not;
import org.apache.doris.nereids.trees.expressions.Or;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.RelationId;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.types.IntegerType;
import org.apache.doris.nereids.types.VarcharType;
import org.apache.doris.utframe.TestWithFeService;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
+import java.util.LinkedHashSet;
import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
public class PartitionPrunerTest extends TestWithFeService {
private Method canBePrunedOutMethod;
@@ -308,5 +323,290 @@ public class PartitionPrunerTest extends TestWithFeService {
Assertions.assertFalse(result.first);
Assertions.assertFalse(result.second);
}
-}
+ // test prune predicate
+ // Test basis: some predicates are pruned
+ @Test
+ public void testPrunePartialPredicates() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ LessThan lt = new LessThan(slotB, Literal.of(20));
+ predicates.add(gt);
+ predicates.add(lt);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(gt), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+ Assertions.assertEquals(1, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(lt));
+ Assertions.assertFalse(prunedFilter.getConjuncts().contains(gt));
+ }
+
+ // all predicates are pruned
+ @Test
+ public void testPruneAllPredicates() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ predicates.add(gt);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(gt), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalOlapScan.class, prunedPlan);
+ }
+
+ // no predicates are pruned
+ @Test
+ public void testPruneNoPredicates() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ LessThan lt = new LessThan(slotB, Literal.of(20));
+ predicates.add(gt);
+ predicates.add(lt);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ EqualTo nonExistentPredicate = new EqualTo(slotC, Literal.of(30));
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(nonExistentPredicate), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+ Assertions.assertEquals(2, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(gt));
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(lt));
+ }
+
+ @Test
+ public void testPruneCompoundPredicate() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ LessThan lt = new LessThan(slotB, Literal.of(20));
+ EqualTo eq = new EqualTo(slotC, Literal.of(30));
+ predicates.add(gt);
+ predicates.add(lt);
+ predicates.add(eq);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ // (a > 10 AND b < 20)
+ And compoundPredicate = new And(gt, lt);
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(compoundPredicate), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+ Assertions.assertEquals(1, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(eq));
+ Assertions.assertFalse(prunedFilter.getConjuncts().contains(gt));
+ Assertions.assertFalse(prunedFilter.getConjuncts().contains(lt));
+ }
+
+ @Test
+ public void testSkipPrunePredicate() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ predicates.add(gt);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ Plan prunedPlan = PartitionPruner.prunePredicate(true, Optional.of(gt), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+ Assertions.assertEquals(1, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(gt));
+ }
+
+ @Test
+ public void testEmptyPrunedPredicates() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ predicates.add(gt);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ // prunedPredicates is empty
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.empty(), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+ Assertions.assertEquals(1, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(gt));
+ }
+
+ @Test
+ public void testPruneDuplicatePredicates() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt1 = new GreaterThan(slotA, Literal.of(10));
+ GreaterThan gt2 = new GreaterThan(slotA, Literal.of(10)); // duplicated predicate
+ predicates.add(gt1);
+ predicates.add(gt2);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(gt1), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalOlapScan.class, prunedPlan);
+ }
+
+ @Test
+ public void testPruneWithNullLiteral() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ EqualTo nullEq = new EqualTo(slotB, new NullLiteral());
+ predicates.add(gt);
+ predicates.add(nullEq);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(gt), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+ Assertions.assertEquals(1, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(nullEq));
+ }
+
+ @Test
+ public void testPruneMultiplePredicatesPartially() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ LessThan lt = new LessThan(slotB, Literal.of(20));
+ EqualTo eq1 = new EqualTo(slotC, Literal.of(30));
+ EqualTo eq2 = new EqualTo(slotC, Literal.of(40));
+ predicates.add(gt);
+ predicates.add(lt);
+ predicates.add(eq1);
+ predicates.add(eq2);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ // (a > 10 AND b < 20)
+ And compoundPredicate = new And(gt, lt);
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(compoundPredicate), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+ Assertions.assertEquals(2, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(eq1));
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(eq2));
+ Assertions.assertFalse(prunedFilter.getConjuncts().contains(gt));
+ Assertions.assertFalse(prunedFilter.getConjuncts().contains(lt));
+ }
+
+ @Test
+ public void testPruneNestedCompoundPredicate() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ LessThan lt = new LessThan(slotB, Literal.of(20));
+ EqualTo eq = new EqualTo(slotC, Literal.of(30));
+ predicates.add(gt);
+ predicates.add(lt);
+ predicates.add(eq);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ // (a > 10 AND (b < 20 AND c = 30))
+ And innerAnd = new And(lt, eq);
+ And outerAnd = new And(gt, innerAnd);
+
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(outerAnd), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalOlapScan.class, prunedPlan);
+ }
+
+ @Test
+ public void testPruneWhenFilterContainsOr() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ Or orPredicate = new Or(
+ new GreaterThan(slotA, Literal.of(10)),
+ new LessThan(slotB, Literal.of(20))
+ );
+ predicates.add(orPredicate);
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(gt), filter, scan);
+
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+ Assertions.assertEquals(1, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(orPredicate));
+ }
+
+ @Test
+ public void testPruneWhenFilterContainsAndOrMix() {
+ Set<Expression> predicates = new LinkedHashSet<>();
+ // filter :a > 10 AND (b < 20 OR c = 30)
+ Or orPredicate = new Or(
+ new LessThan(slotB, Literal.of(20)),
+ new EqualTo(slotC, Literal.of(30))
+ );
+ GreaterThan gt = new GreaterThan(slotA, Literal.of(10));
+
+ predicates.add(gt);
+ predicates.add(orPredicate);
+
+ LogicalOlapScan scan = new LogicalOlapScan(new RelationId(1), new OlapTable());
+ LogicalFilter<Plan> filter = new LogicalFilter<>(predicates, scan);
+ // a > 10
+ Plan prunedPlan = PartitionPruner.prunePredicate(false, Optional.of(gt), filter, scan);
+ Assertions.assertInstanceOf(LogicalFilter.class, prunedPlan);
+ LogicalFilter<?> prunedFilter = (LogicalFilter<?>) prunedPlan;
+
+ Assertions.assertEquals(1, prunedFilter.getConjuncts().size());
+ Assertions.assertTrue(prunedFilter.getConjuncts().contains(orPredicate));
+ }
+
+ @Test
+ public void testPruneWithResultIgnoresNonPruningPartitionPredicate() throws AnalysisException {
+ Map<String, PartitionItem> idToPartitions = ImmutableMap.of(
+ "p1", createListPartitionItem("1"),
+ "p2", createListPartitionItem("2"));
+
+ PartitionPruneResult<String> result = PartitionPruner.pruneWithResult(
+ ImmutableList.of(slotA), new Not(new IsNull(slotA)), idToPartitions, cascadesContext,
+ PartitionTableType.OLAP, Optional.empty());
+
+ Assertions.assertEquals(2, result.partitions.size());
+ Assertions.assertFalse(result.hasPartitionPredicate);
+ }
+
+ @Test
+ public void testPruneWithResultMarksEffectivePartitionPredicate() throws AnalysisException {
+ Map<String, PartitionItem> idToPartitions = ImmutableMap.of(
+ "p1", createListPartitionItem("1"),
+ "p2", createListPartitionItem("2"));
+
+ PartitionPruneResult<String> result = PartitionPruner.pruneWithResult(
+ ImmutableList.of(slotA), new EqualTo(slotA, Literal.of(1)), idToPartitions, cascadesContext,
+ PartitionTableType.OLAP, Optional.empty());
+
+ Assertions.assertEquals(1, result.partitions.size());
+ Assertions.assertTrue(result.hasPartitionPredicate);
+ }
+
+ private ListPartitionItem createListPartitionItem(String... values) throws AnalysisException {
+ ImmutableList.Builder<PartitionKey> partitionKeys = ImmutableList.builder();
+ for (String value : values) {
+ PartitionValue partitionValue = new PartitionValue(value);
+ partitionKeys.add(PartitionKey.createPartitionKey(
+ ImmutableList.of(partitionValue), ImmutableList.of(partitionColumn)));
+ }
+ return new ListPartitionItem(partitionKeys.build());
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/SqlBlockRuleCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/SqlBlockRuleCommandTest.java
new file mode 100644
index 00000000000..f3bfe97ef74
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/SqlBlockRuleCommandTest.java
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.nereids.parser.NereidsParser;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class SqlBlockRuleCommandTest {
+ private static final String CREATE_RULE =
+ "create sql_block_rule %s properties(\"require_partition_filter\" = \"true\", "
+ + "\"global\" = \"true\", \"enable\" = \"true\")";
+ private static final String CREATE_RULE_WITH_PARTITION_NUM =
+ "create sql_block_rule %s properties(\"require_partition_filter\" = \"true\", "
+ + "\"partition_num\" = \"3\", \"global\" = \"true\", \"enable\" = \"true\")";
+ private static final String CREATE_RULE_WITH_SQL =
+ "create sql_block_rule %s properties(\"require_partition_filter\" = \"true\", "
+ + "\"sql\" = \"select \\\\*\", \"global\" = \"true\", \"enable\" = \"true\")";
+ private static final String CREATE_RULE_WITH_SQL_HASH =
+ "create sql_block_rule %s properties(\"require_partition_filter\" = \"true\", "
+ + "\"sqlHash\" = \"abc\", \"global\" = \"true\", \"enable\" = \"true\")";
+ private static final String ALTER_RULE =
+ "alter sql_block_rule %s properties(\"require_partition_filter\" = \"true\")";
+ private static final String ALTER_RULE_INVALID_REQUIRE_PARTITION_FILTER =
+ "alter sql_block_rule %s properties(\"require_partition_filter\" = \"not_a_bool\")";
+
+ @Test
+ public void testCreateRequirePartitionFilterRule() throws Exception {
+ String ruleName = "test_require_partition_filter_create";
+ LogicalPlan plan = new NereidsParser().parseSingle(String.format(CREATE_RULE, ruleName));
+ Assertions.assertInstanceOf(CreateSqlBlockRuleCommand.class, plan);
+
+ CreateSqlBlockRuleCommand command = (CreateSqlBlockRuleCommand) plan;
+ command.setProperties(command.properties);
+ Assertions.assertTrue(command.getRequirePartitionFilter());
+ Assertions.assertTrue(command.getGlobal());
+ Assertions.assertTrue(command.getEnable());
+ }
+
+ @Test
+ public void testAlterRequirePartitionFilterRule() throws Exception {
+ String ruleName = "test_require_partition_filter_alter";
+ LogicalPlan alterPlan = new NereidsParser().parseSingle(String.format(ALTER_RULE, ruleName));
+ Assertions.assertInstanceOf(AlterSqlBlockRuleCommand.class, alterPlan);
+
+ AlterSqlBlockRuleCommand command = (AlterSqlBlockRuleCommand) alterPlan;
+ command.setProperties(command.properties);
+ Assertions.assertTrue(command.getRequirePartitionFilter());
+ }
+
+ @Test
+ public void testRequirePartitionFilterConflictsWithSql() {
+ String ruleName = "test_require_partition_filter_sql_conflict";
+ LogicalPlan plan = new NereidsParser().parseSingle(String.format(CREATE_RULE_WITH_SQL, ruleName));
+ Assertions.assertInstanceOf(CreateSqlBlockRuleCommand.class, plan);
+
+ Assertions.assertThrows(AnalysisException.class,
+ () -> ((CreateSqlBlockRuleCommand) plan).setProperties(((CreateSqlBlockRuleCommand) plan).properties));
+ }
+
+ @Test
+ public void testRequirePartitionFilterConflictsWithSqlHash() {
+ String ruleName = "test_require_partition_filter_sql_hash_conflict";
+ LogicalPlan plan = new NereidsParser().parseSingle(String.format(CREATE_RULE_WITH_SQL_HASH, ruleName));
+ Assertions.assertInstanceOf(CreateSqlBlockRuleCommand.class, plan);
+
+ Assertions.assertThrows(AnalysisException.class,
+ () -> ((CreateSqlBlockRuleCommand) plan).setProperties(((CreateSqlBlockRuleCommand) plan).properties));
+ }
+
+ @Test
+ public void testRequirePartitionFilterCoexistsWithPartitionNum() throws Exception {
+ String ruleName = "test_require_partition_filter_partition_num";
+ LogicalPlan plan = new NereidsParser().parseSingle(String.format(CREATE_RULE_WITH_PARTITION_NUM, ruleName));
+ Assertions.assertInstanceOf(CreateSqlBlockRuleCommand.class, plan);
+
+ CreateSqlBlockRuleCommand command = (CreateSqlBlockRuleCommand) plan;
+ command.setProperties(command.properties);
+ Assertions.assertEquals(3L, command.getPartitionNum());
+ Assertions.assertTrue(command.getRequirePartitionFilter());
+ }
+
+ @Test
+ public void testAlterRequirePartitionFilterRejectInvalidBoolean() {
+ String ruleName = "test_require_partition_filter_invalid_boolean";
+ LogicalPlan alterPlan = new NereidsParser().parseSingle(
+ String.format(ALTER_RULE_INVALID_REQUIRE_PARTITION_FILTER, ruleName));
+ Assertions.assertInstanceOf(AlterSqlBlockRuleCommand.class, alterPlan);
+
+ AnalysisException exception = Assertions.assertThrows(AnalysisException.class,
+ () -> ((AlterSqlBlockRuleCommand) alterPlan)
+ .setProperties(((AlterSqlBlockRuleCommand) alterPlan).properties));
+ Assertions.assertTrue(exception.getMessage().contains("require_partition_filter should be a boolean"));
+ }
+
+ @Test
+ public void testShowSqlBlockRuleRequirePartitionFilterColumnUseBooleanType() {
+ ShowSqlBlockRuleCommand command = new ShowSqlBlockRuleCommand(null);
+ ShowResultSetMetaData metaData = command.getMetaData();
+ Assertions.assertEquals(9, metaData.getColumnCount());
+ Assertions.assertEquals(PrimitiveType.BOOLEAN,
+ metaData.getColumn(8).getType().getPrimitiveType());
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/OlapScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/OlapScanNodeTest.java
index 2bc14919b04..1b04ca152b7 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/OlapScanNodeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/OlapScanNodeTest.java
@@ -17,11 +17,16 @@
package org.apache.doris.planner;
+import org.apache.doris.analysis.BinaryPredicate;
import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.InPredicate;
import org.apache.doris.analysis.IntLiteral;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.SlotId;
import org.apache.doris.analysis.SlotRef;
import org.apache.doris.analysis.TableName;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.analysis.TupleId;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.PartitionKey;
import org.apache.doris.catalog.PrimitiveType;
@@ -175,4 +180,50 @@ public class OlapScanNodeTest {
SlotRef slot = new SlotRef(new TableName("DB.TBL"), Column.DELETE_SIGN);
Assert.assertTrue(slot.getTableName().toString().equals("DB.tbl"));
}
+
+ @Test
+ public void testHasPartitionPredicateWithEquality() {
+ TupleDescriptor tupleDescriptor = new TupleDescriptor(new TupleId(1));
+ SlotDescriptor partitionSlot = addSlot(tupleDescriptor, 1, "p1");
+ addSlot(tupleDescriptor, 2, "c1");
+
+ List<Expr> conjuncts = Lists.newArrayList(new BinaryPredicate(BinaryPredicate.Operator.EQ,
+ new SlotRef(partitionSlot), new IntLiteral(1)));
+
+ Assert.assertTrue(ScanNode.containsPartitionPredicate(
+ Lists.newArrayList(partitionSlot.getColumn()), tupleDescriptor, conjuncts, null));
+ }
+
+ @Test
+ public void testHasPartitionPredicateWithInPredicate() {
+ TupleDescriptor tupleDescriptor = new TupleDescriptor(new TupleId(1));
+ SlotDescriptor partitionSlot = addSlot(tupleDescriptor, 1, "p1");
+ addSlot(tupleDescriptor, 2, "c1");
+
+ List<Expr> inList = Lists.newArrayList(new IntLiteral(1), new IntLiteral(2));
+ List<Expr> conjuncts = Lists.newArrayList(new InPredicate(new SlotRef(partitionSlot), inList, false));
+
+ Assert.assertTrue(ScanNode.containsPartitionPredicate(
+ Lists.newArrayList(partitionSlot.getColumn()), tupleDescriptor, conjuncts, null));
+ }
+
+ @Test
+ public void testHasPartitionPredicateIgnoresNonPartitionColumn() {
+ TupleDescriptor tupleDescriptor = new TupleDescriptor(new TupleId(1));
+ SlotDescriptor partitionSlot = addSlot(tupleDescriptor, 1, "p1");
+ SlotDescriptor nonPartitionSlot = addSlot(tupleDescriptor, 2, "c1");
+
+ List<Expr> conjuncts = Lists.newArrayList(new BinaryPredicate(BinaryPredicate.Operator.EQ,
+ new SlotRef(nonPartitionSlot), new IntLiteral(1)));
+
+ Assert.assertFalse(ScanNode.containsPartitionPredicate(
+ Lists.newArrayList(partitionSlot.getColumn()), tupleDescriptor, conjuncts, null));
+ }
+
+ private SlotDescriptor addSlot(TupleDescriptor tupleDescriptor, int slotId, String columnName) {
+ SlotDescriptor slotDescriptor = new SlotDescriptor(new SlotId(slotId), tupleDescriptor);
+ slotDescriptor.setColumn(new Column(columnName, PrimitiveType.BIGINT));
+ tupleDescriptor.addSlot(slotDescriptor);
+ return slotDescriptor;
+ }
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
index 3e3f4ba9d49..c5eb0b45c3f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
@@ -36,6 +36,8 @@ import org.junit.Assert;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
import java.io.IOException;
import java.lang.reflect.Field;
@@ -220,18 +222,21 @@ public class StmtExecutorTest extends TestWithFeService {
columns.add(new Column());
ResultSet resultSet = new CommonResultSet(new CommonResultSetMetaData(columns), rows);
AtomicInteger i = new AtomicInteger();
- Mockito.doAnswer(invocation -> {
- byte[] expected0 = new byte[]{-5, 4, 114, 111, 119, 49};
- byte[] expected1 = new byte[]{4, 49, 50, 51, 52, 4, 114, 111, 119, 50};
- ByteBuffer buffer = invocation.getArgument(0);
- if (i.get() == 0) {
- Assertions.assertArrayEquals(expected0, buffer.array());
- i.getAndIncrement();
- } else if (i.get() == 1) {
- Assertions.assertArrayEquals(expected1, buffer.array());
- i.getAndIncrement();
+ Mockito.doAnswer(new Answer<Void>() {
+ @Override
+ public Void answer(InvocationOnMock invocation) {
+ byte[] expected0 = new byte[] {-5, 4, 114, 111, 119, 49};
+ byte[] expected1 = new byte[] {4, 49, 50, 51, 52, 4, 114, 111, 119, 50};
+ ByteBuffer buffer = invocation.getArgument(0);
+ if (i.get() == 0) {
+ Assertions.assertArrayEquals(expected0, buffer.array());
+ i.getAndIncrement();
+ } else if (i.get() == 1) {
+ Assertions.assertArrayEquals(expected1, buffer.array());
+ i.getAndIncrement();
+ }
+ return null;
}
- return null;
}).when(channel).sendOnePacket(Mockito.any(ByteBuffer.class));
StmtExecutor executor = new StmtExecutor(mockCtx, stmt, false);
@@ -264,18 +269,62 @@ public class StmtExecutorTest extends TestWithFeService {
columns.add(new Column("col2", PrimitiveType.DATETIMEV2));
ResultSet resultSet = new CommonResultSet(new CommonResultSetMetaData(columns), rows);
AtomicInteger i = new AtomicInteger();
- Mockito.doAnswer(invocation -> {
- byte[] expected0 = new byte[] {0, 4, 7, -23, 7, 1, 1, 1, 2, 3};
- byte[] expected1 = new byte[] {0, 0, -46, 4, 0, 0, 0, 0, 0, 0, 11, -23, 7, 1, 1, 1, 2, 3, 64, -30, 1, 0};
- ByteBuffer buffer = invocation.getArgument(0);
- if (i.get() == 0) {
- Assertions.assertArrayEquals(expected0, buffer.array());
- i.getAndIncrement();
- } else if (i.get() == 1) {
- Assertions.assertArrayEquals(expected1, buffer.array());
- i.getAndIncrement();
+ Mockito.doAnswer(new Answer<Void>() {
+ @Override
+ public Void answer(InvocationOnMock invocation) {
+ byte[] expected0 = new byte[] {0, 4, 7, -23, 7, 1, 1, 1, 2, 3};
+ byte[] expected1 = new byte[] {0, 0, -46, 4, 0, 0, 0, 0, 0, 0, 11, -23, 7, 1, 1, 1, 2, 3,
+ 64, -30, 1, 0};
+ ByteBuffer buffer = invocation.getArgument(0);
+ if (i.get() == 0) {
+ Assertions.assertArrayEquals(expected0, buffer.array());
+ i.getAndIncrement();
+ } else if (i.get() == 1) {
+ Assertions.assertArrayEquals(expected1, buffer.array());
+ i.getAndIncrement();
+ }
+ return null;
+ }
+ }).when(channel).sendOnePacket(Mockito.any(ByteBuffer.class));
+
+ StmtExecutor executor = new StmtExecutor(mockCtx, stmt, false);
+ executor.sendBinaryResultRow(resultSet);
+ }
+
+ @Test
+ public void testSendBinaryBooleanResultRow() throws IOException {
+ ConnectContext mockCtx = Mockito.mock(ConnectContext.class);
+ MysqlChannel channel = Mockito.mock(MysqlChannel.class);
+ Mockito.when(mockCtx.getConnectType()).thenReturn(ConnectType.MYSQL);
+ Mockito.when(mockCtx.getMysqlChannel()).thenReturn(channel);
+ MysqlSerializer mysqlSerializer = MysqlSerializer.newInstance();
+ Mockito.when(channel.getSerializer()).thenReturn(mysqlSerializer);
+ SessionVariable sessionVariable = VariableMgr.newSessionVariable();
+ Mockito.when(mockCtx.getSessionVariable()).thenReturn(sessionVariable);
+ OriginStatement stmt = new OriginStatement("", 1);
+
+ List<List<String>> rows = Lists.newArrayList();
+ rows.add(Lists.newArrayList("false"));
+ rows.add(Lists.newArrayList("1"));
+ List<Column> columns = Lists.newArrayList();
+ columns.add(new Column("col1", PrimitiveType.BOOLEAN));
+ ResultSet resultSet = new CommonResultSet(new CommonResultSetMetaData(columns), rows);
+ AtomicInteger i = new AtomicInteger();
+ Mockito.doAnswer(new Answer<Void>() {
+ @Override
+ public Void answer(InvocationOnMock invocation) {
+ byte[] expected0 = new byte[] {0, 0, 0};
+ byte[] expected1 = new byte[] {0, 0, 1};
+ ByteBuffer buffer = invocation.getArgument(0);
+ if (i.get() == 0) {
+ Assertions.assertArrayEquals(expected0, buffer.array());
+ i.getAndIncrement();
+ } else if (i.get() == 1) {
+ Assertions.assertArrayEquals(expected1, buffer.array());
+ i.getAndIncrement();
+ }
+ return null;
}
- return null;
}).when(channel).sendOnePacket(Mockito.any(ByteBuffer.class));
StmtExecutor executor = new StmtExecutor(mockCtx, stmt, false);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java b/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java
index dddbaf9f096..9fbd1f98319 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java
@@ -767,7 +767,8 @@ public abstract class TestWithFeService {
Env.getCurrentEnv().getSqlBlockRuleMgr()
.createSqlBlockRule(new SqlBlockRule(command.getRuleName(), command.getSql(), command.getSqlHash(),
command.getPartitionNum(), command.getTabletNum(), command.getCardinality(),
- command.getGlobal(), command.getEnable()), command.isIfNotExists());
+ command.getRequirePartitionFilter(), command.getGlobal(), command.getEnable()),
+ command.isIfNotExists());
}
protected void alterSqlBlockRule(String sql) throws Exception {
@@ -776,7 +777,7 @@ public abstract class TestWithFeService {
Env.getCurrentEnv().getSqlBlockRuleMgr()
.alterSqlBlockRule(new SqlBlockRule(command.getRuleName(), command.getSql(), command.getSqlHash(),
command.getPartitionNum(), command.getTabletNum(), command.getCardinality(),
- command.getGlobal(), command.getEnable()));
+ command.getRequirePartitionFilter(), command.getGlobal(), command.getEnable()));
}
protected void dropSqlBlockRule(String sql) throws Exception {
diff --git a/regression-test/suites/external_table_p0/hive/test_external_sql_block_rule.groovy b/regression-test/suites/external_table_p0/hive/test_external_sql_block_rule.groovy
index 4629d9583e3..0673df77d4f 100644
--- a/regression-test/suites/external_table_p0/hive/test_external_sql_block_rule.groovy
+++ b/regression-test/suites/external_table_p0/hive/test_external_sql_block_rule.groovy
@@ -25,6 +25,8 @@ suite("test_external_sql_block_rule", "external_docker,hive,external_docker_hive
String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
String hms_port = context.config.otherConfigs.get("hive2HmsPort")
String catalog_name = "test_hive2_external_sql_block_rule"
+ String db_name = "test_external_sql_block_rule_db"
+ String table_name = "test_external_sql_block_rule_tbl"
sql """drop catalog if exists ${catalog_name}"""
@@ -34,24 +36,59 @@ suite("test_external_sql_block_rule", "external_docker,hive,external_docker_hive
'hadoop.username' = 'hive'
);"""
- sql "use ${catalog_name}.`default`";
- qt_sql01 """select * from parquet_partition_table order by l_linenumber,l_orderkey limit 10;"""
+ sql """switch ${catalog_name}"""
+ sql """CREATE DATABASE IF NOT EXISTS ${catalog_name}.${db_name}"""
+ sql """use `${catalog_name}`.`${db_name}`"""
+ sql """DROP TABLE IF EXISTS ${table_name}"""
+ sql """
+ CREATE TABLE ${catalog_name}.${db_name}.${table_name} (
+ `id` INT,
+ `val` INT,
+ `pt1` STRING,
+ `pt2` STRING
+ ) ENGINE=hive
+ PARTITION BY LIST (pt1, pt2) ()
+ PROPERTIES (
+ 'file_format'='parquet'
+ )
+ """
+ sql """
+ INSERT INTO ${catalog_name}.${db_name}.${table_name} (id, val, pt1, pt2) VALUES
+ (1, 10, 'a', 'x'),
+ (2, 20, 'a', 'y'),
+ (3, 30, 'b', 'x'),
+ (4, 40, 'b', 'y')
+ """
+ def baseRows = sql """
+ SELECT id, val, pt1, pt2
+ FROM ${catalog_name}.${db_name}.${table_name}
+ ORDER BY id
+ """
+ assertEquals(4, baseRows.size())
+ assertEquals("1", baseRows[0][0].toString())
+ assertEquals("a", baseRows[0][2].toString())
+ assertEquals("4", baseRows[3][0].toString())
+ assertEquals("y", baseRows[3][3].toString())
// Clean up existing rules and users
sql """drop sql_block_rule if exists hive_partition_rule"""
sql """drop sql_block_rule if exists hive_split_rule"""
sql """drop sql_block_rule if exists hive_cardinality_rule"""
sql """drop sql_block_rule if exists hive_regex_rule"""
+ sql """drop sql_block_rule if exists hive_require_partition_filter_rule"""
sql """drop user if exists hive_block_user1"""
sql """drop user if exists hive_block_user2"""
sql """drop user if exists hive_block_user3"""
sql """drop user if exists hive_block_user4"""
+ sql """drop user if exists hive_block_user5"""
// Create non-global rules (won't affect other parallel tests)
sql """create sql_block_rule hive_partition_rule properties("partition_num" = "3", "global" = "false");"""
sql """create sql_block_rule hive_split_rule properties("tablet_num" = "3", "global" = "false");"""
sql """create sql_block_rule hive_cardinality_rule properties("cardinality" = "3", "global" = "false");"""
sql """create sql_block_rule hive_regex_rule properties("sql" = "SELECT \\\\*", "global" = "false");"""
+ sql """create sql_block_rule hive_require_partition_filter_rule
+ properties("require_partition_filter" = "true", "global" = "false");"""
// Create test users and bind rules
sql """create user hive_block_user1;"""
@@ -70,6 +107,10 @@ suite("test_external_sql_block_rule", "external_docker,hive,external_docker_hive
sql """SET PROPERTY FOR 'hive_block_user4' 'sql_block_rules' = 'hive_regex_rule';"""
sql """grant all on *.*.* to hive_block_user4;"""
+ sql """create user hive_block_user5;"""
+ sql """SET PROPERTY FOR 'hive_block_user5' 'sql_block_rules' = 'hive_require_partition_filter_rule';"""
+ sql """grant all on *.*.* to hive_block_user5;"""
+
// cloud-mode: grant cluster privileges
if (isCloudMode()) {
def clusters = sql " SHOW CLUSTERS; "
@@ -79,46 +120,74 @@ suite("test_external_sql_block_rule", "external_docker,hive,external_docker_hive
sql """GRANT USAGE_PRIV ON CLUSTER `${validCluster}` TO hive_block_user2;"""
sql """GRANT USAGE_PRIV ON CLUSTER `${validCluster}` TO hive_block_user3;"""
sql """GRANT USAGE_PRIV ON CLUSTER `${validCluster}` TO hive_block_user4;"""
+ sql """GRANT USAGE_PRIV ON CLUSTER `${validCluster}` TO hive_block_user5;"""
}
// Test 1: partition_num rule
connect('hive_block_user1', '', context.config.jdbcUrl) {
test {
- sql """select * from ${catalog_name}.`default`.parquet_partition_table order by l_linenumber limit 10;"""
+ sql """select * from ${catalog_name}.${db_name}.${table_name} order by id limit 10;"""
exception """sql hits sql block rule: hive_partition_rule, reach partition_num : 3"""
}
// Test EXPLAIN should not be blocked
- sql """explain select * from ${catalog_name}.`default`.parquet_partition_table order by l_linenumber limit 10;"""
+ sql """explain select * from ${catalog_name}.${db_name}.${table_name} order by id limit 10;"""
}
// Test 2: tablet_num (split) rule
connect('hive_block_user2', '', context.config.jdbcUrl) {
test {
- sql """select * from ${catalog_name}.`default`.parquet_partition_table order by l_linenumber limit 10;"""
+ sql """select * from ${catalog_name}.${db_name}.${table_name} order by id limit 10;"""
exception """sql hits sql block rule: hive_split_rule, reach tablet_num : 3"""
}
// Test EXPLAIN should not be blocked
- sql """explain select * from ${catalog_name}.`default`.parquet_partition_table order by l_linenumber limit 10;"""
+ sql """explain select * from ${catalog_name}.${db_name}.${table_name} order by id limit 10;"""
}
// Test 3: cardinality rule
connect('hive_block_user3', '', context.config.jdbcUrl) {
test {
- sql """select * from ${catalog_name}.`default`.parquet_partition_table order by l_linenumber limit 10;"""
+ sql """select * from ${catalog_name}.${db_name}.${table_name} order by id limit 10;"""
exception """sql hits sql block rule: hive_cardinality_rule, reach cardinality : 3"""
}
// Test EXPLAIN should not be blocked
- sql """explain select * from ${catalog_name}.`default`.parquet_partition_table order by l_linenumber limit 10;"""
+ sql """explain select * from ${catalog_name}.${db_name}.${table_name} order by id limit 10;"""
}
// Test 4: regex rule
connect('hive_block_user4', '', context.config.jdbcUrl) {
test {
- sql """SELECT * FROM ${catalog_name}.`default`.parquet_partition_table limit 10;"""
+ sql """SELECT * FROM ${catalog_name}.${db_name}.${table_name} limit 10;"""
exception """sql match regex sql block rule: hive_regex_rule"""
}
// Test EXPLAIN should not be blocked by regex rule
- sql """EXPLAIN SELECT * FROM ${catalog_name}.`default`.parquet_partition_table limit 10;"""
+ sql """EXPLAIN SELECT * FROM ${catalog_name}.${db_name}.${table_name} limit 10;"""
+ }
+
+ // Test 5: require_partition_filter rule on Hive partition table
+ connect('hive_block_user5', '', context.config.jdbcUrl) {
+ test {
+ sql """select * from ${catalog_name}.${db_name}.${table_name} order by id limit 10;"""
+ exception """sql hits sql block rule: hive_require_partition_filter_rule, missing partition filter"""
+ }
+
+ test {
+ sql """select * from ${catalog_name}.${db_name}.${table_name}
+ where val = 10 limit 10;"""
+ exception """sql hits sql block rule: hive_require_partition_filter_rule, missing partition filter"""
+ }
+
+ def partitionFilteredRows = sql """
+ select id, pt1, pt2 from ${catalog_name}.${db_name}.${table_name}
+ where pt1 = 'a'
+ order by id;
+ """
+ assertEquals(2, partitionFilteredRows.size())
+ assertEquals("1", partitionFilteredRows[0][0].toString())
+ assertEquals("2", partitionFilteredRows[1][0].toString())
+
+ // Test EXPLAIN should not be blocked
+ sql """explain select * from ${catalog_name}.${db_name}.${table_name}
+ where pt1 = 'a' order by id;"""
}
// Cleanup
@@ -126,9 +195,13 @@ suite("test_external_sql_block_rule", "external_docker,hive,external_docker_hive
sql """drop user if exists hive_block_user2"""
sql """drop user if exists hive_block_user3"""
sql """drop user if exists hive_block_user4"""
+ sql """drop user if exists hive_block_user5"""
sql """drop sql_block_rule if exists hive_partition_rule"""
sql """drop sql_block_rule if exists hive_split_rule"""
sql """drop sql_block_rule if exists hive_cardinality_rule"""
sql """drop sql_block_rule if exists hive_regex_rule"""
+ sql """drop sql_block_rule if exists hive_require_partition_filter_rule"""
+ sql """DROP TABLE IF EXISTS ${catalog_name}.${db_name}.${table_name}"""
+ sql """DROP DATABASE IF EXISTS ${catalog_name}.${db_name}"""
sql """drop catalog if exists ${catalog_name}"""
}
diff --git a/regression-test/suites/query_p0/schema_table/test_sql_block_rule_status.groovy b/regression-test/suites/query_p0/schema_table/test_sql_block_rule_status.groovy
index e9c38c4b94c..f7eb9970628 100644
--- a/regression-test/suites/query_p0/schema_table/test_sql_block_rule_status.groovy
+++ b/regression-test/suites/query_p0/schema_table/test_sql_block_rule_status.groovy
@@ -54,7 +54,16 @@ suite("test_sql_block_rule_status") {
exception "sql match"
}
order_qt_count "SELECT count(*) FROM information_schema.sql_block_rule_status where name ='${blockRuleName}'"
- order_qt_select "SELECT NAME,PATTERN,SQL_HASH,PARTITION_NUM,TABLET_NUM,CARDINALITY,GLOBAL,ENABLE,BLOCKS FROM information_schema.sql_block_rule_status where name ='${blockRuleName}'"
+ def statusRows = sql """
+ SELECT NAME, PATTERN, SQL_HASH, PARTITION_NUM, TABLET_NUM, CARDINALITY, GLOBAL, ENABLE,
+ REQUIRE_PARTITION_FILTER, BLOCKS
+ FROM information_schema.sql_block_rule_status
+ WHERE name ='${blockRuleName}'
+ """
+ assertEquals(1, statusRows.size())
+ assertEquals(blockRuleName, statusRows[0][0].toString())
+ assertEquals("false", statusRows[0][8].toString())
+ assertEquals("1", statusRows[0][9].toString())
sql """
drop SQL_BLOCK_RULE if exists ${blockRuleName};
"""
@@ -62,4 +71,3 @@ suite("test_sql_block_rule_status") {
drop table if exists ${tableName};
"""
}
-
diff --git a/regression-test/suites/sql_block_rule_p0/test_sql_block_rule.groovy b/regression-test/suites/sql_block_rule_p0/test_sql_block_rule.groovy
index e24cfe2d71a..29645fe9ed5 100644
--- a/regression-test/suites/sql_block_rule_p0/test_sql_block_rule.groovy
+++ b/regression-test/suites/sql_block_rule_p0/test_sql_block_rule.groovy
@@ -21,6 +21,10 @@ suite("test_sql_block_rule", "nonConcurrent") {
DROP SQL_BLOCK_RULE if exists test_rule_partition
"""
+ sql """
+ DROP SQL_BLOCK_RULE if exists test_rule_require_partition_filter
+ """
+
sql """
DROP SQL_BLOCK_RULE if exists test_rule_tablet
"""
@@ -77,19 +81,21 @@ suite("test_sql_block_rule", "nonConcurrent") {
checkNereidsExecute("SHOW SQL_BLOCK_RULE")
- qt_select1 """
- SHOW SQL_BLOCK_RULE
- """
+ def showRuleRows = sql """SHOW SQL_BLOCK_RULE"""
+ assertEquals(1, showRuleRows.size())
+ assertEquals("test_rule_sql", showRuleRows[0][0])
+ assertEquals("SELECT abcd FROM table_2", showRuleRows[0][1])
+ assertEquals("true", showRuleRows[0][6].toString())
+ assertEquals("true", showRuleRows[0][7].toString())
+ assertEquals("false", showRuleRows[0][8].toString())
- qt_select2 """
- SHOW SQL_BLOCK_RULE FOR test_rule_sql
- """
+ def showSingleRuleRows = sql """SHOW SQL_BLOCK_RULE FOR test_rule_sql"""
+ assertEquals(1, showSingleRuleRows.size())
+ assertEquals(showRuleRows[0], showSingleRuleRows[0])
checkNereidsExecute("DROP SQL_BLOCK_RULE if exists test_rule_sql")
- qt_select3_notexist """
- SHOW SQL_BLOCK_RULE
- """
+ assertTrue(sql("""SHOW SQL_BLOCK_RULE""").isEmpty())
sql """
CREATE SQL_BLOCK_RULE if not exists test_rule_sql
@@ -100,9 +106,12 @@ suite("test_sql_block_rule", "nonConcurrent") {
PROPERTIES("sql"="SELECT \\\\* FROM table_2", "global"= "true", "enable"= "true")
"""
- order_qt_select4_exist """
- SHOW SQL_BLOCK_RULE
- """
+ def showAllRuleRows = sql("""SHOW SQL_BLOCK_RULE""").sort { a, b -> a[0] <=> b[0] }
+ assertEquals(2, showAllRuleRows.size())
+ assertEquals("test_rule_sql", showAllRuleRows[0][0])
+ assertEquals("false", showAllRuleRows[0][8].toString())
+ assertEquals("test_rule_sql1", showAllRuleRows[1][0])
+ assertEquals("false", showAllRuleRows[1][8].toString())
sql """
DROP SQL_BLOCK_RULE if exists test_rule_sql,test_rule_sql1
@@ -122,9 +131,7 @@ suite("test_sql_block_rule", "nonConcurrent") {
exception "sql hits sql block rule: test_rule_num, reach tablet_num : 1"
}
*/
- qt_select5_not_exist """
- SHOW SQL_BLOCK_RULE
- """
+ assertTrue(sql("""SHOW SQL_BLOCK_RULE""").isEmpty())
sql """
DROP SQL_BLOCK_RULE if exists test_rule_num
@@ -223,6 +230,80 @@ suite("test_sql_block_rule", "nonConcurrent") {
"""
}
+ sql """delete from table_2 where abcd = '1'"""
+ sql """
+ CREATE SQL_BLOCK_RULE if not exists test_rule_require_partition_filter PROPERTIES (
+ "require_partition_filter" = "true",
+ "global" = "true",
+ "enable" = "true");
+ """
+ try {
+ test {
+ sql("""SELECT * FROM a_partitioned_table_for_sql_block_rule;""", false)
+ exception """sql hits sql block rule: test_rule_require_partition_filter, missing partition filter"""
+ }
+
+ test {
+ sql("""SELECT * FROM a_partitioned_table_for_sql_block_rule WHERE val = 5;""", false)
+ exception """sql hits sql block rule: test_rule_require_partition_filter, missing partition filter"""
+ }
+
+ def filteredPartitionRows = sql """
+ SELECT id
+ FROM a_partitioned_table_for_sql_block_rule
+ WHERE id = 1
+ ORDER BY id
+ """
+ assertEquals(1, filteredPartitionRows.size())
+ assertEquals("1", filteredPartitionRows[0][0].toString())
+
+ def explicitlySelectedPartitionRows = sql """
+ SELECT id
+ FROM a_partitioned_table_for_sql_block_rule PARTITION(p1)
+ ORDER BY id
+ """
+ assertEquals(1, explicitlySelectedPartitionRows.size())
+ assertEquals("1", explicitlySelectedPartitionRows[0][0].toString())
+
+ def tabletsInPartition = sql """SHOW TABLETS FROM a_partitioned_table_for_sql_block_rule PARTITION(p1)"""
+ assertTrue(tabletsInPartition.size() > 0)
+ sql """
+ SELECT count(*)
+ FROM a_partitioned_table_for_sql_block_rule TABLET(${tabletsInPartition[0][0]})
+ """
+
+ test {
+ sql("""
+ INSERT INTO table_2
+ SELECT cast(id as varchar(150)), cast('2024-01-01 00:00:00' as datetime)
+ FROM a_partitioned_table_for_sql_block_rule
+ """, false)
+ exception """sql hits sql block rule: test_rule_require_partition_filter, missing partition filter"""
+ }
+
+ sql """
+ INSERT INTO table_2
+ SELECT cast(id as varchar(150)), cast('2024-01-01 00:00:00' as datetime)
+ FROM a_partitioned_table_for_sql_block_rule
+ WHERE id = 1
+ """
+ def insertedRows = sql """
+ SELECT count(*)
+ FROM table_2
+ WHERE abcd = '1' AND create_time = '2024-01-01 00:00:00'
+ """
+ assertEquals("1", insertedRows[0][0].toString())
+
+ def showPartitionFilterRuleRows = sql """SHOW SQL_BLOCK_RULE FOR test_rule_require_partition_filter"""
+ assertEquals(1, showPartitionFilterRuleRows.size())
+ assertEquals("true", showPartitionFilterRuleRows[0][8].toString())
+ } finally {
+ sql """
+ drop SQL_BLOCK_RULE if exists test_rule_require_partition_filter;
+ """
+ sql """delete from table_2 where abcd = '1'"""
+ }
+
sql """
CREATE SQL_BLOCK_RULE if not exists test_rule_tablet PROPERTIES ( "tablet_num" = "3", "global" = "true",
"enable"="true");
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org