You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/12/21 09:49:34 UTC
[GitHub] [doris] Kikyou1997 opened a new pull request, #15243: [feature](nereids) Support query on specific partitions
Kikyou1997 opened a new pull request, #15243:
URL: https://github.com/apache/doris/pull/15243
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #15243: [feature](nereids) Support query on specific partitions
Posted by GitBox <gi...@apache.org>.
morrySnow merged PR #15243:
URL: https://github.com/apache/doris/pull/15243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15243: [feature](nereids) Support query on specific partitions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15243:
URL: https://github.com/apache/doris/pull/15243#issuecomment-1365275072
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #15243: [feature](nereids) Support query on specific partitions
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #15243:
URL: https://github.com/apache/doris/pull/15243#discussion_r1056117739
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundRelation.java:
##########
@@ -44,34 +43,21 @@
*/
public class UnboundRelation extends LogicalRelation implements Unbound {
private final List<String> nameParts;
+ private final List<String> partNames;
public UnboundRelation(RelationId id, List<String> nameParts) {
- this(id, nameParts, Optional.empty(), Optional.empty());
+ this(id, nameParts, Optional.empty(), Optional.empty(), Collections.emptyList());
}
- public UnboundRelation(RelationId id, List<String> nameParts, Optional<GroupExpression> groupExpression,
- Optional<LogicalProperties> logicalProperties) {
- super(id, PlanType.LOGICAL_UNBOUND_RELATION, groupExpression, logicalProperties);
- this.nameParts = nameParts;
- }
-
- public UnboundRelation(RelationId id, TableIdentifier identifier) {
- this(id, identifier, Optional.empty(), Optional.empty());
+ public UnboundRelation(RelationId id, List<String> nameParts, List<String> partNames) {
+ this(id, nameParts, Optional.empty(), Optional.empty(), partNames);
}
- /**
- * Constructor for UnboundRelation.
- *
- * @param identifier relation identifier
- */
- public UnboundRelation(RelationId id, TableIdentifier identifier, Optional<GroupExpression> groupExpression,
- Optional<LogicalProperties> logicalProperties) {
+ public UnboundRelation(RelationId id, List<String> nameParts, Optional<GroupExpression> groupExpression,
+ Optional<LogicalProperties> logicalProperties, List<String> partNames) {
super(id, PlanType.LOGICAL_UNBOUND_RELATION, groupExpression, logicalProperties);
- this.nameParts = Lists.newArrayList();
- if (identifier.getDatabaseName().isPresent()) {
- nameParts.add(identifier.getDatabaseName().get());
- }
- nameParts.add(identifier.getTableName());
+ this.nameParts = nameParts;
+ this.partNames = partNames;
Review Comment:
```suggestion
this.partNames = ImmutableList.copyOf(Objects.requireNotNull(partNames, "partNames should not null"));
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -410,7 +412,18 @@ private LogicalPlan withCheckPolicy(LogicalPlan plan) {
@Override
public LogicalPlan visitTableName(TableNameContext ctx) {
List<String> tableId = visitMultipartIdentifier(ctx.multipartIdentifier());
- LogicalPlan checkedRelation = withCheckPolicy(new UnboundRelation(RelationUtil.newRelationId(), tableId));
+ List<String> partitionNames = new ArrayList<>();
+ if (ctx.specified_partition() != null) {
+ if (ctx.specified_partition().identifier() != null) {
+ partitionNames.add(ctx.specified_partition().identifier().getText());
+ } else {
+ partitionNames.addAll(Arrays
+ .stream(ctx.specified_partition().identifierList().identifierSeq().getText().split(","))
+ .collect(Collectors.toList()));
Review Comment:
```suggestion
partitionNames.addAll(visitIdentifierList(ctx.specified_partition().identifierList()));
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java:
##########
@@ -126,6 +135,7 @@ public LogicalOlapScan(RelationId id, Table table, List<String> qualifier,
this.selectedIndexId = selectedIndexId <= 0 ? getTable().getBaseIndexId() : selectedIndexId;
this.indexSelected = indexSelected;
this.preAggStatus = preAggStatus;
+ this.manuallySpecifiedPartitions = partitions;
Review Comment:
```suggestion
this.manuallySpecifiedPartitions = ImmutableList.copyOf(partitions);
```
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -210,7 +210,7 @@ identifierSeq
;
relationPrimary
- : multipartIdentifier tableAlias #tableName
+ : multipartIdentifier specified_partition? tableAlias #tableName
Review Comment:
```suggestion
: multipartIdentifier specifiedPartition? tableAlias #tableName
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java:
##########
@@ -101,13 +103,19 @@ public LogicalOlapScan(RelationId id, OlapTable table, List<String> qualifier) {
this(id, table, qualifier, Optional.empty(), Optional.empty(),
table.getPartitionIds(), false,
ImmutableList.of(), false,
- -1, false, PreAggStatus.on());
+ -1, false, PreAggStatus.on(), Collections.emptyList());
+ }
+
+ public LogicalOlapScan(RelationId id, OlapTable table, List<String> qualifier, List<Long> specifiedPartitions) {
+ this(id, table, qualifier, Optional.empty(), Optional.empty(),
+ specifiedPartitions, false, ImmutableList.of(), false,
+ -1, false, PreAggStatus.on(), specifiedPartitions);
}
public LogicalOlapScan(RelationId id, Table table, List<String> qualifier) {
this(id, table, qualifier, Optional.empty(), Optional.empty(),
((OlapTable) table).getPartitionIds(), false, ImmutableList.of(), false,
- -1, false, PreAggStatus.on());
+ -1, false, PreAggStatus.on(), Collections.emptyList());
Review Comment:
```suggestion
-1, false, PreAggStatus.on(), ImmutableList.of());
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java:
##########
@@ -80,25 +86,39 @@ private LogicalPlan bindWithCurrentDb(CascadesContext cascadesContext, String ta
String dbName = cascadesContext.getConnectContext().getDatabase();
Table table = cascadesContext.getTable(dbName, tableName, cascadesContext.getConnectContext().getEnv());
// TODO: should generate different Scan sub class according to table's type
+ List<Long> partIds = getPartitionIds(table, r);
if (table.getType() == TableType.OLAP) {
- return new LogicalOlapScan(RelationUtil.newRelationId(), (OlapTable) table, ImmutableList.of(dbName));
+ if (!CollectionUtils.isEmpty(partIds)) {
+ return new LogicalOlapScan(RelationUtil.newRelationId(),
+ (OlapTable) table, ImmutableList.of(dbName), partIds);
+ } else {
+ return new LogicalOlapScan(RelationUtil.newRelationId(),
+ (OlapTable) table, ImmutableList.of(dbName));
+ }
} else if (table.getType() == TableType.VIEW) {
Plan viewPlan = parseAndAnalyzeView(table.getDdlSql(), cascadesContext);
return new LogicalSubQueryAlias<>(table.getName(), viewPlan);
}
throw new AnalysisException("Unsupported tableType:" + table.getType());
}
- private LogicalPlan bindWithDbNameFromNamePart(CascadesContext cascadesContext, List<String> nameParts) {
+ private LogicalPlan bindWithDbNameFromNamePart(CascadesContext cascadesContext, UnboundRelation r) {
Review Comment:
ditto
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java:
##########
@@ -52,19 +57,20 @@ public Rule build() {
switch (nameParts.size()) {
case 1: { // table
// Use current database name from catalog.
- return bindWithCurrentDb(ctx.cascadesContext, nameParts.get(0));
+ return bindWithCurrentDb(ctx.cascadesContext, ctx.root);
}
case 2: { // db.table
// Use database name from table name parts.
- return bindWithDbNameFromNamePart(ctx.cascadesContext, nameParts);
+ return bindWithDbNameFromNamePart(ctx.cascadesContext, ctx.root);
}
default:
throw new IllegalStateException("Table name [" + ctx.root.getTableName() + "] is invalid.");
}
}).toRule(RuleType.BINDING_RELATION);
}
- private LogicalPlan bindWithCurrentDb(CascadesContext cascadesContext, String tableName) {
+ private LogicalPlan bindWithCurrentDb(CascadesContext cascadesContext, UnboundRelation r) {
Review Comment:
```suggestion
private LogicalPlan bindWithCurrentDb(CascadesContext cascadesContext, UnboundRelation unboundRelation) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15243: [feature](nereids) Support query on specific partitions
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #15243:
URL: https://github.com/apache/doris/pull/15243#issuecomment-1361299960
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.75 seconds
load time: 629 seconds
storage size: 17123253915 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221221131522_clickbench_pr_66370.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15243: [feature](nereids) Support query on specific partitions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15243:
URL: https://github.com/apache/doris/pull/15243#issuecomment-1365275057
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #15243: [feature](nereids) Support query on specific partitions
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #15243:
URL: https://github.com/apache/doris/pull/15243#discussion_r1057061380
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ResolveOrdinalInOrderByAndGroupBy.java:
##########
@@ -74,11 +72,6 @@ public List<Rule> buildRules() {
IntegerLikeLiteral i = (IntegerLikeLiteral) groupByExpr;
int ord = i.getIntValue();
Expression aggExpr = aggOutput.get(ord - 1);
- if (!CollectionUtils.isEmpty(aggExpr.children())
- && aggExpr.child(0) instanceof Literal) {
- continue;
- }
- checkOrd(ord, aggOutput.size());
Review Comment:
why remove these codes in this PR?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateGroupByConstant.java:
##########
@@ -50,12 +50,18 @@ public Rule build() {
List<Expression> groupByExprs = aggregate.getGroupByExpressions();
List<NamedExpression> outputExprs = aggregate.getOutputExpressions();
Set<Expression> slotGroupByExprs = Sets.newLinkedHashSet();
+ Expression lit = null;
for (Expression expression : groupByExprs) {
expression = FoldConstantRule.INSTANCE.rewrite(expression);
if (!(expression instanceof Literal)) {
slotGroupByExprs.add(expression);
+ } else {
+ lit = expression;
}
}
+ if (slotGroupByExprs.isEmpty() && lit != null) {
+ slotGroupByExprs.add(lit);
+ }
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org