You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2023/06/21 05:12:42 UTC
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
BiteTheDDDDt opened a new pull request, #21067:
URL: https://github.com/apache/doris/pull/21067
## Proposed changes
support query match mv with agg_state on nereids planner
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1600249348
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1614763745
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1614529245
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt merged pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #21067:
URL: https://github.com/apache/doris/pull/21067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1600795524
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.45 seconds
stream load tsv: 453 seconds loaded 74807831229 Bytes, about 157 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 69.9 seconds inserted 10000000 Rows, about 143K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230621125133_clickbench_pr_166242.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #21067:
URL: https://github.com/apache/doris/pull/21067#discussion_r1247533577
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -251,6 +252,21 @@ public static Optional<Slot> extractSlotOrCastOnSlot(Expression expr) {
}
}
+ /**
+ * get all slot child of expr
+ */
+ public static List<Slot> getAllSlot(Expression expr) {
Review Comment:
u could use `Expression.collect()` directly: `expr.collect(SlotRefenrece.class::isInstance)`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java:
##########
@@ -1391,6 +1395,36 @@ public Expression visitNdv(Ndv ndv, RewriteContext context) {
}
return ndv;
}
+
+ /**
+ * agg(col) -> agg_merge(mva_generic_aggregation__agg_state(col)) eg: max_by(k2,
+ * k3) -> max_by_merge(mva_generic_aggregation__max_by_state(k2, k3))
+ */
+ @Override
+ public Expression visitAggregateFunction(AggregateFunction aggregateFunction, RewriteContext context) {
Review Comment:
we have override some agg function in this Rewriter, such as `ndv`. So, if we have a `ndv(xx)` in sql, then it cannot processed by this function by default. if u want `visitAggregateFunction` process all aggregate function, u should call this function in `visitXXX` in this Rewriter explicitly. for example
```java
public Expression visitNdv(Ndv ndv, RewriteContext context) {
Expression newNdv = visitAggregateFunction(ndv, context);
if (newNdv != ndv) {
return newNdv;
}
...
}
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java:
##########
@@ -437,9 +437,7 @@ private void analyzeSubquery(Analyzer analyzer) throws UserException {
mentionedColumns.add(col.getName());
targetColumns.add(col);
}
- realTargetColumnNames = targetColumns.stream().map(column -> column.getName()).collect(Collectors.toList());
} else {
- realTargetColumnNames = targetColumnNames;
Review Comment:
if the logical of insert changed, please ensure Nereids' insert could be work well, to test Nereids' dml, set session variable 'enable_nereids_dml' to `true`
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java:
##########
@@ -453,8 +451,8 @@ private void analyzeSubquery(Analyzer analyzer) throws UserException {
// hll column mush in mentionedColumns
for (Column col : targetTable.getBaseSchema()) {
if (col.getType().isObjectStored() && !mentionedColumns.contains(col.getName())) {
- throw new AnalysisException(" object-stored column " + col.getName()
- + " must in insert into columns");
+ throw new AnalysisException(
+ " object-stored column " + col.getName() + " mush in insert into columns");
Review Comment:
why change this? BTW, typo: must -> mush
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1600404083
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1614535278
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1614697065
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 41.21 seconds
stream load tsv: 455 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 70.4 seconds inserted 10000000 Rows, about 142K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230630140004_clickbench_pr_170586.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1614763682
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1605948416
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1606010618
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.51 seconds
stream load tsv: 458 seconds loaded 74807831229 Bytes, about 155 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 68.4 seconds inserted 10000000 Rows, about 146K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230625101715_clickbench_pr_167124.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1600620626
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #21067: [Feature](materialized view) support query match mv with agg_state on nereids planner
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #21067:
URL: https://github.com/apache/doris/pull/21067#issuecomment-1612390973
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org