You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "LiBinfeng-01 (via GitHub)" <gi...@apache.org> on 2023/06/26 06:22:41 UTC
[GitHub] [doris] LiBinfeng-01 opened a new pull request, #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
LiBinfeng-01 opened a new pull request, #21171:
URL: https://github.com/apache/doris/pull/21171
## Proposed changes
Problem:
When inferring predicate, we assume that slot reference need to be inferred. But in this case:
carete table tb1(l1 smallint) ...;
create table tb2(l2 int) ...;
select * from tb1 inner join tb2 where tb1.l1 = tb2.l2 and tb2.l2 = 1;
We can not get tb1.l1 = 1 filter because we will add a cast to l1 (Cast smallint to int l1) = l2.
Solved:
Add cast consideration when inferring predicate, also add change judgement when judging equals to slotreference and cast expression
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1621173091
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.08 seconds
stream load tsv: 457 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 56 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 68.2 seconds inserted 10000000 Rows, about 146K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230705071925_clickbench_pr_172566.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1623412191
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 52.65 seconds
stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 90.5 seconds inserted 10000000 Rows, about 110K ops/s
storage size: 17167473914 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230706181641_clickbench_pr_173630.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1638079125
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1621961733
(From new mechine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 56.08 seconds
stream load tsv: 516 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 87.3 seconds inserted 10000000 Rows, about 114K ops/s
storage size: 17167412237 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230705231101_clickbench_pr_172941.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1621810332
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #21171:
URL: https://github.com/apache/doris/pull/21171#discussion_r1251388990
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
Review Comment:
get
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #21171:
URL: https://github.com/apache/doris/pull/21171#discussion_r1261934280
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -70,19 +72,44 @@ public Expression visit(Expression expr, Void context) {
@Override
public Expression visitComparisonPredicate(ComparisonPredicate cp, Void context) {
- if (cp.left().isSlot() && cp.right().isConstant()) {
- return replaceSlot(cp);
- } else if (cp.left().isConstant() && cp.right().isSlot()) {
- return replaceSlot(cp);
+ if (isExpressionSlotCoveredByCast(cp.left()) && (cp.right().isConstant())) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.left()));
+ } else if (isExpressionSlotCoveredByCast(cp.right()) && cp.left().isConstant()) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.right()));
}
return super.visit(cp, context);
}
- private Expression replaceSlot(Expression expr) {
+ private boolean isTwoExpressionEqualWithCast(Expression left, Expression right) {
+ return getSlotCoveredByCast(left).equals(getSlotCoveredByCast(right));
+ }
+
+ private boolean canCompareType(DataType dataType) {
+ return dataType.isBigIntType() || dataType.isIntegerType() || dataType.isSmallIntType()
+ || dataType.isTinyIntType() || dataType.isLargeIntType();
+ }
+
+ private boolean isOriginDataTypeBigger(DataType originDataType, Expression expr) {
+ if (canCompareType(leftSlotEqualToRightSlot.child(0).getDataType())
+ && canCompareType(leftSlotEqualToRightSlot.child(1).getDataType())
+ && canCompareType(originDataType)) {
+ // infer filter can not be lower than original datatype, or dataset would be wrong
+ if (originDataType.width() > leftSlotEqualToRightSlot.child(0).getDataType().width()
+ || originDataType.width() > leftSlotEqualToRightSlot.child(1).getDataType().width()) {
Review Comment:
use width not a good idea. maybe u should add widerThan or similar method to IntegralType
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -92,13 +119,48 @@ private Expression replaceSlot(Expression expr) {
}, null);
}
+ /**
+ * judge if expression is slot covered by cast
+ * example: cast(cast(table.columnA))
+ */
+ private boolean isExpressionSlotCoveredByCast(Expression expr) {
+ if (expr instanceof Cast) {
+ return isExpressionSlotCoveredByCast(((Cast) expr).child());
+ }
+ return expr instanceof SlotReference;
+ }
+
+ /**
+ * get slot covered by cast
+ * example: input: cast(cast(table.columnA)) output: columnA.datatype
+ *
+ */
+ private DataType getDatatypeCoveredByCast(Expression expr) {
+ if (expr instanceof Cast) {
+ return getDatatypeCoveredByCast(((Cast) expr).child());
+ }
+ return expr.getDataType();
+ }
+
+ /**
+ * get slot covered by cast
+ * example: input: cast(cast(table.columnA)) output: table.columnA
+ */
+ private Expression getSlotCoveredByCast(Expression cast) {
+ if (cast instanceof Cast) {
+ return getSlotCoveredByCast(((Cast) cast).child());
+ }
+ return cast;
+ }
Review Comment:
move it into ExpressionUtils
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -92,13 +119,48 @@ private Expression replaceSlot(Expression expr) {
}, null);
}
+ /**
+ * judge if expression is slot covered by cast
+ * example: cast(cast(table.columnA))
+ */
+ private boolean isExpressionSlotCoveredByCast(Expression expr) {
+ if (expr instanceof Cast) {
+ return isExpressionSlotCoveredByCast(((Cast) expr).child());
+ }
+ return expr instanceof SlotReference;
+ }
Review Comment:
move it into ExpressionUtils
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -70,19 +72,44 @@ public Expression visit(Expression expr, Void context) {
@Override
public Expression visitComparisonPredicate(ComparisonPredicate cp, Void context) {
- if (cp.left().isSlot() && cp.right().isConstant()) {
- return replaceSlot(cp);
- } else if (cp.left().isConstant() && cp.right().isSlot()) {
- return replaceSlot(cp);
+ if (isExpressionSlotCoveredByCast(cp.left()) && (cp.right().isConstant())) {
Review Comment:
add comments to explain why need `isExpressionSlotCoveredByCast` and `getDatatypeCoveredByCast `
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -70,19 +72,44 @@ public Expression visit(Expression expr, Void context) {
@Override
public Expression visitComparisonPredicate(ComparisonPredicate cp, Void context) {
- if (cp.left().isSlot() && cp.right().isConstant()) {
- return replaceSlot(cp);
- } else if (cp.left().isConstant() && cp.right().isSlot()) {
- return replaceSlot(cp);
+ if (isExpressionSlotCoveredByCast(cp.left()) && (cp.right().isConstant())) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.left()));
+ } else if (isExpressionSlotCoveredByCast(cp.right()) && cp.left().isConstant()) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.right()));
}
return super.visit(cp, context);
}
- private Expression replaceSlot(Expression expr) {
+ private boolean isTwoExpressionEqualWithCast(Expression left, Expression right) {
+ return getSlotCoveredByCast(left).equals(getSlotCoveredByCast(right));
+ }
Review Comment:
move it into ExpressionUtils
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -70,19 +72,44 @@ public Expression visit(Expression expr, Void context) {
@Override
public Expression visitComparisonPredicate(ComparisonPredicate cp, Void context) {
- if (cp.left().isSlot() && cp.right().isConstant()) {
- return replaceSlot(cp);
- } else if (cp.left().isConstant() && cp.right().isSlot()) {
- return replaceSlot(cp);
+ if (isExpressionSlotCoveredByCast(cp.left()) && (cp.right().isConstant())) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.left()));
+ } else if (isExpressionSlotCoveredByCast(cp.right()) && cp.left().isConstant()) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.right()));
}
return super.visit(cp, context);
}
- private Expression replaceSlot(Expression expr) {
+ private boolean isTwoExpressionEqualWithCast(Expression left, Expression right) {
+ return getSlotCoveredByCast(left).equals(getSlotCoveredByCast(right));
+ }
+
+ private boolean canCompareType(DataType dataType) {
Review Comment:
if u want get all integer type , u could use `instance of IntegralType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1624500666
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.7 seconds
stream load tsv: 456 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 67.7 seconds inserted 10000000 Rows, about 147K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707011857_clickbench_pr_174031.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1606911071
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1637308325
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 54.66 seconds
stream load tsv: 503 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17169748213 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1621862112
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.46 seconds
stream load tsv: 457 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 56 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 67.7 seconds inserted 10000000 Rows, about 147K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230705142027_clickbench_pr_172936.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #21171:
URL: https://github.com/apache/doris/pull/21171#discussion_r1264792387
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -70,19 +72,44 @@ public Expression visit(Expression expr, Void context) {
@Override
public Expression visitComparisonPredicate(ComparisonPredicate cp, Void context) {
- if (cp.left().isSlot() && cp.right().isConstant()) {
- return replaceSlot(cp);
- } else if (cp.left().isConstant() && cp.right().isSlot()) {
- return replaceSlot(cp);
+ if (isExpressionSlotCoveredByCast(cp.left()) && (cp.right().isConstant())) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.left()));
+ } else if (isExpressionSlotCoveredByCast(cp.right()) && cp.left().isConstant()) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.right()));
}
return super.visit(cp, context);
}
- private Expression replaceSlot(Expression expr) {
+ private boolean isTwoExpressionEqualWithCast(Expression left, Expression right) {
+ return getSlotCoveredByCast(left).equals(getSlotCoveredByCast(right));
+ }
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -92,13 +119,48 @@ private Expression replaceSlot(Expression expr) {
}, null);
}
+ /**
+ * judge if expression is slot covered by cast
+ * example: cast(cast(table.columnA))
+ */
+ private boolean isExpressionSlotCoveredByCast(Expression expr) {
+ if (expr instanceof Cast) {
+ return isExpressionSlotCoveredByCast(((Cast) expr).child());
+ }
+ return expr instanceof SlotReference;
+ }
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -92,13 +119,48 @@ private Expression replaceSlot(Expression expr) {
}, null);
}
+ /**
+ * judge if expression is slot covered by cast
+ * example: cast(cast(table.columnA))
+ */
+ private boolean isExpressionSlotCoveredByCast(Expression expr) {
+ if (expr instanceof Cast) {
+ return isExpressionSlotCoveredByCast(((Cast) expr).child());
+ }
+ return expr instanceof SlotReference;
+ }
+
+ /**
+ * get slot covered by cast
+ * example: input: cast(cast(table.columnA)) output: columnA.datatype
+ *
+ */
+ private DataType getDatatypeCoveredByCast(Expression expr) {
+ if (expr instanceof Cast) {
+ return getDatatypeCoveredByCast(((Cast) expr).child());
+ }
+ return expr.getDataType();
+ }
+
+ /**
+ * get slot covered by cast
+ * example: input: cast(cast(table.columnA)) output: table.columnA
+ */
+ private Expression getSlotCoveredByCast(Expression cast) {
+ if (cast instanceof Cast) {
+ return getSlotCoveredByCast(((Cast) cast).child());
+ }
+ return cast;
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #21171:
URL: https://github.com/apache/doris/pull/21171#discussion_r1252604110
##########
regression-test/suites/nereids_p0/infer_predicate/load.groovy:
##########
@@ -0,0 +1,22 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("load") {
+ sql 'create database if not exists nereids_infer_predicate_test'
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1608688480
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1637290404
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1624628301
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.53 seconds
stream load tsv: 502 seconds loaded 74807831229 Bytes, about 142 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 88.8 seconds inserted 10000000 Rows, about 112K ops/s
storage size: 17166995123 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707114339_clickbench_pr_174139.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1624672614
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.61 seconds
stream load tsv: 449 seconds loaded 74807831229 Bytes, about 158 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 68.8 seconds inserted 10000000 Rows, about 145K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707041006_clickbench_pr_174123.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1624585762
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1611033496
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1624716064
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.13 seconds
stream load tsv: 447 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select: 68.7 seconds inserted 10000000 Rows, about 145K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230707043705_clickbench_pr_174136.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1624583533
run clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #21171:
URL: https://github.com/apache/doris/pull/21171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #21171:
URL: https://github.com/apache/doris/pull/21171#discussion_r1264792470
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -70,19 +72,44 @@ public Expression visit(Expression expr, Void context) {
@Override
public Expression visitComparisonPredicate(ComparisonPredicate cp, Void context) {
- if (cp.left().isSlot() && cp.right().isConstant()) {
- return replaceSlot(cp);
- } else if (cp.left().isConstant() && cp.right().isSlot()) {
- return replaceSlot(cp);
+ if (isExpressionSlotCoveredByCast(cp.left()) && (cp.right().isConstant())) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.left()));
+ } else if (isExpressionSlotCoveredByCast(cp.right()) && cp.left().isConstant()) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.right()));
}
return super.visit(cp, context);
}
- private Expression replaceSlot(Expression expr) {
+ private boolean isTwoExpressionEqualWithCast(Expression left, Expression right) {
+ return getSlotCoveredByCast(left).equals(getSlotCoveredByCast(right));
+ }
+
+ private boolean canCompareType(DataType dataType) {
+ return dataType.isBigIntType() || dataType.isIntegerType() || dataType.isSmallIntType()
+ || dataType.isTinyIntType() || dataType.isLargeIntType();
+ }
+
+ private boolean isOriginDataTypeBigger(DataType originDataType, Expression expr) {
+ if (canCompareType(leftSlotEqualToRightSlot.child(0).getDataType())
+ && canCompareType(leftSlotEqualToRightSlot.child(1).getDataType())
+ && canCompareType(originDataType)) {
+ // infer filter can not be lower than original datatype, or dataset would be wrong
+ if (originDataType.width() > leftSlotEqualToRightSlot.child(0).getDataType().width()
+ || originDataType.width() > leftSlotEqualToRightSlot.child(1).getDataType().width()) {
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
@@ -70,19 +72,44 @@ public Expression visit(Expression expr, Void context) {
@Override
public Expression visitComparisonPredicate(ComparisonPredicate cp, Void context) {
- if (cp.left().isSlot() && cp.right().isConstant()) {
- return replaceSlot(cp);
- } else if (cp.left().isConstant() && cp.right().isSlot()) {
- return replaceSlot(cp);
+ if (isExpressionSlotCoveredByCast(cp.left()) && (cp.right().isConstant())) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.left()));
+ } else if (isExpressionSlotCoveredByCast(cp.right()) && cp.left().isConstant()) {
+ return replaceSlot(cp, getDatatypeCoveredByCast(cp.right()));
}
return super.visit(cp, context);
}
- private Expression replaceSlot(Expression expr) {
+ private boolean isTwoExpressionEqualWithCast(Expression left, Expression right) {
+ return getSlotCoveredByCast(left).equals(getSlotCoveredByCast(right));
+ }
+
+ private boolean canCompareType(DataType dataType) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1621111812
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1624484310
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1611021802
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #21171:
URL: https://github.com/apache/doris/pull/21171#discussion_r1247463386
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java:
##########
Review Comment:
shoud use getExpressionCoveredByCast to process expression like `cast(cast(cast...(a)))`
##########
regression-test/suites/nereids_p0/infer_predicate/load.groovy:
##########
@@ -0,0 +1,22 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("load") {
+ sql 'create database if not exists nereids_infer_predicate_test'
Review Comment:
not need create database here, remove load.groovy completely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21171: [Fix](Nereids) Add cast comparison with slot reference when inferring predicate
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21171:
URL: https://github.com/apache/doris/pull/21171#issuecomment-1638079260
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org