You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sohardforaname (via GitHub)" <gi...@apache.org> on 2023/05/19 06:42:47 UTC
[GitHub] [doris] sohardforaname opened a new pull request, #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
sohardforaname opened a new pull request, #19863:
URL: https://github.com/apache/doris/pull/19863
# Proposed changes
Issue Number: close #xxx
## Problem summary
the fold constant on datetime functions ignores timezone. Now we fix it.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #19863: [Fix](Nereids) fold constant result is wrong on functions relative to timezone
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #19863:
URL: https://github.com/apache/doris/pull/19863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1562163597
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #19863:
URL: https://github.com/apache/doris/pull/19863#discussion_r1198879408
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnFE.java:
##########
@@ -290,8 +290,9 @@ public Expression visitCast(Cast cast, ExpressionRewriteContext context) {
@Override
public Expression visitBoundFunction(BoundFunction boundFunction, ExpressionRewriteContext context) {
boundFunction = rewriteChildren(boundFunction, context);
- //functions, like current_date, do not have arg
- if (boundFunction.getArguments().isEmpty()) {
+ // functions related to current/local/now will be calculated by be.
Review Comment:
i think we should do constant folding on these functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1568240376
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1559251027
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1554099485
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1560402198
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1566981266
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1560733558
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1561156298
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.04 seconds
stream load tsv: 448 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 78.2 seconds inserted 10000000 Rows, about 127K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230524132837_clickbench_pr_149864.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1560440902
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.5 seconds
stream load tsv: 443 seconds loaded 74807831229 Bytes, about 161 MB/s
stream load json: 23 seconds loaded 2358488459 Bytes, about 97 MB/s
stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 79.6 seconds inserted 10000000 Rows, about 125K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230524042639_clickbench_pr_149403.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1554301238
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1560611166
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1560952671
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1560955341
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1560540587
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1569669698
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1560623884
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.43 seconds
stream load tsv: 446 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 80.1 seconds inserted 10000000 Rows, about 124K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230524075235_clickbench_pr_149587.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1561103287
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.11 seconds
stream load tsv: 446 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 77.9 seconds inserted 10000000 Rows, about 128K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230524125954_clickbench_pr_149847.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #19863:
URL: https://github.com/apache/doris/pull/19863#discussion_r1210292615
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java:
##########
@@ -49,6 +51,7 @@
* Abstract class for all Expression in Nereids.
*/
public abstract class Expression extends AbstractTreeNode<Expression> implements ExpressionTrait {
+ private static final Set<String> foldableLeafExpression = ImmutableSet.of("current_time", "current_date");
Review Comment:
i think u should add a interface named Foldable and let CurrentTime and CurrentDate implement it
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ExpressionEvaluator.java:
##########
@@ -49,6 +49,7 @@ public enum ExpressionEvaluator {
INSTANCE;
private ImmutableMultimap<String, FunctionInvoker> functions;
+ // the not eval non-empty arguments datetime function
Review Comment:
what the comment mean?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/DateTimeExtractAndTransform.java:
##########
@@ -448,18 +449,17 @@ public static Expression fromUnixTime(IntegerLiteral second, VarcharLiteral form
.plusSeconds(second.getValue())
.atZone(ZoneId.of("UTC+0"))
.toOffsetDateTime()
- .atZoneSameInstant(ZoneId.systemDefault());
+ .atZoneSameInstant(getTimeZone());
return dateFormat(new DateTimeLiteral(dateTime.getYear(), dateTime.getMonthValue(),
dateTime.getDayOfMonth(), dateTime.getHour(), dateTime.getMinute(), dateTime.getSecond()),
format);
}
- /**
- * date transformation function: unix_timestamp
- */
- @ExecFunction(name = "unix_timestamp", argTypes = {}, returnType = "INT")
- public static Expression unixTimestamp() {
- return new IntegerLiteral(getTimestamp(LocalDateTime.now()));
+ public static ZoneId getTimeZone() {
+ if (ConnectContext.get() == null || ConnectContext.get().getSessionVariable() == null) {
+ return ZoneId.systemDefault();
+ }
+ return ZoneId.of(ConnectContext.get().getSessionVariable().getTimeZone());
Review Comment:
why we have two same function in two different files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1569408368
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19863: [Fix](Nereids)fix nereids fold constant bug on functions relative to timezone
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19863:
URL: https://github.com/apache/doris/pull/19863#issuecomment-1569669640
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org