You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2023/05/05 09:22:35 UTC
[GitHub] [doris] englefly opened a new pull request, #19312: prune rf by ndv
englefly opened a new pull request, #19312:
URL: https://github.com/apache/doris/pull/19312
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19312: [feature](nereids)prune runtime filter by ndv
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1696650447
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1713493075
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.75 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162388597 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #19312:
URL: https://github.com/apache/doris/pull/19312#discussion_r1324402130
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPruner.java:
##########
@@ -199,7 +189,7 @@ private boolean isEffectiveRuntimeFilter(EqualTo equalTo, PhysicalHashJoin join)
if (probeColumnStat.isUnKnown || buildColumnStat.isUnKnown) {
return true;
}
- return probeColumnStat.notEnclosed(buildColumnStat)
- || buildColumnStat.ndv < probeColumnStat.ndv * 0.95;
+ double buildNdvInProbeRange = buildColumnStat.ndvIntersection(probeColumnStat);
+ return probeColumnStat.ndv > buildNdvInProbeRange * (1 + ColumnStatistic.STATS_ERROR);
Review Comment:
the error introduced by sampling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on pull request #19312: prune rf by ndv
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1536942756
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19312: [feature](nereids)prune runtime filter by ndv
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1696722130
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on a diff in pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #19312:
URL: https://github.com/apache/doris/pull/19312#discussion_r1324395860
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java:
##########
@@ -153,8 +154,11 @@ public void removeFilter(ExprId targetId, PhysicalHashJoin builderNode) {
if (filters != null) {
Iterator<RuntimeFilter> iter = filters.iterator();
while (iter.hasNext()) {
- if (iter.next().getBuilderNode().equals(builderNode)) {
+ RuntimeFilter rf = iter.next();
+ if (rf.getBuilderNode().equals(builderNode)) {
+ builderNode.getRuntimeFilters().remove(rf);
iter.remove();
+ prunedRF.add(rf);
Review Comment:
Seems that prunedRF is not used further?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPruner.java:
##########
@@ -199,7 +189,7 @@ private boolean isEffectiveRuntimeFilter(EqualTo equalTo, PhysicalHashJoin join)
if (probeColumnStat.isUnKnown || buildColumnStat.isUnKnown) {
return true;
}
- return probeColumnStat.notEnclosed(buildColumnStat)
- || buildColumnStat.ndv < probeColumnStat.ndv * 0.95;
+ double buildNdvInProbeRange = buildColumnStat.ndvIntersection(probeColumnStat);
+ return probeColumnStat.ndv > buildNdvInProbeRange * (1 + ColumnStatistic.STATS_ERROR);
Review Comment:
what is the STATS_ERROR meaning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly merged pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #19312:
URL: https://github.com/apache/doris/pull/19312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #19312:
URL: https://github.com/apache/doris/pull/19312#discussion_r1324401685
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java:
##########
@@ -153,8 +154,11 @@ public void removeFilter(ExprId targetId, PhysicalHashJoin builderNode) {
if (filters != null) {
Iterator<RuntimeFilter> iter = filters.iterator();
while (iter.hasNext()) {
- if (iter.next().getBuilderNode().equals(builderNode)) {
+ RuntimeFilter rf = iter.next();
+ if (rf.getBuilderNode().equals(builderNode)) {
+ builderNode.getRuntimeFilters().remove(rf);
iter.remove();
+ prunedRF.add(rf);
Review Comment:
it is for ut.
ut comes latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1713795798
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.54 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162203607 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] doris-robot commented on pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1713330378
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.7 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162455464 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on pull request #19312: [feature](nereids)prune runtime filter by ndv
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1543050024
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19312: [feature](nereids)prune runtime filter by ndv
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1696672000
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19312: [feature](nereids)prune runtime filter by ndv
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1700627543
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on pull request #19312: [feature](nereids)prune runtime filter by ndv
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1537778064
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1713747627
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1716818240
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1713436516
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1713275854
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19312: [feature](nereids)prune runtime filter (tpch part)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19312:
URL: https://github.com/apache/doris/pull/19312#issuecomment-1716818219
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org