You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xzj7019 (via GitHub)" <gi...@apache.org> on 2023/06/12 09:43:33 UTC
[GitHub] [doris] xzj7019 opened a new pull request, #20714: [feature](nereids) Support multi target rf
xzj7019 opened a new pull request, #20714:
URL: https://github.com/apache/doris/pull/20714
## Proposed changes
Issue Number: close #xxx
Support multi target runtime filter, mainly for set operation, such as union/intersect/except.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly merged pull request #20714: [feature](nereids) Support multi target rf
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #20714:
URL: https://github.com/apache/doris/pull/20714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1587320198
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1594041418
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1592929972
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1587086942
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on a diff in pull request #20714: [feature](nereids) Support multi target rf
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #20714:
URL: https://github.com/apache/doris/pull/20714#discussion_r1227438782
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java:
##########
@@ -105,29 +111,77 @@ public PhysicalPlan visitPhysicalHashJoin(PhysicalHashJoin<? extends Plan, ? ext
if (type == TRuntimeFilterType.BITMAP) {
continue;
}
- // currently, we can ensure children in the two side are corresponding to the equal_to's.
- // so right maybe an expression and left is a slot
- Slot unwrappedSlot = checkTargetChild(equalTo.left());
- // aliasTransMap doesn't contain the key, means that the path from the olap scan to the join
- // contains join with denied join type. for example: a left join b on a.id = b.id
- if (unwrappedSlot == null || !aliasTransferMap.containsKey(unwrappedSlot)) {
- continue;
- }
- Slot olapScanSlot = aliasTransferMap.get(unwrappedSlot).second;
- PhysicalRelation scan = aliasTransferMap.get(unwrappedSlot).first;
- // in-filter is not friendly to pipeline
- if (type == TRuntimeFilterType.IN_OR_BLOOM
- && ctx.getSessionVariable().enablePipelineEngine()
- && hasRemoteTarget(join, scan)) {
- type = TRuntimeFilterType.BLOOM;
+ if (join.left() instanceof PhysicalUnion
Review Comment:
It's not convenient to handle set operation as visit framework since the join related info needs to be acquired. So just leave this inside the visit join function and support this pattern currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1586975888
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1587314955
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1593358910
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1594042352
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1590366437
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1592294478
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #20714: [feature](nereids) Support multi target rf
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #20714:
URL: https://github.com/apache/doris/pull/20714#discussion_r1226475624
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java:
##########
@@ -105,29 +111,77 @@ public PhysicalPlan visitPhysicalHashJoin(PhysicalHashJoin<? extends Plan, ? ext
if (type == TRuntimeFilterType.BITMAP) {
continue;
}
- // currently, we can ensure children in the two side are corresponding to the equal_to's.
- // so right maybe an expression and left is a slot
- Slot unwrappedSlot = checkTargetChild(equalTo.left());
- // aliasTransMap doesn't contain the key, means that the path from the olap scan to the join
- // contains join with denied join type. for example: a left join b on a.id = b.id
- if (unwrappedSlot == null || !aliasTransferMap.containsKey(unwrappedSlot)) {
- continue;
- }
- Slot olapScanSlot = aliasTransferMap.get(unwrappedSlot).second;
- PhysicalRelation scan = aliasTransferMap.get(unwrappedSlot).first;
- // in-filter is not friendly to pipeline
- if (type == TRuntimeFilterType.IN_OR_BLOOM
- && ctx.getSessionVariable().enablePipelineEngine()
- && hasRemoteTarget(join, scan)) {
- type = TRuntimeFilterType.BLOOM;
+ if (join.left() instanceof PhysicalUnion
Review Comment:
there are some missed cases.
for example:
`Agg(A union B) join C`, where `A union B` is not join left child.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1590338017
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1594523773
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1593252476
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1590374294
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1587299972
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1592305404
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1590365420
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1586979625
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1586975702
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on pull request #20714: [feature](nereids) Support multi target rf
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #20714:
URL: https://github.com/apache/doris/pull/20714#issuecomment-1592922695
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org