You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "snuyanzin (via GitHub)" <gi...@apache.org> on 2024/09/24 22:44:08 UTC
[PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
snuyanzin opened a new pull request, #25388:
URL: https://github.com/apache/flink/pull/25388
## What is the purpose of the change
The idea is to use just set of RelNodes and do not generate toString each time.
Also use the fact that Calcite can compare RelNodes
## Verifying this change
This change is already covered by existing tests, such as *(please describe tests)*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): ( no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
- The serializers: ( no )
- The runtime per-record code paths (performance sensitive): (yes / no / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
- The S3 file system connector: ( no)
## Documentation
- Does this pull request introduce a new feature? ( no)
- If yes, how is the feature documented? (not applicable )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #25388:
URL: https://github.com/apache/flink/pull/25388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #25388:
URL: https://github.com/apache/flink/pull/25388#discussion_r1774176158
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/rules/logical/JoinDependentConditionDerivationRuleTest.xml:
##########
@@ -98,7 +98,7 @@ LogicalProject(a=[$0], d=[$3])
<Resource name="optimized rel plan">
<![CDATA[
LogicalProject(a=[$0], d=[$3])
-+- LogicalJoin(condition=[AND(OR(AND(=($0, 1), =($1, 1), =($3, 2), =($4, 2)), AND(=($0, 2), =($1, 2), =($3, 1), =($4, 1))), OR(AND(=($0, 1), =($1, 1)), AND(=($0, 2), =($1, 2))), OR(AND(=($3, 2), =($4, 2)), AND(=($3, 1), =($4, 1))))], joinType=[inner])
++- LogicalJoin(condition=[AND(OR(AND(=($0, 1), =($1, 1), =($3, 2), =($4, 2)), AND(=($0, 2), =($1, 2), =($3, 1), =($4, 1))), OR(AND(=(1, $0), =(1, $1)), AND(=(2, $0), =(2, $1))), OR(AND(=(2, $3), =(2, $4)), AND(=(1, $3), =(1, $4))))], joinType=[inner])
Review Comment:
here and in others just swap of left and right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #25388:
URL: https://github.com/apache/flink/pull/25388#issuecomment-2373541247
thanks for taking a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #25388:
URL: https://github.com/apache/flink/pull/25388#issuecomment-2373050542
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "jnh5y (via GitHub)" <gi...@apache.org>.
jnh5y commented on code in PR #25388:
URL: https://github.com/apache/flink/pull/25388#discussion_r1775188382
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/FlinkRexUtil.scala:
##########
@@ -439,17 +439,16 @@ object FlinkRexUtil {
})
private class EquivalentExprShuttle(rexBuilder: RexBuilder) extends RexShuttle {
- private val equiExprMap = mutable.HashMap[String, RexNode]()
+ private val equiExprSet = mutable.HashSet[RexNode]()
override def visitCall(call: RexCall): RexNode = {
call.getOperator match {
case EQUALS | NOT_EQUALS | GREATER_THAN | LESS_THAN | GREATER_THAN_OR_EQUAL |
LESS_THAN_OR_EQUAL =>
- val swapped = swapOperands(call)
- if (equiExprMap.contains(swapped.toString)) {
- swapped
+ if (equiExprSet.contains(call)) {
Review Comment:
> 2. I faced one float issue for a couple of plans: depending on how tests are running (just one or for the whole class) the order of operands is different (the operator expression is valid for both).
> So need more time to debug and fix it
Interesting. Is it a bug in RexNormalize or something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "jnh5y (via GitHub)" <gi...@apache.org>.
jnh5y commented on code in PR #25388:
URL: https://github.com/apache/flink/pull/25388#discussion_r1774214379
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/FlinkRexUtil.scala:
##########
@@ -439,17 +439,16 @@ object FlinkRexUtil {
})
private class EquivalentExprShuttle(rexBuilder: RexBuilder) extends RexShuttle {
- private val equiExprMap = mutable.HashMap[String, RexNode]()
+ private val equiExprSet = mutable.HashSet[RexNode]()
override def visitCall(call: RexCall): RexNode = {
call.getOperator match {
case EQUALS | NOT_EQUALS | GREATER_THAN | LESS_THAN | GREATER_THAN_OR_EQUAL |
LESS_THAN_OR_EQUAL =>
- val swapped = swapOperands(call)
- if (equiExprMap.contains(swapped.toString)) {
- swapped
+ if (equiExprSet.contains(call)) {
Review Comment:
Did you already try the RexNormalize change?
That does sound like a good idea / nice change. Have you filled a JIRA for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #25388:
URL: https://github.com/apache/flink/pull/25388#issuecomment-2372519096
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d2bdb6986b4c962e75a542e66b2f95e06c7f4725",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2bdb6986b4c962e75a542e66b2f95e06c7f4725",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2bdb6986b4c962e75a542e66b2f95e06c7f4725 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #25388:
URL: https://github.com/apache/flink/pull/25388#discussion_r1774883976
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/FlinkRexUtil.scala:
##########
@@ -439,17 +439,16 @@ object FlinkRexUtil {
})
private class EquivalentExprShuttle(rexBuilder: RexBuilder) extends RexShuttle {
- private val equiExprMap = mutable.HashMap[String, RexNode]()
+ private val equiExprSet = mutable.HashSet[RexNode]()
override def visitCall(call: RexCall): RexNode = {
call.getOperator match {
case EQUALS | NOT_EQUALS | GREATER_THAN | LESS_THAN | GREATER_THAN_OR_EQUAL |
LESS_THAN_OR_EQUAL =>
- val swapped = swapOperands(call)
- if (equiExprMap.contains(swapped.toString)) {
- swapped
+ if (equiExprSet.contains(call)) {
Review Comment:
jira issue https://issues.apache.org/jira/browse/FLINK-36367
I tried
1. it touches more than 100 plans which is still doable
2. I faced one float issue for a couple of plans: depending on how tests are running (just one or for the whole class) the order of operands is different (the operator expression is valid for both).
So need more time to debug and fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36361][table] Do not use StringBuilder for EquivalentExprShuttle [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #25388:
URL: https://github.com/apache/flink/pull/25388#discussion_r1774177868
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/FlinkRexUtil.scala:
##########
@@ -439,17 +439,16 @@ object FlinkRexUtil {
})
private class EquivalentExprShuttle(rexBuilder: RexBuilder) extends RexShuttle {
- private val equiExprMap = mutable.HashMap[String, RexNode]()
+ private val equiExprSet = mutable.HashSet[RexNode]()
override def visitCall(call: RexCall): RexNode = {
call.getOperator match {
case EQUALS | NOT_EQUALS | GREATER_THAN | LESS_THAN | GREATER_THAN_OR_EQUAL |
LESS_THAN_OR_EQUAL =>
- val swapped = swapOperands(call)
- if (equiExprMap.contains(swapped.toString)) {
- swapped
+ if (equiExprSet.contains(call)) {
Review Comment:
For Calcite `>($1, $2)` and `<($2, $1)` are equal, same for others
this fact could be used here.
Ideally we could reuse Calcite's RexNormalize however it will lead to large plan related changes, may be makes sense to do it with separate PR/jira
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org