You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/06/23 12:49:06 UTC
[GitHub] [spark] mcdull-zhang opened a new pull request, #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
mcdull-zhang opened a new pull request, #36966:
URL: https://github.com/apache/spark/pull/36966
### What changes were proposed in this pull request?
This unit test sometimes fails to run. for example, https://github.com/apache/spark/pull/35715#discussion_r892247619
When the left side is completed first, and then the right side is completed, since it is known that there are many empty partitions on the left side, the broadcast on the right side is demoted.
However, if the right side is completed first and the left side is still being executed, the right side does not know whether there are many empty partitions on the left side, so there is no demote, and then the right side is broadcast in the planning stage.
This PR does this:
When it is found that the other side is QueryStage, if the QueryStage has not been materialized, demote it first. When the other side is completed, judge again whether demote is needed.
### Why are the changes needed?
Fix small problems in logic
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
manual testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
URL: https://github.com/apache/spark/pull/36966
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36966:
URL: https://github.com/apache/spark/pull/36966#discussion_r907040117
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala:
##########
@@ -710,18 +711,20 @@ class AdaptiveQueryExecSuite
test("SPARK-37753: Inhibit broadcast in left outer join when there are many empty" +
" partitions on outer/left side") {
- withSQLConf(
- SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
- SQLConf.NON_EMPTY_PARTITION_RATIO_FOR_BROADCAST_JOIN.key -> "0.5") {
- // `testData` is small enough to be broadcast but has empty partition ratio over the config.
- withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
- val (plan, adaptivePlan) = runAdaptiveAndVerifyResult(
- "SELECT * FROM (select * from testData where value = '1') td" +
- " left outer join testData2 ON key = a")
- val smj = findTopLevelSortMergeJoin(plan)
- assert(smj.size == 1)
- val bhj = findTopLevelBroadcastHashJoin(adaptivePlan)
- assert(bhj.isEmpty)
+ eventually(timeout(40.seconds), interval(500.milliseconds)) {
Review Comment:
the interval is 500 ms, do you know how long this test takes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mcdull-zhang commented on a diff in pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
mcdull-zhang commented on code in PR #36966:
URL: https://github.com/apache/spark/pull/36966#discussion_r907052628
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala:
##########
@@ -710,18 +711,20 @@ class AdaptiveQueryExecSuite
test("SPARK-37753: Inhibit broadcast in left outer join when there are many empty" +
" partitions on outer/left side") {
- withSQLConf(
- SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
- SQLConf.NON_EMPTY_PARTITION_RATIO_FOR_BROADCAST_JOIN.key -> "0.5") {
- // `testData` is small enough to be broadcast but has empty partition ratio over the config.
- withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
- val (plan, adaptivePlan) = runAdaptiveAndVerifyResult(
- "SELECT * FROM (select * from testData where value = '1') td" +
- " left outer join testData2 ON key = a")
- val smj = findTopLevelSortMergeJoin(plan)
- assert(smj.size == 1)
- val bhj = findTopLevelBroadcastHashJoin(adaptivePlan)
- assert(bhj.isEmpty)
+ eventually(timeout(40.seconds), interval(500.milliseconds)) {
Review Comment:
After communication, the unit test takes about 3s, the timeout is set to 15s, and it can be retried up to 5 times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #36966:
URL: https://github.com/apache/spark/pull/36966#issuecomment-1164965670
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mcdull-zhang commented on pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
mcdull-zhang commented on PR #36966:
URL: https://github.com/apache/spark/pull/36966#issuecomment-1165137130
cc @cloud-fan @ekoifman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #36966:
URL: https://github.com/apache/spark/pull/36966#issuecomment-1167374600
thanks, merging to master/3.3!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mcdull-zhang commented on a diff in pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
mcdull-zhang commented on code in PR #36966:
URL: https://github.com/apache/spark/pull/36966#discussion_r908417063
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala:
##########
@@ -710,18 +711,20 @@ class AdaptiveQueryExecSuite
test("SPARK-37753: Inhibit broadcast in left outer join when there are many empty" +
" partitions on outer/left side") {
- withSQLConf(
- SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
- SQLConf.NON_EMPTY_PARTITION_RATIO_FOR_BROADCAST_JOIN.key -> "0.5") {
- // `testData` is small enough to be broadcast but has empty partition ratio over the config.
- withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
- val (plan, adaptivePlan) = runAdaptiveAndVerifyResult(
- "SELECT * FROM (select * from testData where value = '1') td" +
- " left outer join testData2 ON key = a")
- val smj = findTopLevelSortMergeJoin(plan)
- assert(smj.size == 1)
- val bhj = findTopLevelBroadcastHashJoin(adaptivePlan)
- assert(bhj.isEmpty)
+ eventually(timeout(15.seconds), interval(500.milliseconds)) {
Review Comment:
It's really hard to understand without writing comments, solved by this https://github.com/apache/spark/pull/37018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36966:
URL: https://github.com/apache/spark/pull/36966#discussion_r907411473
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala:
##########
@@ -710,18 +711,20 @@ class AdaptiveQueryExecSuite
test("SPARK-37753: Inhibit broadcast in left outer join when there are many empty" +
" partitions on outer/left side") {
- withSQLConf(
- SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
- SQLConf.NON_EMPTY_PARTITION_RATIO_FOR_BROADCAST_JOIN.key -> "0.5") {
- // `testData` is small enough to be broadcast but has empty partition ratio over the config.
- withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200") {
- val (plan, adaptivePlan) = runAdaptiveAndVerifyResult(
- "SELECT * FROM (select * from testData where value = '1') td" +
- " left outer join testData2 ON key = a")
- val smj = findTopLevelSortMergeJoin(plan)
- assert(smj.size == 1)
- val bhj = findTopLevelBroadcastHashJoin(adaptivePlan)
- assert(bhj.isEmpty)
+ eventually(timeout(15.seconds), interval(500.milliseconds)) {
Review Comment:
ah missed one thing: can we add some comments to explain the reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #36966: [SPARK-37753] [FOLLOWUP] [SQL] Fix unit tests sometimes failing
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #36966:
URL: https://github.com/apache/spark/pull/36966#issuecomment-1165257012
I'm not sure about this approach. Does it introduce extra latency and may hurt performance?
Good catch on the flaky test! Shall we simply use the `eventually` test util to run the test multiple times?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org