You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/23 13:20:30 UTC
[GitHub] [spark] EnricoMi opened a new pull request, #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
EnricoMi opened a new pull request, #38358:
URL: https://github.com/apache/spark/pull/38358
### What changes were proposed in this pull request?
FileFormatWriter materializes an AdaptiveQueryPlan before accessing the plan's `outputOrdering`. This is required for Spark 3.0 to 3.3. Spark 3.4 does not need this because FileFormatWriter gets the final plan.
### Why are the changes needed?
FileFormatWriter enforces an ordering if the written plan does not provide that ordering. An AdaptiveQueryPlan does not know its final ordering (Spark 3.0 to 3.3), so FileFormatWriter always enforces the ordering even if the plan provides it. In case of spilling, that order gets broken (see SPARK-40588).
### Does this PR introduce _any_ user-facing change?
This fixes [SPARK-40588](https://issues.apache.org/jira/browse/SPARK-40588), which was introduced in 3.0.
### How was this patch tested?
The final plan that is written to files cannot be extracted from FileFormatWriter. The bug explained in [SPARK-40588](https://issues.apache.org/jira/browse/SPARK-40588) can only be asserted on the result files when spilling occurs. This is very hard to control in an unit test scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1009100809
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:
##########
@@ -221,6 +221,8 @@ case class AdaptiveSparkPlanExec(
.map(_.toLong).filter(SQLExecution.getQueryExecution(_) eq context.qe)
}
+ private[sql] lazy val finalPlan: SparkPlan = getFinalPhysicalPlan()
Review Comment:
Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1306782288
There is no Spark 3.x release that does not suffer from this. This blocks people from moving to Spark 3, while Spark 3.0 and 3.1 are already EOL. Please reconsider providing a fix before the Spark 3.4 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1297328942
Yes, we can get this back ported to Spark 3.2.3 once this PR is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1008880020
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala:
##########
@@ -187,8 +188,17 @@ object FileFormatWriter extends Logging {
// We should first sort by partition columns, then bucket id, and finally sorting columns.
val requiredOrdering =
partitionColumns ++ writerBucketSpec.map(_.bucketIdExpression) ++ sortColumns
+
+ // SPARK-40588: plan may contain an AdaptiveSparkPlanExec, which does not know
+ // its final plan's ordering, so we have to materialize that plan first
+ def materializeAdaptiveSparkPlan(plan: SparkPlan): SparkPlan = plan match {
Review Comment:
I am not sure what you mean with "is brough"` in "the missing ordering field of Sort in the top layer is also brought"?
It is also not clear to me what you are referring to with "removing 'AdaptiveSparkPlanExec'". Do you mean the change in this this PR? The 'AdaptiveSparkPlanExec' is not _removed_ but _replaced_ with the final plan. It is the same plan that is otherwise used when calling `.execute`, so AQE should work as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1306765838
@sunchao I don't think this blocks 3.3. It's not a new regression in 3.3, but a long-standing issue since we have AQE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1016662371
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:
##########
@@ -222,7 +222,7 @@ case class AdaptiveSparkPlanExec(
.map(_.toLong).filter(SQLExecution.getQueryExecution(_) eq context.qe)
}
- private def getFinalPhysicalPlan(): SparkPlan = lock.synchronized {
+ private[sql] def getFinalPhysicalPlan(): SparkPlan = lock.synchronized {
Review Comment:
probably just remove `private`, as other Spark vendors may hit the same problem and need to materialize the AQE plan manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1455620898
Yes, it looks like it removes the **empty** table location after **overwriting** the table failed due to the `ArithmeticException`.
@cloud-fan do you consider the removal of an empty table location after overwriting the table fails is a regression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zzzzming95 commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
zzzzming95 commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1008670694
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala:
##########
@@ -187,8 +188,17 @@ object FileFormatWriter extends Logging {
// We should first sort by partition columns, then bucket id, and finally sorting columns.
val requiredOrdering =
partitionColumns ++ writerBucketSpec.map(_.bucketIdExpression) ++ sortColumns
+
+ // SPARK-40588: plan may contain an AdaptiveSparkPlanExec, which does not know
+ // its final plan's ordering, so we have to materialize that plan first
+ def materializeAdaptiveSparkPlan(plan: SparkPlan): SparkPlan = plan match {
Review Comment:
I found a bug similar to this issue. The core of these problems is that `V1Writes#prepareQuery()` will generate a new Sort .
https://github.com/apache/spark/blob/a2f3958a29ad16a1b3e372156d6c6ae4959d5e8c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/V1Writes.scala#L70
In this pr : https://github.com/apache/spark/pull/38356
My solution is that in `V1Writes#prepareQuery()`, the missing ordering field of Sort in the top layer is also brought.
If `AdaptiveSparkPlanExec` is removed and the plan is extracted, will it cause other situations that do not meet expectations? For example, removing 'AdaptiveSparkPlanExec' causes some AQE features to fail to take effect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1302512656
@zzzzming95 Your query is affected by this issue as well (Spark 3.3):
```
Sort [input[2, string, true] ASC NULLS FIRST], false, 0
+- Project [id#37, sort_col#38, empty2null(p#39) AS p#46]
+- AdaptiveSparkPlan isFinalPlan=false
+- Sort [p#39 ASC NULLS FIRST, sort_col#38 ASC NULLS FIRST], false, 0
+- Exchange RoundRobinPartitioning(10), REPARTITION_BY_NUM, [plan_id=42]
+- LocalTableScan [id#37, sort_col#38, p#39]
```
The outer `Sort` will break the inner `Sort` when spills occur.
But you are right, the issue in #38356 is different and has been introduced in Spark 3.4.
For completeness, this is how nullable string partition columns look like after the fix (Spark 3.3):
```
Project [id#10, sort_col#11, empty2null(p#12) AS p#19]
+- *(1) Sort [p#12 ASC NULLS FIRST, sort_col#11 ASC NULLS FIRST], false, 0
+- ShuffleQueryStage 0
+- Exchange RoundRobinPartitioning(10), REPARTITION_BY_NUM, [plan_id=12]
+- LocalTableScan [id#10, sort_col#11, p#12]
```
Btw., this is what I would expect from Spark 3.4 when #38356 is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1296701255
@cloud-fan @sunchao is there a chance to get this regression fix into 3.2 before the upcoming 3.2.3 release?
There is currently no Spark 3 release that does not suffer from this regression (see [SPARK-40588](https://issues.apache.org/jira/browse/SPARK-40588)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1009005673
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:
##########
@@ -221,6 +221,8 @@ case class AdaptiveSparkPlanExec(
.map(_.toLong).filter(SQLExecution.getQueryExecution(_) eq context.qe)
}
+ private[sql] lazy val finalPlan: SparkPlan = getFinalPhysicalPlan()
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1296712835
@HyukjinKwon this PR targets `branch-3.3`, not `master`. How are tests kicked off in my fork?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1016671794
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala:
##########
@@ -187,8 +188,17 @@ object FileFormatWriter extends Logging {
// We should first sort by partition columns, then bucket id, and finally sorting columns.
val requiredOrdering =
partitionColumns ++ writerBucketSpec.map(_.bucketIdExpression) ++ sortColumns
+
+ // SPARK-40588: plan may contain an AdaptiveSparkPlanExec, which does not know
+ // its final plan's ordering, so we have to materialize that plan first
Review Comment:
```suggestion
// SPARK-40588: AdaptiveSparkPlanExec does not know its output partitioning and ordering before
// all its query stages are materialized. Here we have to materialize it first, before adding SortExec.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1016668422
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:
##########
@@ -222,7 +222,7 @@ case class AdaptiveSparkPlanExec(
.map(_.toLong).filter(SQLExecution.getQueryExecution(_) eq context.qe)
}
- private def getFinalPhysicalPlan(): SparkPlan = lock.synchronized {
+ private[sql] def getFinalPhysicalPlan(): SparkPlan = lock.synchronized {
Review Comment:
looking at the code again, I think it's risky to call `getFinalPhysicalPlan` directly. Other places call `withFinalPlanUpdate` which can handle UI update. How about we add a new function here?
```
def finalPhysicalPlan = withFinalPlanUpdate(identity)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1008881156
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala:
##########
@@ -187,8 +188,17 @@ object FileFormatWriter extends Logging {
// We should first sort by partition columns, then bucket id, and finally sorting columns.
val requiredOrdering =
partitionColumns ++ writerBucketSpec.map(_.bucketIdExpression) ++ sortColumns
+
+ // SPARK-40588: plan may contain an AdaptiveSparkPlanExec, which does not know
+ // its final plan's ordering, so we have to materialize that plan first
+ def materializeAdaptiveSparkPlan(plan: SparkPlan): SparkPlan = plan match {
Review Comment:
`V1WriteCommand` is a concept introduced after Spark 3.3. The issue addressed in this PR does not exist post Spark 3.3 as the `FieFormatWriter` does not see `AdaptiveSparkPlanExec` but the final plan (exactly what this change is trying to achieve).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1288156851
@cloud-fan this fixes a very peculiar bug introduced via AQE in Spark 3.0. With Spark 3.4, the issue disappeared.
A user-defined ordering, that starts with the partition columns, is broken by `FileFormaWriter` because it cannot see the actual order as `AdaptiveSparkPlanExec` does not know the final plan yet. Sorting by the partition columns breaks the existing user-defined order **only when spills occur**, because `UnsafeSorterSpillMerger` round-robins over the spill files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1308354718
thanks, merging to 3.3/3.2!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1008880020
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala:
##########
@@ -187,8 +188,17 @@ object FileFormatWriter extends Logging {
// We should first sort by partition columns, then bucket id, and finally sorting columns.
val requiredOrdering =
partitionColumns ++ writerBucketSpec.map(_.bucketIdExpression) ++ sortColumns
+
+ // SPARK-40588: plan may contain an AdaptiveSparkPlanExec, which does not know
+ // its final plan's ordering, so we have to materialize that plan first
+ def materializeAdaptiveSparkPlan(plan: SparkPlan): SparkPlan = plan match {
Review Comment:
I am not sure what you mean with "is brought"` in "the missing ordering field of Sort in the top layer is also brought"?
It is also not clear to me what you are referring to with "removing 'AdaptiveSparkPlanExec'". Do you mean the change in this this PR? The 'AdaptiveSparkPlanExec' is not _removed_ but _replaced_ with the final plan. It is the same plan that is otherwise used when calling `.execute`, so AQE should work as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1296765612
@HyukjinKwon never mind, I have looked at the wrong workflow ("Build") but branch 3.3 has the old workflows ("Build and test").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
maryannxue commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1003780647
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:
##########
@@ -221,6 +221,8 @@ case class AdaptiveSparkPlanExec(
.map(_.toLong).filter(SQLExecution.getQueryExecution(_) eq context.qe)
}
+ private[sql] lazy val finalPlan: SparkPlan = getFinalPhysicalPlan()
Review Comment:
can we just change access of `getFinalPhysicalPlan()` to `private[sql]` instead? I don't think we need a duplicate of `executedPlan` in this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zzzzming95 commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
zzzzming95 commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1012409686
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala:
##########
@@ -187,8 +188,17 @@ object FileFormatWriter extends Logging {
// We should first sort by partition columns, then bucket id, and finally sorting columns.
val requiredOrdering =
partitionColumns ++ writerBucketSpec.map(_.bucketIdExpression) ++ sortColumns
+
+ // SPARK-40588: plan may contain an AdaptiveSparkPlanExec, which does not know
+ // its final plan's ordering, so we have to materialize that plan first
+ def materializeAdaptiveSparkPlan(plan: SparkPlan): SparkPlan = plan match {
Review Comment:
I understand what you mean. The problem of https://github.com/apache/spark/pull/38356 is also caused by enforces the ordering at `V1WriteCommand`, but not the same case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1307611139
All done, all green: https://github.com/G-Research/spark/actions/runs/3420780216
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1307268665
@EnricoMi I updated my comment... https://github.com/apache/spark/pull/38358#discussion_r1016668422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1288157127
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
URL: https://github.com/apache/spark/pull/38358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1308492632
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1458251751
@wangyum do you know why it's a problem only in 3.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1373297981
I'll give it a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1373123850
shall we simply port this PR to master? It will be a noop when planned write is enabled, as there will be no `AdaptiveSparkPlanExec` in the input plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1372759273
Unfortunately, a similar issue exists for Spark 3.4.0 as well, created issue SPARK-41914 to track that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zzzzming95 commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
zzzzming95 commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1301560852
> @kristopherkane 3.1 is EOL unfortunately.
>
> @zzzzming95 Does this PR fix your problem?
I have no problem with this issue. Another similar issue I found https://github.com/apache/spark/pull/38356 is not the same case as this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38358:
URL: https://github.com/apache/spark/pull/38358#discussion_r1016662371
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:
##########
@@ -222,7 +222,7 @@ case class AdaptiveSparkPlanExec(
.map(_.toLong).filter(SQLExecution.getQueryExecution(_) eq context.qe)
}
- private def getFinalPhysicalPlan(): SparkPlan = lock.synchronized {
+ private[sql] def getFinalPhysicalPlan(): SparkPlan = lock.synchronized {
Review Comment:
probably just remove `private`, as other Spark vendors may hit the same problem and need to materialize the AQE plan manually in their Spark extensions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1306126062
Hi, curious what's the status of this PR? @EnricoMi @cloud-fan . I plan to start the 3.2.3 release process this week. Will this be fixed soon?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1455371977
@EnricoMi It seems it will remove the table location if a `java.lang.ArithmeticException` is thrown after this change.
How to reproduce:
```scala
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.sql.QueryTest
import org.apache.spark.sql.catalyst.TableIdentifier
sql("CREATE TABLE IF NOT EXISTS spark32_overwrite(amt1 int) STORED AS ORC")
sql("CREATE TABLE IF NOT EXISTS spark32_overwrite2(amt1 long) STORED AS ORC")
sql("INSERT OVERWRITE TABLE spark32_overwrite2 select 6000044164")
sql("set spark.sql.ansi.enabled=true")
val loc =
spark.sessionState.catalog.getTableMetadata(TableIdentifier("spark32_overwrite")).location
val fs = FileSystem.get(loc, spark.sparkContext.hadoopConfiguration)
println("Location exists: " + fs.exists(new Path(loc)))
try {
sql("INSERT OVERWRITE TABLE spark32_overwrite select amt1 from " +
"(select cast(amt1 as int) as amt1 from spark32_overwrite2 distribute by amt1)")
} finally {
println("Location exists: " + fs.exists(new Path(loc)))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] lordk911 commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by "lordk911 (via GitHub)" <gi...@apache.org>.
lordk911 commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1469726173
> @wangyum do you know why it's a problem only in 3.2?
Spark3.2.1 don't have this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kristopherkane commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
kristopherkane commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1299119179
Thanks for the fix! Is it possible this could land in 3.1 as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #38358: [SPARK-40588] FileFormatWriter materializes AQE plan before accessing outputOrdering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #38358:
URL: https://github.com/apache/spark/pull/38358#issuecomment-1299867696
@kristopherkane 3.1 is EOL unfortunately.
@zzzzming95 Does this PR fix your problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org