You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/27 17:13:08 UTC
[GitHub] [spark] peter-toth opened a new pull request, #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
peter-toth opened a new pull request, #37693:
URL: https://github.com/apache/spark/pull/37693
### What changes were proposed in this pull request?
Unfortunately the fix in https://github.com/apache/spark/pull/31848 was not correct in all cases. When the partition or data filter contains a column that is not in `readSchema()` the filter nornalization in `FileScan.equals()` doesn't work.
### Why are the changes needed?
To fix `FileScan.equals()` to fix reuse issues.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added new UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a diff in pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
peter-toth commented on code in PR #37693:
URL: https://github.com/apache/spark/pull/37693#discussion_r956977079
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4003,6 +4003,36 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
+ test("SPARK-40245: Fix FileScan canonicalization when partition or data filter columns are not " +
+ "read") {
+ withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {
+ withTempPath { path =>
+ spark.range(5)
+ .withColumn("p", $"id" % 2)
+ .write
+ .mode("overwrite")
+ .partitionBy("p")
+ .parquet(path.toString)
+ withTempView("t") {
+ spark.read.parquet(path.toString).createOrReplaceTempView("t")
+ val df = sql(
+ """
+ |SELECT t1.id, t2.id, t3.id
Review Comment:
No, onlu exchange reuse didn't happen.
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4003,6 +4003,36 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
+ test("SPARK-40245: Fix FileScan canonicalization when partition or data filter columns are not " +
+ "read") {
+ withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {
+ withTempPath { path =>
+ spark.range(5)
+ .withColumn("p", $"id" % 2)
+ .write
+ .mode("overwrite")
+ .partitionBy("p")
+ .parquet(path.toString)
+ withTempView("t") {
+ spark.read.parquet(path.toString).createOrReplaceTempView("t")
+ val df = sql(
+ """
+ |SELECT t1.id, t2.id, t3.id
Review Comment:
No, only exchange reuse didn't happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #37693:
URL: https://github.com/apache/spark/pull/37693#issuecomment-1229886688
LGTM, please fix conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
URL: https://github.com/apache/spark/pull/37693
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a diff in pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
peter-toth commented on code in PR #37693:
URL: https://github.com/apache/spark/pull/37693#discussion_r956977079
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4003,6 +4003,36 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
+ test("SPARK-40245: Fix FileScan canonicalization when partition or data filter columns are not " +
+ "read") {
+ withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {
+ withTempPath { path =>
+ spark.range(5)
+ .withColumn("p", $"id" % 2)
+ .write
+ .mode("overwrite")
+ .partitionBy("p")
+ .parquet(path.toString)
+ withTempView("t") {
+ spark.read.parquet(path.toString).createOrReplaceTempView("t")
+ val df = sql(
+ """
+ |SELECT t1.id, t2.id, t3.id
Review Comment:
No, just exchange reuse didn't happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
peter-toth commented on PR #37693:
URL: https://github.com/apache/spark/pull/37693#issuecomment-1230506954
Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
peter-toth commented on PR #37693:
URL: https://github.com/apache/spark/pull/37693#issuecomment-1230107763
> LGTM, please fix conflicts
Ok, merged latest `master` into this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #37693:
URL: https://github.com/apache/spark/pull/37693#issuecomment-1230505007
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
peter-toth commented on PR #37693:
URL: https://github.com/apache/spark/pull/37693#issuecomment-1229844776
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37693: [SPARK-40245][SQL] Fix FileScan equality check when partition or data filter columns are not read
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37693:
URL: https://github.com/apache/spark/pull/37693#discussion_r956971059
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4003,6 +4003,36 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
+ test("SPARK-40245: Fix FileScan canonicalization when partition or data filter columns are not " +
+ "read") {
+ withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {
+ withTempPath { path =>
+ spark.range(5)
+ .withColumn("p", $"id" % 2)
+ .write
+ .mode("overwrite")
+ .partitionBy("p")
+ .parquet(path.toString)
+ withTempView("t") {
+ spark.read.parquet(path.toString).createOrReplaceTempView("t")
+ val df = sql(
+ """
+ |SELECT t1.id, t2.id, t3.id
Review Comment:
do we return wrong result before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org