[GitHub] [spark] viirya commented on pull request #33564: [SPARK-36136][SQL][TESTS] Refactor PruneFileSourcePartitionsSuite etc to a different package

[GitBox <gi...@apache.org>]

viirya commented on pull request #33564:
URL: https://github.com/apache/spark/pull/33564#issuecomment-889492218


   I see. lgtm. I am not in front of laptop. Will merge this after I come back.
   
   On Thu, Jul 29, 2021, 15:06 Chao Sun ***@***.***> wrote:
   
   > ***@***.**** commented on this pull request.
   > ------------------------------
   >
   > In
   > sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala
   > <https://github.com/apache/spark/pull/33564#discussion_r679516263>:
   >
   > > @@ -116,7 +107,7 @@ class PruneFileSourcePartitionsSuite extends PrunePartitionSuiteBase {
   >      // Force datasource v2 for parquet
   >      withSQLConf((SQLConf.USE_V1_SOURCE_LIST.key, "")) {
   >        withTempPath { dir =>
   > -        spark.range(10).selectExpr("id", "id % 3 as p")
   > +        spark.range(10).coalesce(1).selectExpr("id", "id % 3 as p")
   >
   > @viirya <https://github.com/viirya> @huaxingao
   > <https://github.com/huaxingao> this is because TestHiveSingleton uses
   > local[1] while SharedSparkSession uses local[2] so it will generate 2
   > Parquet files. As result, ParquetScan will generate 2 input splits which
   > makes the test fail since the provided value for expectedPartitionCount
   > is 1.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/spark/pull/33564#discussion_r679516263>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AAAQZ54KNSERYH4IFNPIYE3T2HGGDANCNFSM5BFLC3HA>
   > .
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org