You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2018/12/07 05:47:00 UTC
[jira] [Updated] (SPARK-26188) Spark 2.4.0 Partitioning behavior
breaks backwards compatibility
[ https://issues.apache.org/jira/browse/SPARK-26188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li updated SPARK-26188:
----------------------------
Labels: release-notes (was: )
> Spark 2.4.0 Partitioning behavior breaks backwards compatibility
> ----------------------------------------------------------------
>
> Key: SPARK-26188
> URL: https://issues.apache.org/jira/browse/SPARK-26188
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Damien Doucet-Girard
> Assignee: Gengliang Wang
> Priority: Critical
> Labels: correctness, release-notes
> Fix For: 2.4.1, 3.0.0
>
>
> My team uses spark to partition and output parquet files to amazon S3. We typically use 256 partitions, from 00 to ff.
> We've observed that in spark 2.3.2 and prior, it reads the partitions as strings by default. However, in spark 2.4.0 and later, the type of each partition is inferred by default, and partitions such as 00 become 0 and 4d become 4.0.
> Here is a log sample of this behavior from one of our jobs:
> 2.4.0:
> {code:java}
> 18/11/27 14:02:27 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=00/part-00061-hashredacted.parquet, range: 0-662, partition values: [0]
> 18/11/27 14:02:28 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ef/part-00034-hashredacted.parquet, range: 0-662, partition values: [ef]
> 18/11/27 14:02:29 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4a/part-00151-hashredacted.parquet, range: 0-662, partition values: [4a]
> 18/11/27 14:02:30 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=74/part-00180-hashredacted.parquet, range: 0-662, partition values: [74]
> 18/11/27 14:02:32 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=f5/part-00156-hashredacted.parquet, range: 0-662, partition values: [f5]
> 18/11/27 14:02:33 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=50/part-00195-hashredacted.parquet, range: 0-662, partition values: [50]
> 18/11/27 14:02:34 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=70/part-00054-hashredacted.parquet, range: 0-662, partition values: [70]
> 18/11/27 14:02:35 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=b9/part-00012-hashredacted.parquet, range: 0-662, partition values: [b9]
> 18/11/27 14:02:37 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=d2/part-00016-hashredacted.parquet, range: 0-662, partition values: [d2]
> 18/11/27 14:02:38 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=51/part-00003-hashredacted.parquet, range: 0-662, partition values: [51]
> 18/11/27 14:02:39 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=84/part-00135-hashredacted.parquet, range: 0-662, partition values: [84]
> 18/11/27 14:02:40 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=b5/part-00190-hashredacted.parquet, range: 0-662, partition values: [b5]
> 18/11/27 14:02:41 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=88/part-00143-hashredacted.parquet, range: 0-662, partition values: [88]
> 18/11/27 14:02:42 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4d/part-00120-hashredacted.parquet, range: 0-662, partition values: [4.0]
> 18/11/27 14:02:43 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ac/part-00119-hashredacted.parquet, range: 0-662, partition values: [ac]
> 18/11/27 14:02:44 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=24/part-00139-hashredacted.parquet, range: 0-662, partition values: [24]
> 18/11/27 14:02:45 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=fd/part-00167-hashredacted.parquet, range: 0-662, partition values: [fd]
> 18/11/27 14:02:46 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=52/part-00033-hashredacted.parquet, range: 0-662, partition values: [52]
> 18/11/27 14:02:47 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ab/part-00083-hashredacted.parquet, range: 0-662, partition values: [ab]
> 18/11/27 14:02:48 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=f8/part-00018-hashredacted.parquet, range: 0-662, partition values: [f8]
> 18/11/27 14:02:49 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=7a/part-00093-hashredacted.parquet, range: 0-662, partition values: [7a]
> 18/11/27 14:02:50 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ba/part-00020-hashredacted.parquet, range: 0-662, partition values: [ba]
> 18/11/27 14:02:51 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=2d/part-00085-hashredacted.parquet, range: 0-662, partition values: [2.0]
> 18/11/27 14:02:52 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=03/part-00099-hashredacted.parquet, range: 0-662, partition values: [3]
> 18/11/27 14:02:53 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=57/part-00196-hashredacted.parquet, range: 0-662, partition values: [57]
> 18/11/27 14:02:54 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=81/part-00122-hashredacted.parquet, range: 0-662, partition values: [81]
> 18/11/27 14:02:55 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=6d/part-00167-hashredacted.parquet, range: 0-662, partition values: [6.0]
> 18/11/27 14:02:56 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=36/part-00154-hashredacted.parquet, range: 0-662, partition values: [36]
> 18/11/27 14:02:57 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4b/part-00093-hashredacted.parquet, range: 0-662, partition values: [4b]{code}
> 2.3.2:
> {code:java}
> 18/11/27 14:09:00 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=60/part-00082-hashredacted.parquet, range: 0-662, partition values: [60]
> 18/11/27 14:09:01 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=00/part-00061-hashredacted.parquet, range: 0-662, partition values: [00]
> 18/11/27 14:09:02 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ef/part-00034-hashredacted.parquet, range: 0-662, partition values: [ef]
> 18/11/27 14:09:02 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4a/part-00151-hashredacted.parquet, range: 0-662, partition values: [4a]
> 18/11/27 14:09:03 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=74/part-00180-hashredacted.parquet, range: 0-662, partition values: [74]
> 18/11/27 14:09:04 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=f5/part-00156-hashredacted.parquet, range: 0-662, partition values: [f5]
> 18/11/27 14:09:04 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=50/part-00195-hashredacted.parquet, range: 0-662, partition values: [50]
> 18/11/27 14:09:05 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=70/part-00054-hashredacted.parquet, range: 0-662, partition values: [70]
> 18/11/27 14:09:05 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=b9/part-00012-hashredacted.parquet, range: 0-662, partition values: [b9]
> 18/11/27 14:09:06 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=d2/part-00016-hashredacted.parquet, range: 0-662, partition values: [d2]
> 18/11/27 14:09:06 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=51/part-00003-hashredacted.parquet, range: 0-662, partition values: [51]
> 18/11/27 14:09:07 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=84/part-00135-hashredacted.parquet, range: 0-662, partition values: [84]
> 18/11/27 14:09:08 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=b5/part-00190-hashredacted.parquet, range: 0-662, partition values: [b5]
> 18/11/27 14:09:08 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=88/part-00143-hashredacted.parquet, range: 0-662, partition values: [88]
> 18/11/27 14:09:09 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4d/part-00120-hashredacted.parquet, range: 0-662, partition values: [4d]
> 18/11/27 14:09:09 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ac/part-00119-hashredacted.parquet, range: 0-662, partition values: [ac]
> 18/11/27 14:09:10 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=24/part-00139-hashredacted.parquet, range: 0-662, partition values: [24]
> 18/11/27 14:09:11 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=fd/part-00167-hashredacted.parquet, range: 0-662, partition values: [fd]
> 18/11/27 14:09:11 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=52/part-00033-hashredacted.parquet, range: 0-662, partition values: [52]
> 18/11/27 14:09:12 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ab/part-00083-hashredacted.parquet, range: 0-662, partition values: [ab]
> 18/11/27 14:09:12 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=f8/part-00018-hashredacted.parquet, range: 0-662, partition values: [f8]
> 18/11/27 14:09:13 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=7a/part-00093-hashredacted.parquet, range: 0-662, partition values: [7a]
> 18/11/27 14:09:13 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ba/part-00020-hashredacted.parquet, range: 0-662, partition values: [ba]
> 18/11/27 14:09:14 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=2d/part-00085-hashredacted.parquet, range: 0-662, partition values: [2d]
> 18/11/27 14:09:15 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=03/part-00099-hashredacted.parquet, range: 0-662, partition values: [03]
> 18/11/27 14:09:15 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=57/part-00196-hashredacted.parquet, range: 0-662, partition values: [57]
> 18/11/27 14:09:16 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=81/part-00122-hashredacted.parquet, range: 0-662, partition values: [81]
> 18/11/27 14:09:17 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=6d/part-00167-hashredacted.parquet, range: 0-662, partition values: [6d]
> 18/11/27 14:09:17 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=36/part-00154-hashredacted.parquet, range: 0-662, partition values: [36]
> 18/11/27 14:09:18 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4b/part-00093-hashredacted.parquet, range: 0-662, partition values: [4b]
> {code}
> After some investigation, we've isolated the issue to
> [https://github.com/apache/spark/blob/02b510728c31b70e6035ad541bfcdc2b59dcd79a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L132-L136]
>
> In the inferPartitioning method, 2.3.2 sets the type inference to false by default:
> {code:java}
> val spec = PartitioningUtils.parsePartitions(
> leafDirs,
> typeInference = false,
> basePaths = basePaths,
> timeZoneId = timeZoneId){code}
> However, in version 2.4.0, the typeInference flag has been replace with a config flag
> [https://github.com/apache/spark/blob/075447b3965489ffba4e6afb2b120880bc307505/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L129-L133]
>
> {code:java}
> val inferredPartitionSpec = PartitioningUtils.parsePartitions(
> leafDirs,
> typeInference = sparkSession.sessionState.conf.partitionColumnTypeInferenceEnabled,
> basePaths = basePaths,
> timeZoneId = timeZoneId){code}
> And this conf's default value is true
> {code:java}
> val PARTITION_COLUMN_TYPE_INFERENCE =
> buildConf("spark.sql.sources.partitionColumnTypeInference.enabled")
> .doc("When true, automatically infer the data types for partitioned columns.")
> .booleanConf
> .createWithDefault(true){code}
> [https://github.com/apache/spark/blob/075447b3965489ffba4e6afb2b120880bc307505/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L636-L640]
>
> I was wondering if a bug report would be appropriate to preserve backwards compatibility and change the default conf value to false.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org