[jira] [Commented] (SPARK-22398) Partition directories with leading 0s cause wrong results

["Marco Gaido (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-22398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226991#comment-16226991 ] 

Marco Gaido commented on SPARK-22398:
-------------------------------------

you just need to set `spark.sql.sources.partitionColumnTypeInference.enabled` to `false`.

> Partition directories with leading 0s cause wrong results
> ---------------------------------------------------------
>
>                 Key: SPARK-22398
>                 URL: https://issues.apache.org/jira/browse/SPARK-22398
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Bogdan Raducanu
>
> Repro case:
> {code}
> spark.range(8).selectExpr("'0' || cast(id as string) as id", "id as b").write.mode("overwrite").partitionBy("id").parquet("/tmp/bug1")
> spark.read.parquet("/tmp/bug1").where("id in ('01')").show
> +---+---+
> |  b| id|
> +---+---+
> +---+---+
> spark.read.parquet("/tmp/bug1").where("id = '01'").show
> +---+---+
> |  b| id|
> +---+---+
> |  1|  1|
> +---+---+
> {code}
> I think somewhere there is some special handling of this case for equals but not the same for IN.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org