You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Maxim Gekk (JIRA)" <ji...@apache.org> on 2019/01/17 20:34:00 UTC
[jira] [Updated] (SPARK-26384) CSV schema inferring does not
respect spark.sql.legacy.timeParser.enabled
[ https://issues.apache.org/jira/browse/SPARK-26384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maxim Gekk updated SPARK-26384:
-------------------------------
Issue Type: Sub-task (was: Bug)
Parent: SPARK-26651
> CSV schema inferring does not respect spark.sql.legacy.timeParser.enabled
> -------------------------------------------------------------------------
>
> Key: SPARK-26384
> URL: https://issues.apache.org/jira/browse/SPARK-26384
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Assignee: Maxim Gekk
> Priority: Major
> Fix For: 3.0.0
>
>
> Starting from the commit [https://github.com/apache/spark/commit/f982ca07e80074bdc1e3b742c5e21cf368e4ede2] , add logging like in the comment https://github.com/apache/spark/pull/23150#discussion_r242021998 and run:
> {code:shell}
> $ ./bin/spark-shell --conf spark.sql.legacy.timeParser.enabled=true
> {code}
> and in the shell:
> {code:scala}
> scala> spark.conf.get("spark.sql.legacy.timeParser.enabled")
> res0: String = true
> scala> Seq("2010|10|10").toDF.repartition(1).write.mode("overwrite").text("/tmp/foo")
> scala> spark.read.option("inferSchema", "true").option("header", "false").option("timestampFormat", "yyyy|MM|dd").csv("/tmp/foo").printSchema()
> 18/12/17 12:11:47 ERROR TimestampFormatter: Iso8601TimestampFormatter is being used
> root
> |-- _c0: timestamp (nullable = true)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org