[jira] [Created] (SPARK-34679) inferTimestamp option is missing from the list of options in DataFrameReader.json.

["Prashanth Babu (Jira)" <ji...@apache.org>]

Prashanth Babu created SPARK-34679:
--------------------------------------

             Summary: inferTimestamp option is missing from the list of options in DataFrameReader.json.
                 Key: SPARK-34679
                 URL: https://issues.apache.org/jira/browse/SPARK-34679
             Project: Spark
          Issue Type: Documentation
          Components: Documentation
    Affects Versions: 3.1.1, 3.1.0, 3.0.2, 3.0.1
            Reporter: Prashanth Babu


{color:#e01e5a}inferTimestamp{color} option is missing in the list of options in {color:#e01e5a}DataFrameReader.json{color} method in the API docs missing from the [Scaladocs here|[DataFrameReader.json|https://github.com/apache/spark/blob/v3.1.1/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L432-L520]].

Simiarly in the [Pyspark docs|[pyspark.sql.DataFrameReader.json|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameReader.json.html?highlight=json#pyspark.sql.DataFrameReader.json]] as well.

However we have this blurb in the [migration guide|[Spark 3.0 to 3.0.1 migration guide|https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-30-to-301]]
 * In Spark 3.0, JSON datasource and JSON function {{schema_of_json}} infer TimestampType from string values if they match to the pattern defined by the JSON option {{timestampFormat}}. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option {{inferTimestamp}} to {{true}} to enable such type inference.

We should add this in the documentation as well as there is a possibility that the Data Engineers might not be aware of this option.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org