[jira] [Created] (SPARK-10947) With schema inference from JSON into a Dataframe, add option to infer all primitive object types as strings

["Ewan Leith (JIRA)" <ji...@apache.org>]

Ewan Leith created SPARK-10947:
----------------------------------

             Summary: With schema inference from JSON into a Dataframe, add option to infer all primitive object types as strings
                 Key: SPARK-10947
                 URL: https://issues.apache.org/jira/browse/SPARK-10947
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 1.5.1
            Reporter: Ewan Leith
            Priority: Minor


Currently, when a schema is inferred from a JSON file using sqlContext.read.json, the primitive object types are inferred as string, long, boolean, etc.

However, if the inferred type is too specific (JSON obviously does not enforce types itself), this causes issues with merging dataframe schemas.

Instead, we would like an option in the JSON inferField function to treat all primitive objects as strings.

We'll create and submit a pull request for this for review.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org