[jira] [Assigned] (SPARK-40820) Creating StructType from Json

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-40820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon reassigned SPARK-40820:
------------------------------------

    Assignee: Anthony Wainer Cachay Guivin

> Creating StructType from Json
> -----------------------------
>
>                 Key: SPARK-40820
>                 URL: https://issues.apache.org/jira/browse/SPARK-40820
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, Spark Core
>    Affects Versions: 3.5.0
>            Reporter: Anthony Wainer Cachay Guivin
>            Assignee: Anthony Wainer Cachay Guivin
>            Priority: Minor
>              Labels: pull-request-available
>
> When create a StructType from a Python dictionary you use [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L792] or in scala [DataType.fromJson|https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala#L158C7-L158C15]
> To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default.
> {code:python}
> schema = {
>      "name": "name", "type": "string" 
> }
> StructField.fromJson(schema)
> {code}
> Python Error:
> {code:python}
> from pyspark.sql.types import StructField
> schema = {
>      "name": "c1", "type": "string" 
> }
> StructField.fromJson(schema)
> >>
> Traceback (most recent call last):
> File "code.py", line 90, in runcode
> exec(code, self.locals)
> File "<input>", line 1, in <module>
> File "pyspark/sql/types.py", line 583, in fromJson
> json["nullable"],
> KeyError: 'nullable' 
> {code}
> Scala Error:
> {code:scala}
> val schema = """
>         |{
>         |    "type": "struct",
>         |    "fields": [
>         |        {
>         |            "name": "c1",
>         |            "type": "string",
>         |            "nullable": false
>         |        }
>         |    ]
>         |}
>         |""".stripMargin
> DataType.fromJson(schema)
> >>
> Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field.
> java.lang.IllegalArgumentException: Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field.
> at org.apache.spark.sql.types.DataType$.parseStructField(DataType.scala:268)
> at org.apache.spark.sql.types.DataType$.$anonfun$parseDataType$1(DataType.scala:225)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org