[jira] [Updated] (SPARK-40820) Creating StructType from Json

["Anthony Wainer Cachay Guivin (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-40820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Anthony Wainer Cachay Guivin updated SPARK-40820:
-------------------------------------------------
    Summary: Creating StructType from Json  (was: creating StructType from Json)

> Creating StructType from Json
> -----------------------------
>
>                 Key: SPARK-40820
>                 URL: https://issues.apache.org/jira/browse/SPARK-40820
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 3.3.0
>            Reporter: Anthony Wainer Cachay Guivin
>            Priority: Minor
>
> When create a StructType from a Python dictionary you utilize the [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571] method.
> To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default.
> {code:python}
> json = {
>             "name": "name",
>             "type": "string"
>         }
> StructField.fromJson(json)
> {code}
> Error:
> {code:python}
> from pyspark.sql.types import StructField
> json = {
>             "name": "name",
>             "type": "string"
>         }
> StructField.fromJson(json)
> >>
> Traceback (most recent call last):
>   File "code.py", line 90, in runcode
>     exec(code, self.locals)
>   File "<input>", line 1, in <module>
>   File "pyspark/sql/types.py", line 583, in fromJson
>     json["nullable"],
> KeyError: 'nullable' {code}
>  
> Proposed coding solution:
> Instead use indexes for getting from a dictionary, it would be better to use .get
> {code:python}
> def fromJson(cls, json: Dict[str, Any]) -> "StructField":
>         return StructField(
>             json["name"],
>             _parse_datatype_json_value(json["type"]),
>             json.get("nullable"),
>             json.get("metadata"),
>         )
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org