[jira] [Created] (SPARK-40820) creating StructType from Json

["Anthony Wainer Cachay Guivin (Jira)" <ji...@apache.org>]

Anthony Wainer Cachay Guivin created SPARK-40820:
----------------------------------------------------

             Summary: creating StructType from Json
                 Key: SPARK-40820
                 URL: https://issues.apache.org/jira/browse/SPARK-40820
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 3.3.0
            Reporter: Anthony Wainer Cachay Guivin


When create a StructType from a Python dictionary you utilize the [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571] method.

To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default.
{code:python}
json = {
            "name": "name",
            "type": "string"
        }

StructField.fromJson(json)
{code}
Error:

!image-2022-10-17-15-52-21-787.png!

 

Proposed coding solution:

Instead use indexes for getting from a dictionary, it would be better to use .get
{code:python}
def fromJson(cls, json: Dict[str, Any]) -> "StructField":
        return StructField(
            json["name"],
            _parse_datatype_json_value(json["type"]),
            json.get("nullable"),
            json.get("metadata"),
        )
{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org