You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jack Roof (Jira)" <ji...@apache.org> on 2020/09/11 00:48:00 UTC
[jira] [Updated] (SPARK-32849) Default options for `metadata` &
`nullable` when
[ https://issues.apache.org/jira/browse/SPARK-32849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jack Roof updated SPARK-32849:
------------------------------
Description:
When create a StructType from a Python dictionary you utilize the [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571]` function. This function then internally calls [StructField.fromJson|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L425-L430] for each field in the list. This function requires that you must have all keys populated:
* name
* type
* nullable
* metadata
name and type most definitely have to be required. However, if you take a look at the constructor for `[StructField|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L395]` there are default arguments for nullable and metadata. This is turn makes me think that these fields should be optional when creating a StructType from a Python dict.
was:
When create a StructType from a Python dictionary you utilize the `[StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571]` function. This function then internally calls `[StructField.fromJson|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L425-L430]` for each field in the list. This function requires that you must have all keys populated:
* `name`
* `type`
* `nullable`
* `metadata`
`name` and `type` most definitely have to be required. However, if you take a look at the constructor for `[StructField|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L395]` there are default arguments for `nullable` and `metadata`. This is turn makes me think that these fields should be optional when creating a StructType from a Python dict.
> Default options for `metadata` & `nullable` when
> ------------------------------------------------
>
> Key: SPARK-32849
> URL: https://issues.apache.org/jira/browse/SPARK-32849
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.0.1
> Reporter: Jack Roof
> Priority: Minor
> Labels: Json, PySpark, Schema, structtype
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When create a StructType from a Python dictionary you utilize the [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571]` function. This function then internally calls [StructField.fromJson|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L425-L430] for each field in the list. This function requires that you must have all keys populated:
> * name
> * type
> * nullable
> * metadata
>
> name and type most definitely have to be required. However, if you take a look at the constructor for `[StructField|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L395]` there are default arguments for nullable and metadata. This is turn makes me think that these fields should be optional when creating a StructType from a Python dict.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org