You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/09/11 01:22:00 UTC
[jira] [Commented] (SPARK-32849) Default options for metadata &
nullable when creating StructType
[ https://issues.apache.org/jira/browse/SPARK-32849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193943#comment-17193943 ]
Apache Spark commented on SPARK-32849:
--------------------------------------
User 'jroof88' has created a pull request for this issue:
https://github.com/apache/spark/pull/29720
> Default options for metadata & nullable when creating StructType
> ----------------------------------------------------------------
>
> Key: SPARK-32849
> URL: https://issues.apache.org/jira/browse/SPARK-32849
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.0.1
> Reporter: Jack Roof
> Priority: Minor
> Labels: Json, PySpark, Schema, structtype
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> When create a StructType from a Python dictionary you utilize the [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571] function. This function then internally calls [StructField.fromJson|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L425-L430] for each field in the list. This function requires that you must have all keys populated:
> * name
> * type
> * nullable
> * metadata
>
> name and type most definitely have to be required. However, if you take a look at the constructor for [StructField|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L395] there are default arguments for nullable and metadata. This is turn makes me think that these fields should be optional when creating a StructType from a Python dict.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org