You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jack Roof (Jira)" <ji...@apache.org> on 2020/09/11 00:49:00 UTC
[jira] [Commented] (SPARK-32849) Default options for `metadata` &
`nullable` when
[ https://issues.apache.org/jira/browse/SPARK-32849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193935#comment-17193935 ]
Jack Roof commented on SPARK-32849:
-----------------------------------
I plan to tackle this in the next few days! This is my first time contributing to Spark so starting off with a Jira ticket. Let me know if there is anything else I should add!
> Default options for `metadata` & `nullable` when
> ------------------------------------------------
>
> Key: SPARK-32849
> URL: https://issues.apache.org/jira/browse/SPARK-32849
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.0.1
> Reporter: Jack Roof
> Priority: Minor
> Labels: Json, PySpark, Schema, structtype
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When create a StructType from a Python dictionary you utilize the [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571]` function. This function then internally calls [StructField.fromJson|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L425-L430] for each field in the list. This function requires that you must have all keys populated:
> * name
> * type
> * nullable
> * metadata
>
> name and type most definitely have to be required. However, if you take a look at the constructor for `[StructField|https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L395]` there are default arguments for nullable and metadata. This is turn makes me think that these fields should be optional when creating a StructType from a Python dict.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org