[GitHub] [spark] jroof88 edited a comment on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType

[GitBox <gi...@apache.org>]

jroof88 edited a comment on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-730689336


   @HyukjinKwon I thought I would throw an example here for good measure:
   ```python
   
   import json
   from pyspark.sql.types import StructType
   
   df = spark.sql("SELECT * FROM XXX.YYY")
   schema_dict = df.schema.jsonValue()
   schema_dict['fields'].append({'metadata': {'comment': 'Jacks Test Comment'}, 'name': 'jacks_test_column', 'type': 'string'})
   print("New Column: ", schema_dict['fields'][len(schema_dict['fields'])-1])
   schema = StructType.fromJson(schema_dict)
   
   '''
   /databricks/spark/python/pyspark/sql/types.py in fromJson(cls, json)
       435         return StructField(json["name"],
       436                            _parse_datatype_json_value(json["type"]),
   --> 437                            json["nullable"],
       438                            json["metadata"])
       439 
   
   KeyError: 'nullable'
   '''
   ```
   Small Example Here where I am trying to add some metadata to a table. The key `nullable` should have a default value so I only have to worry about it when I want to explicitly set it.
   
   I just don't really understand the statement `JSON isn't supposed to be constructed by users` if in fact it can easily be constructed and manipulated by the user. I would bet users manually create schemas from a variety of sources and run into this error. 
   
   In addition, I'm happy to implement it in Scala but I would like the OK in Python before spending time there 😄. Are there are other contributors to PySpark that might have an opinion here? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org