You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/11 01:19:16 UTC
[GitHub] [spark] jroof88 opened a new pull request #29720: [SPARK-32849][PYSPARK][WIP] Add default values for non-required keys when creating StructType
jroof88 opened a new pull request #29720:
URL: https://github.com/apache/spark/pull/29720
### What changes were proposed in this pull request?
<!--
When create a StructType from a Python dictionary you utilize the `[StructType.fromJson](https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571)` function. This function then internally calls `[StructField.fromJson](https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L425-L430)` for each field in the list. This function requires that you must have all keys populated.
The constructor for `[StructField](https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L386-L395)` has default arguments:
```
class StructField(DataType):
...
def __init__(self, name, dataType, nullable=True, metadata=None):
...
self.name = name
self.dataType = dataType
self.nullable = nullable
self.metadata = metadata or {}
```
It seems logical to mirror this when creating it from a Python dict. In addition, I added similar logic to `MapType` and `ArrayType` that follow the same pattern of requiring keys in the JSON that have default arguments in the constructor.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
No. There are no user facing changes.
-->
### How was this patch tested?
<!--
Test are currently a work in progress.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488327546
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Correct we have a use case where we build up the JSON elsewhere and we don't want to have to require the default keys. It drives down complexity when defining schemas in external JSON files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29720: [SPARK-32849][PYSPARK][WIP] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-690816310
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488321853
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
The first test case:
```python
(
"ArrayType with default values", ArrayType,
{"elementType": "string"},
ArrayType(StringType())
),
```
Shows that is `containsNull` is not populated in the incoming JSON it will behave the same as if the argument is not given to the `ArrayType` constructor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488745944
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I think going down the rabbit hole of use case is a bad approach. Taking a step back from the use case, the motivation behind this change is for a **consistent API when creating DataTypes** (namely `StructField`, `ArrayType`, & `MapType`). It seems a bit inconsistent that I can do:
```python
>> MapType(StringType(), StringType()) # notice I omit the valueContainsNull argument
MapType(StringType,StringType,true) # valueContainsNull has a default value of True
```
But when using `.fromJson` which is essentially a constructor for `MapType`, with an equivalent JSON
```python
>> MapType.fromJson({"keyType": "string", "valueType": "string"}) # No valueContainsNull argument
KeyError: 'valueContainsNull' # Error thrown tell me I need it in my JSON
```
The same inconsistencies hold true for `StructField` keys `metadata` & `nullable` **_AND_** `ArrayType` key `containsNull`.
On the flip side, lets say I omit a required parameter:
```python
>> MapType(StringType()) # notice I omit the valueType parameter
TypeError: __init__() missing 1 required positional argument: 'valueType' # Error throw telling me I need valueType
```
And when I do the same using a JSON constructing:
```
>> MapType({"keyType": "string"}) # omit the valueType parameter
KeyError: 'valueType' # Error throw telling me I need it in my JSON
``
This change proposes making the constructors, both the Python class constructor and the classMethod `.fromJson`, consistent on what they expect for input.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488324986
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Right so the default value for `containsNull` for `ArrayType` is `True` so this test shows that without supplying it in the JSON or Constructor you get the same result. I will add another `assert` for the resulting JSON.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488326118
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Added!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-729332075
I am still not sure because the JSON isn't supposed to be constructed by users manually as mentioned above. Also, as I mentioned you should fix Scala side as well.
I don't feel strongly about this PR. I am -0. You can cc or find other committers and ask their review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-787211510
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-729332075
I am still not sure because the JSON isn't supposed to be constructed by users manually as mentioned above. I don't feel strongly about this PR. I am -0. You can cc or find other committers and ask their review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r486794529
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488323152
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
It has:
```python
>>> ArrayType(StringType()).jsonValue()
{'type': 'array', 'elementType': 'string', 'containsNull': True}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r487333658
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r487619330
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
In which case `containsNull` does not exist?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488745944
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I think going down the rabbit hole of use case is a bad approach. Taking a step back from the use case, the motivation behind this change is for a **consistent API when creating DataTypes** (namely `StructField`, `ArrayType`, & `MapType`). It seems a bit inconsistent that I can do:
```python
>> MapType(StringType(), StringType()) # notice I omit the valueContainsNull argument
MapType(StringType,StringType,true) # valueContainsNull has a default value of True
```
But when using `.fromJson` which is essentially a constructor for `MapType`, with an equivalent JSON:
```python
>> MapType.fromJson({"keyType": "string", "valueType": "string"}) # No valueContainsNull argument
KeyError: 'valueContainsNull' # Error thrown tell me I need it in my JSON
```
The same inconsistencies hold true for `StructField` keys `metadata` & `nullable` **_AND_** `ArrayType` key `containsNull`.
_On the flip side_, lets say I omit the required parameter `valueType`:
```python
>> MapType(StringType()) # notice I omit the valueType parameter
TypeError: __init__() missing 1 required positional argument: 'valueType' # Error throw telling me I need valueType
```
And when I do the same using a JSON constructing:
```python
>> MapType({"keyType": "string"}) # omit the valueType parameter
KeyError: 'valueType' # Error throw telling me I need it in my JSON
```
This change proposes making the constructors, both the Python class constructor and the classMethod `.fromJson`, consistent on what they expect for input.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488327546
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Correct we have a use case where we build up the JSON elsewhere and we don't want to have to require the default keys.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488745944
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I think going down the rabbit hole of use case is a bad approach. Taking a step back from the use case, the motivation behind this change is for a **consistent API when creating DataTypes** (namely `StructField`, `ArrayType`, & `MapType`). It seems a bit inconsistent that I can do:
```python
>> MapType(StringType(), StringType()) # notice I omit the valueContainsNull argument
MapType(StringType,StringType,true) # valueContainsNull has a default value of True
```
But when using `.fromJson` which is essentially a constructor for `MapType`, with an equivalent JSON:
```python
>> MapType.fromJson({"keyType": "string", "valueType": "string"}) # No valueContainsNull argument
KeyError: 'valueContainsNull' # Error thrown tell me I need it in my JSON
```
The same inconsistencies hold true for `StructField` keys `metadata` & `nullable` **_AND_** `ArrayType` key `containsNull`.
On the flip side, lets say I omit the required parameter `valueType`:
```python
>> MapType(StringType()) # notice I omit the valueType parameter
TypeError: __init__() missing 1 required positional argument: 'valueType' # Error throw telling me I need valueType
```
And when I do the same using a JSON constructing:
```python
>> MapType({"keyType": "string"}) # omit the valueType parameter
KeyError: 'valueType' # Error throw telling me I need it in my JSON
```
This change proposes making the constructors, both the Python class constructor and the classMethod `.fromJson`, consistent on what they expect for input.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29720: [SPARK-32849][PYSPARK][WIP] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-690816310
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #29720:
URL: https://github.com/apache/spark/pull/29720
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r489108002
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
The JSON format itself isn't supposed to be manually constructed. I think it doesn't necessarily have to be consistent with the constructor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29720: [SPARK-32849][PYSPARK][WIP] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-690816764
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r486794529
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29720: [SPARK-32849][PYSPARK][WIP] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-690816310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gwax commented on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
gwax commented on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-787405137
Well, that's a disappointing take on things, eh @jroof88
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gwax commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
gwax commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r489967059
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Unless there are plans to remove `.fromJson`, it is a publicly exposed interface and, I dare say, a rather useful one.
JSON is currently the only schema definition structure that is a) human readable, b) machine readable without `exec`, and c) easy to generate with anything other than Python / Java.
As far as I can tell, this PR:
- Adds additional test coverage for an existing component
- Makes an existing component more flexible for some use cases
- Does not reduce any existing functionality
Getting to use cases, I have frequently found value in providing a machine readable schema that can be validated with JSON schema and used as a unit and integration tests to verify expected schema against a SQL file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488745944
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I think going down the rabbit hole of use case is a bad approach. Taking a step back from the use case, the motivation behind this change is for a **consistent API when creating DataTypes** (namely `StructField`, `ArrayType`, & `MapType`). It seems a bit inconsistent that I can do:
```python
>> MapType(StringType(), StringType()) # notice I omit the valueContainsNull argument
MapType(StringType,StringType,true) # valueContainsNull has a default value of True
```
But when using `.fromJson` which is essentially a constructor for `MapType`, with an equivalent JSON:
```python
>> MapType.fromJson({"keyType": "string", "valueType": "string"}) # No valueContainsNull key
KeyError: 'valueContainsNull' # Error thrown tell me I need valueContainsNull in my JSON
```
The same inconsistencies hold true for `StructField` keys `metadata` & `nullable` **_AND_** `ArrayType` key `containsNull`.
_On the flip side_, lets say I omit the required parameter `valueType`:
```python
>> MapType(StringType()) # notice I omit the valueType argument
TypeError: __init__() missing 1 required positional argument: 'valueType' # Error throw telling me I need valueType
```
And when I do the same using a JSON constructing:
```python
>> MapType({"keyType": "string"}) # omit the valueType key
KeyError: 'valueType' # Error throw telling me I need it in my JSON
```
This change proposes making the constructors, both the Python class constructor and the classMethod `.fromJson`, consistent on what they expect for input.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r487148520
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-730689336
@HyukjinKwon I thought I would throw an example here for good measure:
```python
import json
from pyspark.sql.types import StructType
df = spark.sql("SELECT * FROM XXX.YYY")
schema_dict = df.schema.jsonValue()
schema_dict['fields'].append({'metadata': {'comment': 'Jacks Test Comment'}, 'name': 'jacks_test_column', 'type': 'string'})
print("New Column: ", schema_dict['fields'][len(schema_dict['fields'])-1])
schema = StructType.fromJson(schema_dict)
'''
/databricks/spark/python/pyspark/sql/types.py in fromJson(cls, json)
435 return StructField(json["name"],
436 _parse_datatype_json_value(json["type"]),
--> 437 json["nullable"],
438 json["metadata"])
439
KeyError: 'nullable'
'''
```
Small Example Here where I am trying to add some metadata to a table.
I just don't really understand the statement `JSON isn't supposed to be constructed by users` if in fact it can easily be constructed and manipulated by the user. I would bet users manually create schemas from a variety of sources and run into this error.
In addition, I'm happy to implement it in Scala but I would like the OK in Python before spending time there 😄
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r487148520
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I refactored the test to include a roundtrip test for `jsonValue()` 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r486794746
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r489549168
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
> The JSON format itself isn't supposed to be manually constructed
I see where you're coming from here but the function `.fromJson` for for `ArrayType`, `MapType`, `StructField`, and `StructType` is exposed to developers so it _can_ be used. I have a strong hunch people do use it as it is an easy way to generate and store schemas. I'm curious what it would take to get this through? It's _seems_ like a fairly logical change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488334221
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
I don't think the format is meant to be exposed from end users. It's just for internal ser/de purpose that is supposed to be roundtrip between `jsonValue` and `fromJson` to workaround Py4J limitation.
BTW, here isn't only place where you should handle JSON ser/de. For example, you should also change Scala side at `DataType.parseDataType`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r487148520
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`. With this approach, the `fromJson` call to the constructors of the changes classes emulates the default arguments so the `jsonValue` function will behave in the same way when called upon.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-729306342
@HyukjinKwon Wanted to bump this PR as it is becoming a tad stale and I have run into issues that this PR would fix a few times over the past few months. My [last comment](https://github.com/apache/spark/pull/29720#discussion_r489549168) details on why this is a compatible change and a beneficial API. Let me know your thoughts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488745944
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I think going down the rabbit hole of use case is a bad approach. Taking a step back from the use case, the motivation behind this change is for a **consistent API when creating DataTypes** (namely `StructField`, `ArrayType`, & `MapType`). It seems a bit inconsistent that I can do:
```python
>> MapType(StringType(), StringType()) // notice I omit the valueContainsNull argument
MapType(StringType,StringType,true) // valueContainsNull has a default value of True
```
But when using `.fromJson` which is essentially a constructor for `MapType`, with an equivalent JSON
```python
>> MapType.fromJson({"keyType": "string", "valueType": "string"}) // No valueContainsNull argument
KeyError: 'valueContainsNull' // Error thrown tell me I need it in my JSON
```
The same inconsistencies hold true for `StructField` keys `metadata` & `nullable` **_AND_** `ArrayType` key `containsNull`.
On the flip side, lets say I omit a required parameter:
```python
>> MapType(StringType()) // notice I omit the valueType parameter
TypeError: __init__() missing 1 required positional argument: 'valueType' // Error throw telling me I need valueType
```
And when I do the same using a JSON constructing:
```
>> MapType({"keyType": "string"}) // omit the valueType parameter
KeyError: 'valueType' // Error throw telling me I need it in my JSON
``
This change proposes making the constructors, both the Python class constructor and the classMethod `.fromJson`, consistent on what they expect for input.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r486794529
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-729332075
I am still not sure because the JSON isn't supposed to be constructed by users manually as mentioned above. Also, as I mentioned you should fix Scala side as well at `DataType.parseDataType`.
I don't feel strongly about this PR. I am -0. You can cc or find other committers and ask their review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 edited a comment on pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 edited a comment on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-730689336
@HyukjinKwon I thought I would throw an example here for good measure:
```python
import json
from pyspark.sql.types import StructType
df = spark.sql("SELECT * FROM XXX.YYY")
schema_dict = df.schema.jsonValue()
schema_dict['fields'].append({'metadata': {'comment': 'Jacks Test Comment'}, 'name': 'jacks_test_column', 'type': 'string'})
print("New Column: ", schema_dict['fields'][len(schema_dict['fields'])-1])
schema = StructType.fromJson(schema_dict)
'''
/databricks/spark/python/pyspark/sql/types.py in fromJson(cls, json)
435 return StructField(json["name"],
436 _parse_datatype_json_value(json["type"]),
--> 437 json["nullable"],
438 json["metadata"])
439
KeyError: 'nullable'
'''
```
Small Example Here where I am trying to add some metadata to a table. The key `nullable` should have a default value so I only have to worry about it when I want to explicitly set it.
I just don't really understand the statement `JSON isn't supposed to be constructed by users` if in fact it can easily be constructed and manipulated by the user. I would bet users manually create schemas from a variety of sources and run into this error.
In addition, I'm happy to implement it in Scala but I would like the OK in Python before spending time there 😄. Are there are other contributors to PySpark that might have an opinion here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r489549168
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
> The JSON format itself isn't supposed to be manually constructed
I see where you're coming from here but the function `.fromJson` for for `ArrayType`, `MapType`, `StructField`, and `StructType` is exposed to developers so it _can_ be used. I have a strong hunch people do use it as it is an easy way to generate and store schemas. I'm curious what it would take to get this through? It's _seems_ like a fairly logical change to make the interface for these types more flexible. There is no user facing changes as it only creates a default value when the key does not exist.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r486794529
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
return ArrayType(_parse_datatype_json_value(json["elementType"]),
- json["containsNull"])
+ json.get("containsNull", True))
Review comment:
Just to clarify, does this cause any user-facing error? This isn't an API so we're good if it works internally.
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Just to clarify, does this cause any user-facing error? As long as we can roundtrip jsonValue and fromJson, I think we're good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29720: [SPARK-32849][PYSPARK][WIP] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-690816310
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488745944
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
@HyukjinKwon I think going down the rabbit hole of use case is a bad approach. Taking a step back from the use case, the motivation behind this change is for a **consistent API when creating DataTypes** (namely `StructField`, `ArrayType`, & `MapType`). It seems a bit inconsistent that I can do:
```python
>> MapType(StringType(), StringType()) # notice I omit the valueContainsNull argument
MapType(StringType,StringType,true) # valueContainsNull has a default value of True
```
But when using `.fromJson` which is essentially a constructor for `MapType`, with an equivalent JSON:
```python
>> MapType.fromJson({"keyType": "string", "valueType": "string"}) # No valueContainsNull argument
KeyError: 'valueContainsNull' # Error thrown tell me I need it in my JSON
```
The same inconsistencies hold true for `StructField` keys `metadata` & `nullable` **_AND_** `ArrayType` key `containsNull`.
On the flip side, lets say I omit the required parameter `valueType`:
```python
>> MapType(StringType()) # notice I omit the valueType parameter
TypeError: __init__() missing 1 required positional argument: 'valueType' # Error throw telling me I need valueType
```
And when I do the same using a JSON constructing:
```
>> MapType({"keyType": "string"}) # omit the valueType parameter
KeyError: 'valueType' # Error throw telling me I need it in my JSON
``
This change proposes making the constructors, both the Python class constructor and the classMethod `.fromJson`, consistent on what they expect for input.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r488326022
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
I mean I am trying to understand why you want this change. For example, does it affect anything in the roundtrip between `jsonValue` and `fromJson`, or are you trying to build up the JSON by yourself somewhere?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jroof88 commented on a change in pull request #29720: [SPARK-32849][PYSPARK] Add default values for non-required keys when creating StructType
Posted by GitBox <gi...@apache.org>.
jroof88 commented on a change in pull request #29720:
URL: https://github.com/apache/spark/pull/29720#discussion_r487148520
##########
File path: python/pyspark/sql/types.py
##########
@@ -305,7 +305,7 @@ def jsonValue(self):
@classmethod
def fromJson(cls, json):
Review comment:
Nope it won't cause any user-facing error. By doing:
```python
json.get("containsNull", True)
```
The code is getting the value of the `containsNull` key from the JSON if it exists. If the key does not, it will be `True`. This matches the constructor of `ArrayType` which has `True` as the default argument for `containsNull`.
I'm happy to add some tests if you believe it is needed but this is be a no-op change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org