You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/04 18:45:15 UTC
[GitHub] [spark] khalidmammadov opened a new pull request, #37408: [WIP] Add doc string to StructType.fromJson
khalidmammadov opened a new pull request, #37408:
URL: https://github.com/apache/spark/pull/37408
### What changes were proposed in this pull request?
Documentation provided for StructType.fromJson method
### Why are the changes needed?
To make it easy for a user to understand this method and use it. It was inspired by [this SO question](https://stackoverflow.com/questions/73233593/pyspark-typeerror-col-should-be-column/73235143)
### Does this PR introduce _any_ user-facing change?
Yes, adding missing documentation
### How was this patch tested?
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #37408:
URL: https://github.com/apache/spark/pull/37408#issuecomment-1215018313
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37408: [SPARK-39982][WIP] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37408:
URL: https://github.com/apache/spark/pull/37408#issuecomment-1207133120
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #37408:
URL: https://github.com/apache/spark/pull/37408#issuecomment-1214153608
Minor changes requested above, but if you can make the tweaks, I think this is OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a diff in pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
srowen commented on code in PR #37408:
URL: https://github.com/apache/spark/pull/37408#discussion_r940401977
##########
python/pyspark/sql/types.py:
##########
@@ -815,6 +815,54 @@ def jsonValue(self) -> Dict[str, Any]:
@classmethod
def fromJson(cls, json: Dict[str, Any]) -> "StructType":
+ """
+ Constructs StructType from a scheme defined in json format
+
+ Parameters
+ ----------
+ json : Dict of str/any or a dict like object e.g. json object
Review Comment:
Dict -> dict, "dict like" -> dict-like. Maybe say a tiny bit more about what dict of str/any is
##########
python/pyspark/sql/types.py:
##########
@@ -815,6 +815,54 @@ def jsonValue(self) -> Dict[str, Any]:
@classmethod
def fromJson(cls, json: Dict[str, Any]) -> "StructType":
+ """
+ Constructs StructType from a scheme defined in json format
Review Comment:
Nit: scheme -> schema, json -> JSON
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov commented on a diff in pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on code in PR #37408:
URL: https://github.com/apache/spark/pull/37408#discussion_r945256211
##########
python/pyspark/sql/types.py:
##########
@@ -815,6 +815,96 @@ def jsonValue(self) -> Dict[str, Any]:
@classmethod
def fromJson(cls, json: Dict[str, Any]) -> "StructType":
+ """
+ Constructs :class`StructType` from a schema defined in json format
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov commented on pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on PR #37408:
URL: https://github.com/apache/spark/pull/37408#issuecomment-1214318044
Additionally I have checked how it looks using `bundle exec jekyll serve --host 0.0.0.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
URL: https://github.com/apache/spark/pull/37408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov commented on a diff in pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on code in PR #37408:
URL: https://github.com/apache/spark/pull/37408#discussion_r945256446
##########
python/pyspark/sql/types.py:
##########
@@ -815,6 +815,96 @@ def jsonValue(self) -> Dict[str, Any]:
@classmethod
def fromJson(cls, json: Dict[str, Any]) -> "StructType":
+ """
+ Constructs :class`StructType` from a schema defined in json format
+
+ Parameters
+ ----------
+ json : dict or a dict-like object e.g. json object
Review Comment:
Added JSON schema it must adhere to. I can add the library info but here I am saying that "it must be any dict like object for example json object". So, it can be any custom Python class instance as long as it behaves like dict or json
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov commented on a diff in pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on code in PR #37408:
URL: https://github.com/apache/spark/pull/37408#discussion_r945256157
##########
python/pyspark/sql/types.py:
##########
@@ -815,6 +815,54 @@ def jsonValue(self) -> Dict[str, Any]:
@classmethod
def fromJson(cls, json: Dict[str, Any]) -> "StructType":
+ """
+ Constructs StructType from a scheme defined in json format
Review Comment:
fixed
##########
python/pyspark/sql/types.py:
##########
@@ -815,6 +815,54 @@ def jsonValue(self) -> Dict[str, Any]:
@classmethod
def fromJson(cls, json: Dict[str, Any]) -> "StructType":
+ """
+ Constructs StructType from a scheme defined in json format
+
+ Parameters
+ ----------
+ json : Dict of str/any or a dict like object e.g. json object
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a diff in pull request #37408: [SPARK-39982][DOC] Add doc string to StructType.fromJson
Posted by GitBox <gi...@apache.org>.
srowen commented on code in PR #37408:
URL: https://github.com/apache/spark/pull/37408#discussion_r942444321
##########
python/pyspark/sql/types.py:
##########
@@ -815,6 +815,96 @@ def jsonValue(self) -> Dict[str, Any]:
@classmethod
def fromJson(cls, json: Dict[str, Any]) -> "StructType":
+ """
+ Constructs :class`StructType` from a schema defined in json format
Review Comment:
json -> JSON everywhere
##########
python/pyspark/sql/types.py:
##########
@@ -815,6 +815,96 @@ def jsonValue(self) -> Dict[str, Any]:
@classmethod
def fromJson(cls, json: Dict[str, Any]) -> "StructType":
+ """
+ Constructs :class`StructType` from a schema defined in json format
+
+ Parameters
+ ----------
+ json : dict or a dict-like object e.g. json object
Review Comment:
Clarify what json object means here. Object from what library?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org