You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anthonywainer (via GitHub)" <gi...@apache.org> on 2023/10/21 20:58:42 UTC
[PR] [SPARK-40820][PYTHON] Creating StructType from Json [spark]
anthonywainer opened a new pull request, #43474:
URL: https://github.com/apache/spark/pull/43474
**What changes were proposed in this pull request?**
Get param using .get instead by key
**Why are the changes needed?**
In order to create a StructField from json and avoid having to set implicit values
**Does this PR introduce any user-facing change?**
Yes, avoiding filling json with implicit values
**How was this patch tested?**
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40820][PYTHON][SQL] Creating StructType from Json [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #43474: [SPARK-40820][PYTHON][SQL] Creating StructType from Json
URL: https://github.com/apache/spark/pull/43474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40820][PYTHON][SQL] Creating StructType from Json [spark]
Posted by "anthonywainer (via GitHub)" <gi...@apache.org>.
anthonywainer commented on PR #43474:
URL: https://github.com/apache/spark/pull/43474#issuecomment-1778909243
@HyukjinKwon could you check this, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40820][PYTHON&SCALA] Creating StructType from Json [spark]
Posted by "anthonywainer (via GitHub)" <gi...@apache.org>.
anthonywainer commented on PR #43474:
URL: https://github.com/apache/spark/pull/43474#issuecomment-1774127262
@HyukjinKwon I have re-opened the PR, could you check please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40820][PYTHON][SQL] Creating StructType from Json [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43474:
URL: https://github.com/apache/spark/pull/43474#discussion_r1368037503
##########
python/pyspark/sql/tests/test_types.py:
##########
@@ -1579,6 +1579,12 @@ def test_row_without_field_sorting(self):
self.assertEqual(r, expected)
self.assertEqual(repr(r), "Row(b=1, a=2)")
+ def test_struct_field_from_json(self):
Review Comment:
```suggestion
def test_struct_field_from_json(self):
# SPARK-40820: fromJson with only name and type
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40820][PYTHON][SQL] Creating StructType from Json [spark]
Posted by "anthonywainer (via GitHub)" <gi...@apache.org>.
anthonywainer commented on code in PR #43474:
URL: https://github.com/apache/spark/pull/43474#discussion_r1368660945
##########
python/pyspark/sql/tests/test_types.py:
##########
@@ -1579,6 +1579,12 @@ def test_row_without_field_sorting(self):
self.assertEqual(r, expected)
self.assertEqual(repr(r), "Row(b=1, a=2)")
+ def test_struct_field_from_json(self):
Review Comment:
changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40820][PYTHON][SQL] Creating StructType from Json [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43474:
URL: https://github.com/apache/spark/pull/43474#discussion_r1368037255
##########
sql/core/src/test/scala/org/apache/spark/sql/types/DataTypeTest.scala:
##########
@@ -0,0 +1,68 @@
+/*
Review Comment:
Let's move the tests to `DataTypeSuite.scala`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40820][PYTHON][SQL] Creating StructType from Json [spark]
Posted by "anthonywainer (via GitHub)" <gi...@apache.org>.
anthonywainer commented on code in PR #43474:
URL: https://github.com/apache/spark/pull/43474#discussion_r1368661617
##########
sql/core/src/test/scala/org/apache/spark/sql/types/DataTypeTest.scala:
##########
@@ -0,0 +1,68 @@
+/*
Review Comment:
moved!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-40820][PYTHON][SQL] Creating StructType from Json [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43474:
URL: https://github.com/apache/spark/pull/43474#issuecomment-1782440071
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org