You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2023/11/06 20:41:51 UTC
[PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
HyukjinKwon opened a new pull request, #43687:
URL: https://github.com/apache/spark/pull/43687
### What changes were proposed in this pull request?
This PR proposes to improve the docstring of `DataFrameReader.json`.
### Why are the changes needed?
For end users, and better usability of PySpark.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes the user facing documentation.
### How was this patch tested?
Manually tested.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #43687: [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json`
URL: https://github.com/apache/spark/pull/43687
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43687:
URL: https://github.com/apache/spark/pull/43687#discussion_r1383959348
##########
python/pyspark/sql/readwriter.py:
##########
@@ -380,22 +380,72 @@ def json(
Examples
--------
- Write a DataFrame into a JSON file and read it back.
+ Example 1: Write a DataFrame into a JSON file and read it back.
>>> import tempfile
>>> with tempfile.TemporaryDirectory() as d:
... # Write a DataFrame into a JSON file
... spark.createDataFrame(
- ... [{"age": 100, "name": "Hyukjin Kwon"}]
+ ... [{"age": 100, "name": "Hyukjin"}]
... ).write.mode("overwrite").format("json").save(d)
...
... # Read the JSON file as a DataFrame.
... spark.read.json(d).show()
- +---+------------+
- |age| name|
- +---+------------+
- |100|Hyukjin Kwon|
- +---+------------+
+ +---+-------+
+ |age| name|
+ +---+-------+
+ |100|Hyukjin|
+ +---+-------+
+
+ Example 2: Read JSON from multiple files in a directory
+
+ >>> import tempfile
+ >>> with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:
+ ... # Write a DataFrame into a JSON file
+ ... spark.createDataFrame(
+ ... [{"age": 30, "name": "Bob"}]
+ ... ).write.mode("overwrite").format("json").save(d1)
+ ...
+ ... # Read the JSON files as a DataFrame.
+ ... spark.createDataFrame(
+ ... [{"age": 25, "name": "Alice"}]
+ ... ).write.mode("overwrite").format("json").save(d2)
+ ... spark.read.json([d1, d2]).show()
+ +---+-----+
+ |age| name|
+ +---+-----+
+ | 25|Alice|
+ | 30| Bob|
+ +---+-----+
+
+ Example 3: Read JSON from an RDD of JSON strings
+
+ >>> json_strings = ["{'name': 'Alice', 'age': 25}", "{'name': 'Bob', 'age': 30}"]
+ >>> rdd = spark.sparkContext.parallelize(json_strings) # doctest: +SKIP
+ >>> df = spark.read.json(rdd) # doctest: +SKIP
+ >>> df.show() # doctest: +SKIP
Review Comment:
Skipped because it doesn't work with Spark Connect, and the result might be indeterministic itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #43687:
URL: https://github.com/apache/spark/pull/43687#discussion_r1384245409
##########
python/pyspark/sql/readwriter.py:
##########
@@ -380,22 +380,72 @@ def json(
Examples
--------
- Write a DataFrame into a JSON file and read it back.
+ Example 1: Write a DataFrame into a JSON file and read it back.
>>> import tempfile
>>> with tempfile.TemporaryDirectory() as d:
... # Write a DataFrame into a JSON file
... spark.createDataFrame(
- ... [{"age": 100, "name": "Hyukjin Kwon"}]
+ ... [{"age": 100, "name": "Hyukjin"}]
... ).write.mode("overwrite").format("json").save(d)
...
... # Read the JSON file as a DataFrame.
... spark.read.json(d).show()
- +---+------------+
- |age| name|
- +---+------------+
- |100|Hyukjin Kwon|
- +---+------------+
+ +---+-------+
+ |age| name|
+ +---+-------+
+ |100|Hyukjin|
+ +---+-------+
+
+ Example 2: Read JSON from multiple files in a directory
+
+ >>> import tempfile
+ >>> with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:
+ ... # Write a DataFrame into a JSON file
+ ... spark.createDataFrame(
+ ... [{"age": 30, "name": "Bob"}]
+ ... ).write.mode("overwrite").format("json").save(d1)
+ ...
+ ... # Read the JSON files as a DataFrame.
+ ... spark.createDataFrame(
+ ... [{"age": 25, "name": "Alice"}]
+ ... ).write.mode("overwrite").format("json").save(d2)
+ ... spark.read.json([d1, d2]).show()
+ +---+-----+
+ |age| name|
+ +---+-----+
+ | 25|Alice|
+ | 30| Bob|
+ +---+-----+
+
+ Example 3: Read JSON from an RDD of JSON strings
+
+ >>> json_strings = ["{'name': 'Alice', 'age': 25}", "{'name': 'Bob', 'age': 30}"]
+ >>> rdd = spark.sparkContext.parallelize(json_strings) # doctest: +SKIP
+ >>> df = spark.read.json(rdd) # doctest: +SKIP
+ >>> df.show() # doctest: +SKIP
Review Comment:
Yea my concern is that if users are using Spark Connect, this example won't work for them...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43687:
URL: https://github.com/apache/spark/pull/43687#issuecomment-1796414232
Thank you @dongjoon-hyun !!!
@allisonwang-db BTW do you plan to do this for all other functions, or some frequently used only?
With my PRs, (almost) all under SPARK-44728 are resolved. Let's get the rest done, or file JIRAs for all (if that's your intention)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #43687:
URL: https://github.com/apache/spark/pull/43687#discussion_r1384164698
##########
python/pyspark/sql/readwriter.py:
##########
@@ -380,22 +380,72 @@ def json(
Examples
--------
- Write a DataFrame into a JSON file and read it back.
+ Example 1: Write a DataFrame into a JSON file and read it back.
>>> import tempfile
>>> with tempfile.TemporaryDirectory() as d:
... # Write a DataFrame into a JSON file
... spark.createDataFrame(
- ... [{"age": 100, "name": "Hyukjin Kwon"}]
+ ... [{"age": 100, "name": "Hyukjin"}]
... ).write.mode("overwrite").format("json").save(d)
...
... # Read the JSON file as a DataFrame.
... spark.read.json(d).show()
- +---+------------+
- |age| name|
- +---+------------+
- |100|Hyukjin Kwon|
- +---+------------+
+ +---+-------+
+ |age| name|
+ +---+-------+
+ |100|Hyukjin|
+ +---+-------+
+
+ Example 2: Read JSON from multiple files in a directory
+
+ >>> import tempfile
+ >>> with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:
+ ... # Write a DataFrame into a JSON file
+ ... spark.createDataFrame(
+ ... [{"age": 30, "name": "Bob"}]
+ ... ).write.mode("overwrite").format("json").save(d1)
+ ...
+ ... # Read the JSON files as a DataFrame.
+ ... spark.createDataFrame(
+ ... [{"age": 25, "name": "Alice"}]
+ ... ).write.mode("overwrite").format("json").save(d2)
+ ... spark.read.json([d1, d2]).show()
+ +---+-----+
+ |age| name|
+ +---+-----+
+ | 25|Alice|
+ | 30| Bob|
+ +---+-----+
+
+ Example 3: Read JSON from an RDD of JSON strings
+
+ >>> json_strings = ["{'name': 'Alice', 'age': 25}", "{'name': 'Bob', 'age': 30}"]
+ >>> rdd = spark.sparkContext.parallelize(json_strings) # doctest: +SKIP
+ >>> df = spark.read.json(rdd) # doctest: +SKIP
+ >>> df.show() # doctest: +SKIP
Review Comment:
Shall we remove or change this RDD example to something that can work with Spark Connect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43687:
URL: https://github.com/apache/spark/pull/43687#discussion_r1384213317
##########
python/pyspark/sql/readwriter.py:
##########
@@ -380,22 +380,72 @@ def json(
Examples
--------
- Write a DataFrame into a JSON file and read it back.
+ Example 1: Write a DataFrame into a JSON file and read it back.
>>> import tempfile
>>> with tempfile.TemporaryDirectory() as d:
... # Write a DataFrame into a JSON file
... spark.createDataFrame(
- ... [{"age": 100, "name": "Hyukjin Kwon"}]
+ ... [{"age": 100, "name": "Hyukjin"}]
... ).write.mode("overwrite").format("json").save(d)
...
... # Read the JSON file as a DataFrame.
... spark.read.json(d).show()
- +---+------------+
- |age| name|
- +---+------------+
- |100|Hyukjin Kwon|
- +---+------------+
+ +---+-------+
+ |age| name|
+ +---+-------+
+ |100|Hyukjin|
+ +---+-------+
+
+ Example 2: Read JSON from multiple files in a directory
+
+ >>> import tempfile
+ >>> with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:
+ ... # Write a DataFrame into a JSON file
+ ... spark.createDataFrame(
+ ... [{"age": 30, "name": "Bob"}]
+ ... ).write.mode("overwrite").format("json").save(d1)
+ ...
+ ... # Read the JSON files as a DataFrame.
+ ... spark.createDataFrame(
+ ... [{"age": 25, "name": "Alice"}]
+ ... ).write.mode("overwrite").format("json").save(d2)
+ ... spark.read.json([d1, d2]).show()
+ +---+-----+
+ |age| name|
+ +---+-----+
+ | 25|Alice|
+ | 30| Bob|
+ +---+-----+
+
+ Example 3: Read JSON from an RDD of JSON strings
+
+ >>> json_strings = ["{'name': 'Alice', 'age': 25}", "{'name': 'Bob', 'age': 30}"]
+ >>> rdd = spark.sparkContext.parallelize(json_strings) # doctest: +SKIP
+ >>> df = spark.read.json(rdd) # doctest: +SKIP
+ >>> df.show() # doctest: +SKIP
Review Comment:
Actually, Spark Connect doesn't have a way to convert DataFrame or RDD. So there's no way of doing similar stuff with Spark Connect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43687:
URL: https://github.com/apache/spark/pull/43687#issuecomment-1799350258
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43687:
URL: https://github.com/apache/spark/pull/43687#discussion_r1385327624
##########
python/pyspark/sql/readwriter.py:
##########
@@ -380,22 +380,72 @@ def json(
Examples
--------
- Write a DataFrame into a JSON file and read it back.
+ Example 1: Write a DataFrame into a JSON file and read it back.
>>> import tempfile
>>> with tempfile.TemporaryDirectory() as d:
... # Write a DataFrame into a JSON file
... spark.createDataFrame(
- ... [{"age": 100, "name": "Hyukjin Kwon"}]
+ ... [{"age": 100, "name": "Hyukjin"}]
... ).write.mode("overwrite").format("json").save(d)
...
... # Read the JSON file as a DataFrame.
... spark.read.json(d).show()
- +---+------------+
- |age| name|
- +---+------------+
- |100|Hyukjin Kwon|
- +---+------------+
+ +---+-------+
+ |age| name|
+ +---+-------+
+ |100|Hyukjin|
+ +---+-------+
+
+ Example 2: Read JSON from multiple files in a directory
+
+ >>> import tempfile
+ >>> with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:
+ ... # Write a DataFrame into a JSON file
+ ... spark.createDataFrame(
+ ... [{"age": 30, "name": "Bob"}]
+ ... ).write.mode("overwrite").format("json").save(d1)
+ ...
+ ... # Read the JSON files as a DataFrame.
+ ... spark.createDataFrame(
+ ... [{"age": 25, "name": "Alice"}]
+ ... ).write.mode("overwrite").format("json").save(d2)
+ ... spark.read.json([d1, d2]).show()
+ +---+-----+
+ |age| name|
+ +---+-----+
+ | 25|Alice|
+ | 30| Bob|
+ +---+-----+
+
+ Example 3: Read JSON from an RDD of JSON strings
+
+ >>> json_strings = ["{'name': 'Alice', 'age': 25}", "{'name': 'Bob', 'age': 30}"]
+ >>> rdd = spark.sparkContext.parallelize(json_strings) # doctest: +SKIP
+ >>> df = spark.read.json(rdd) # doctest: +SKIP
+ >>> df.show() # doctest: +SKIP
Review Comment:
okie dokie. removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org