You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/10 06:30:11 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request, #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
HyukjinKwon opened a new pull request, #37461:
URL: https://github.com/apache/spark/pull/37461
### What changes were proposed in this pull request?
This PR proposes to improve the examples in `pyspark.sql.streaming.readwriter` by making each example self-contained with a brief explanation and a bit more realistic example.
### Why are the changes needed?
To make the documentation more readable and able to copy and paste directly in PySpark shell.
### Does this PR introduce _any_ user-facing change?
Yes, it changes the documentation
### How was this patch tested?
Manually ran each doctest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #37461:
URL: https://github.com/apache/spark/pull/37461#discussion_r943059131
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -1176,15 +1459,28 @@ def toTable(
Examples
--------
- >>> sdf.writeStream.format('parquet').queryName('query').toTable('output_table')
- ... # doctest: +SKIP
-
- >>> sdf.writeStream.trigger(processingTime='5 seconds').toTable(
- ... 'output_table',
- ... queryName='that_query',
- ... outputMode="append",
- ... format='parquet',
- ... checkpointLocation='/tmp/checkpoint') # doctest: +SKIP
+ Save a data stream to a table.
+
+ >>> import tempfile
+ >>> import time
+ >>> _ = spark.sql("DROP TABLE IF EXISTS my_table2")
+ >>> with tempfile.TemporaryDirectory() as d:
+ ... # Create a table with Rate source.
+ ... q = spark.readStream.format("rate").option(
+ ... "rowsPerSecond", 10).load().writeStream.toTable(
+ ... "my_table2",
+ ... queryName='that_query',
+ ... outputMode="append",
+ ... format='parquet',
+ ... checkpointLocation=d)
+ ... time.sleep(3)
+ ... q.stop()
+ ... spark.read.table("my_table2").show()
+ ... _ = spark.sql("DROP TABLE my_table2")
+ +...---------+-----+
Review Comment:
Ah OK interesting. The format isn't intuitive but it should be OK of course if doctest requires it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37461:
URL: https://github.com/apache/spark/pull/37461#issuecomment-1211471023
> either 1) you've tested these examples manually or 2) these examples will be automatically tested via CI?
Both, yes :-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37461:
URL: https://github.com/apache/spark/pull/37461#issuecomment-1210220468
cc @viirya and @HeartSaVioR mind taking a look please when you find some time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a diff in pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #37461:
URL: https://github.com/apache/spark/pull/37461#discussion_r942081038
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -908,22 +1168,34 @@ def foreach(self, f: Union[Callable[[Row], None], "SupportsProcess"]) -> "DataSt
Examples
--------
- >>> # Print every row using a function
+ >>> import time
+ >>> df = spark.readStream.format("rate").load()
+
+ Print every row using a function
+
>>> def print_row(row):
... print(row)
...
- >>> writer = sdf.writeStream.foreach(print_row)
- >>> # Print every row using a object with process() method
+ >>> q = df.writeStream.foreach(print_row).start()
+ >>> time.sleep(3)
+ >>> q.stop()
+
+ Print every row using a object with process() method
+
>>> class RowPrinter:
... def open(self, partition_id, epoch_id):
... print("Opened %d, %d" % (partition_id, epoch_id))
... return True
+ ...
... def process(self, row):
... print(row)
+ ...
... def close(self, error):
... print("Closed with error: %s" % str(error))
...
- >>> writer = sdf.writeStream.foreach(RowPrinter())
+ >>> q = df.writeStream.trigger(once=True).foreach(print_row).start()
Review Comment:
ditto
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -775,14 +1022,27 @@ def trigger(
Examples
--------
- >>> # trigger the query for execution every 5 seconds
- >>> writer = sdf.writeStream.trigger(processingTime='5 seconds')
- >>> # trigger the query for just once batch of data
- >>> writer = sdf.writeStream.trigger(once=True)
- >>> # trigger the query for execution every 5 seconds
- >>> writer = sdf.writeStream.trigger(continuous='5 seconds')
- >>> # trigger the query for reading all available data with multiple batches
- >>> writer = sdf.writeStream.trigger(availableNow=True)
+ >>> df = spark.readStream.format("rate").load()
+
+ Trigger the query for execution every 5 seconds
+
+ >>> df.writeStream.trigger(processingTime='5 seconds')
+ <pyspark.sql.streaming.readwriter.DataStreamWriter object ...>
+
+ Trigger the query for just once batch of data
+
+ >>> df.writeStream.trigger(once=True)
Review Comment:
I think `once` is deprecated?
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -638,7 +808,25 @@ def format(self, source: str) -> "DataStreamWriter":
Examples
--------
- >>> writer = sdf.writeStream.format('json')
+ >>> df = spark.readStream.format("rate").load()
+ >>> df.writeStream.format("text")
+ <pyspark.sql.streaming.readwriter.DataStreamWriter object ...>
Review Comment:
Is this redundant? Looks not related to the example below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37461:
URL: https://github.com/apache/spark/pull/37461#discussion_r942353911
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -638,7 +808,25 @@ def format(self, source: str) -> "DataStreamWriter":
Examples
--------
- >>> writer = sdf.writeStream.format('json')
+ >>> df = spark.readStream.format("rate").load()
+ >>> df.writeStream.format("text")
+ <pyspark.sql.streaming.readwriter.DataStreamWriter object ...>
Review Comment:
the purpose is to show the type of `DataStreamWriter`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #37461:
URL: https://github.com/apache/spark/pull/37461#discussion_r943057744
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -1176,15 +1459,28 @@ def toTable(
Examples
--------
- >>> sdf.writeStream.format('parquet').queryName('query').toTable('output_table')
- ... # doctest: +SKIP
-
- >>> sdf.writeStream.trigger(processingTime='5 seconds').toTable(
- ... 'output_table',
- ... queryName='that_query',
- ... outputMode="append",
- ... format='parquet',
- ... checkpointLocation='/tmp/checkpoint') # doctest: +SKIP
+ Save a data stream to a table.
+
+ >>> import tempfile
+ >>> import time
+ >>> _ = spark.sql("DROP TABLE IF EXISTS my_table2")
+ >>> with tempfile.TemporaryDirectory() as d:
+ ... # Create a table with Rate source.
+ ... q = spark.readStream.format("rate").option(
+ ... "rowsPerSecond", 10).load().writeStream.toTable(
+ ... "my_table2",
+ ... queryName='that_query',
+ ... outputMode="append",
+ ... format='parquet',
+ ... checkpointLocation=d)
+ ... time.sleep(3)
+ ... q.stop()
+ ... spark.read.table("my_table2").show()
+ ... _ = spark.sql("DROP TABLE my_table2")
+ +...---------+-----+
Review Comment:
Actually this is not. It leverages `ELLIPSIS` feature in the doctest. Since it replies on timeout (3 secs), I didn't test the exact match of data. In this way, the test passes regardless of how many batches are computed during 3 secs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #37461:
URL: https://github.com/apache/spark/pull/37461#discussion_r942398004
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -1176,15 +1459,28 @@ def toTable(
Examples
--------
- >>> sdf.writeStream.format('parquet').queryName('query').toTable('output_table')
- ... # doctest: +SKIP
-
- >>> sdf.writeStream.trigger(processingTime='5 seconds').toTable(
- ... 'output_table',
- ... queryName='that_query',
- ... outputMode="append",
- ... format='parquet',
- ... checkpointLocation='/tmp/checkpoint') # doctest: +SKIP
+ Save a data stream to a table.
+
+ >>> import tempfile
+ >>> import time
+ >>> _ = spark.sql("DROP TABLE IF EXISTS my_table2")
+ >>> with tempfile.TemporaryDirectory() as d:
+ ... # Create a table with Rate source.
+ ... q = spark.readStream.format("rate").option(
+ ... "rowsPerSecond", 10).load().writeStream.toTable(
+ ... "my_table2",
+ ... queryName='that_query',
+ ... outputMode="append",
+ ... format='parquet',
+ ... checkpointLocation=d)
+ ... time.sleep(3)
+ ... q.stop()
+ ... spark.read.table("my_table2").show()
+ ... _ = spark.sql("DROP TABLE my_table2")
+ +...---------+-----+
Review Comment:
nit: here as well?
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -692,6 +912,28 @@ def partitionBy(self, *cols: str) -> "DataStreamWriter": # type: ignore[misc]
Notes
-----
This API is evolving.
+
+ Examples
+ --------
+ >>> df = spark.readStream.format("rate").load()
+ >>> df.writeStream.partitionBy("value")
+ <pyspark.sql.streaming.readwriter.DataStreamWriter object ...>
+
+ Partition-by timestamp column from Rate source.
+
+ >>> import tempfile
+ >>> import time
+ >>> with tempfile.TemporaryDirectory() as d, tempfile.TemporaryDirectory() as cp:
+ ... df = spark.readStream.format("rate").option("rowsPerSecond", 10).load()
+ ... q = df.writeStream.partitionBy(
+ ... "timestamp").format("parquet").option("checkpointLocation", cp).start(d)
+ ... time.sleep(5)
+ ... q.stop()
+ ... spark.read.schema(df.schema).parquet(d).show()
+ +...---------+-----+
Review Comment:
nit: same here
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -638,7 +808,25 @@ def format(self, source: str) -> "DataStreamWriter":
Examples
--------
- >>> writer = sdf.writeStream.format('json')
+ >>> df = spark.readStream.format("rate").load()
+ >>> df.writeStream.format("text")
+ <pyspark.sql.streaming.readwriter.DataStreamWriter object ...>
+
+ This API allows to configure the source to write. The example below writes a CSV
+ file from Rate source in a streaming manner.
+
+ >>> import tempfile
+ >>> import time
+ >>> with tempfile.TemporaryDirectory() as d, tempfile.TemporaryDirectory() as cp:
+ ... df = spark.readStream.format("rate").load()
+ ... q = df.writeStream.format("csv").option("checkpointLocation", cp).start(d)
+ ... time.sleep(5)
+ ... q.stop()
+ ... spark.read.schema("timestamp TIMESTAMP, value STRING").csv(d).show()
+ +...---------+-----+
Review Comment:
nit: is it broken?
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -775,14 +1022,27 @@ def trigger(
Examples
--------
- >>> # trigger the query for execution every 5 seconds
- >>> writer = sdf.writeStream.trigger(processingTime='5 seconds')
- >>> # trigger the query for just once batch of data
- >>> writer = sdf.writeStream.trigger(once=True)
- >>> # trigger the query for execution every 5 seconds
- >>> writer = sdf.writeStream.trigger(continuous='5 seconds')
- >>> # trigger the query for reading all available data with multiple batches
- >>> writer = sdf.writeStream.trigger(availableNow=True)
+ >>> df = spark.readStream.format("rate").load()
+
+ Trigger the query for execution every 5 seconds
+
+ >>> df.writeStream.trigger(processingTime='5 seconds')
+ <pyspark.sql.streaming.readwriter.DataStreamWriter object ...>
+
+ Trigger the query for just once batch of data
+
+ >>> df.writeStream.trigger(once=True)
Review Comment:
Yeah, better not to guide it, or explicitly mention about deprecation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #37461:
URL: https://github.com/apache/spark/pull/37461#discussion_r943066266
##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -638,7 +808,25 @@ def format(self, source: str) -> "DataStreamWriter":
Examples
--------
- >>> writer = sdf.writeStream.format('json')
+ >>> df = spark.readStream.format("rate").load()
+ >>> df.writeStream.format("text")
+ <pyspark.sql.streaming.readwriter.DataStreamWriter object ...>
Review Comment:
Yeah. It's a separate example. In the documentation, it shows like:
<img width="748" alt="Screen Shot 2022-08-11 at 12 16 45 PM" src="https://user-images.githubusercontent.com/6477701/184059362-4e3ed97d-03b8-4a3e-9735-7a6c19f10107.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
URL: https://github.com/apache/spark/pull/37461
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37461:
URL: https://github.com/apache/spark/pull/37461#issuecomment-1211854687
I am touching these examples a lot. So all posthoc reviews are very appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37461: [SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37461:
URL: https://github.com/apache/spark/pull/37461#issuecomment-1211854292
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org