You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/06/08 05:34:37 UTC
[spark] branch branch-3.3 updated: [SPARK-39394][DOCS][SS][3.3] Improve PySpark Structured Streaming page more readable
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 86f1b6bfe39 [SPARK-39394][DOCS][SS][3.3] Improve PySpark Structured Streaming page more readable
86f1b6bfe39 is described below
commit 86f1b6bfe397646697c299ab5af2157cb1dd2e1e
Author: itholic <ha...@databricks.com>
AuthorDate: Wed Jun 8 14:34:25 2022 +0900
[SPARK-39394][DOCS][SS][3.3] Improve PySpark Structured Streaming page more readable
### What changes were proposed in this pull request?
Hotfix https://github.com/apache/spark/pull/36782 for branch-3.3.
### Why are the changes needed?
The improvement of document readability will also improve the usability for PySpark Structured Streaming.
### Does this PR introduce _any_ user-facing change?
Yes, now the documentation is categorized by its class or their own purpose more clearly as below:
### How was this patch tested?
The existing doc build in CI should cover.
Closes #36797 from itholic/SPARK-39394-3.3.
Authored-by: itholic <ha...@databricks.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/docs/source/reference/index.rst | 2 +-
.../{index.rst => pyspark.ss/core_classes.rst} | 26 +++++-------
.../source/reference/{ => pyspark.ss}/index.rst | 23 ++++------
.../{pyspark.ss.rst => pyspark.ss/io.rst} | 47 ++-------------------
.../query_management.rst} | 49 +---------------------
5 files changed, 24 insertions(+), 123 deletions(-)
diff --git a/python/docs/source/reference/index.rst b/python/docs/source/reference/index.rst
index 1d2db3f4a15..127889afb07 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/index.rst
@@ -29,7 +29,7 @@ Pandas API on Spark follows the API specifications of pandas 1.3.
pyspark.sql/index
pyspark.pandas/index
- pyspark.ss
+ pyspark.ss/index
pyspark.ml
pyspark.streaming
pyspark.mllib
diff --git a/python/docs/source/reference/index.rst b/python/docs/source/reference/pyspark.ss/core_classes.rst
similarity index 68%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.ss/core_classes.rst
index 1d2db3f4a15..4160008881c 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.ss/core_classes.rst
@@ -16,22 +16,16 @@
under the License.
-=============
-API Reference
-=============
+============
+Core Classes
+============
-This page lists an overview of all public PySpark modules, classes, functions and methods.
+.. currentmodule:: pyspark.sql.streaming
-Pandas API on Spark follows the API specifications of pandas 1.3.
+.. autosummary::
+ :toctree: api/
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql/index
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ DataStreamReader
+ DataStreamWriter
+ StreamingQuery
+ StreamingQueryManager
diff --git a/python/docs/source/reference/index.rst b/python/docs/source/reference/pyspark.ss/index.rst
similarity index 70%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.ss/index.rst
index 1d2db3f4a15..2cb0b1216ef 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.ss/index.rst
@@ -16,22 +16,15 @@
under the License.
-=============
-API Reference
-=============
+====================
+Structured Streaming
+====================
-This page lists an overview of all public PySpark modules, classes, functions and methods.
-
-Pandas API on Spark follows the API specifications of pandas 1.3.
+This page gives an overview of all public Structed Streaming API.
.. toctree::
- :maxdepth: 2
+ :maxdepth: 2
- pyspark.sql/index
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ core_classes
+ io
+ query_management
diff --git a/python/docs/source/reference/pyspark.ss.rst b/python/docs/source/reference/pyspark.ss/io.rst
similarity index 61%
copy from python/docs/source/reference/pyspark.ss.rst
copy to python/docs/source/reference/pyspark.ss/io.rst
index cace2d5a6bb..da476fb6fac 100644
--- a/python/docs/source/reference/pyspark.ss.rst
+++ b/python/docs/source/reference/pyspark.ss/io.rst
@@ -16,25 +16,9 @@
under the License.
-====================
-Structured Streaming
-====================
-
-Core Classes
-------------
-
-.. currentmodule:: pyspark.sql.streaming
-
-.. autosummary::
- :toctree: api/
-
- DataStreamReader
- DataStreamWriter
- StreamingQuery
- StreamingQueryManager
-
-Input and Output
-----------------
+============
+Input/Output
+============
.. currentmodule:: pyspark.sql.streaming
@@ -61,28 +45,3 @@ Input and Output
DataStreamWriter.queryName
DataStreamWriter.start
DataStreamWriter.trigger
-
-Query Management
-----------------
-
-.. currentmodule:: pyspark.sql.streaming
-
-.. autosummary::
- :toctree: api/
-
- StreamingQuery.awaitTermination
- StreamingQuery.exception
- StreamingQuery.explain
- StreamingQuery.id
- StreamingQuery.isActive
- StreamingQuery.lastProgress
- StreamingQuery.name
- StreamingQuery.processAllAvailable
- StreamingQuery.recentProgress
- StreamingQuery.runId
- StreamingQuery.status
- StreamingQuery.stop
- StreamingQueryManager.active
- StreamingQueryManager.awaitAnyTermination
- StreamingQueryManager.get
- StreamingQueryManager.resetTerminated
diff --git a/python/docs/source/reference/pyspark.ss.rst b/python/docs/source/reference/pyspark.ss/query_management.rst
similarity index 59%
rename from python/docs/source/reference/pyspark.ss.rst
rename to python/docs/source/reference/pyspark.ss/query_management.rst
index cace2d5a6bb..b580015baa3 100644
--- a/python/docs/source/reference/pyspark.ss.rst
+++ b/python/docs/source/reference/pyspark.ss/query_management.rst
@@ -16,54 +16,9 @@
under the License.
-====================
-Structured Streaming
-====================
-
-Core Classes
-------------
-
-.. currentmodule:: pyspark.sql.streaming
-
-.. autosummary::
- :toctree: api/
-
- DataStreamReader
- DataStreamWriter
- StreamingQuery
- StreamingQueryManager
-
-Input and Output
-----------------
-
-.. currentmodule:: pyspark.sql.streaming
-
-.. autosummary::
- :toctree: api/
-
- DataStreamReader.csv
- DataStreamReader.format
- DataStreamReader.json
- DataStreamReader.load
- DataStreamReader.option
- DataStreamReader.options
- DataStreamReader.orc
- DataStreamReader.parquet
- DataStreamReader.schema
- DataStreamReader.text
- DataStreamWriter.foreach
- DataStreamWriter.foreachBatch
- DataStreamWriter.format
- DataStreamWriter.option
- DataStreamWriter.options
- DataStreamWriter.outputMode
- DataStreamWriter.partitionBy
- DataStreamWriter.queryName
- DataStreamWriter.start
- DataStreamWriter.trigger
-
+================
Query Management
-----------------
+================
.. currentmodule:: pyspark.sql.streaming
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org