You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/06/07 05:27:47 UTC
[spark] branch master updated: [SPARK-39394][DOCS][SS] Improve PySpark Structured Streaming page more readable
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b30a080398b [SPARK-39394][DOCS][SS] Improve PySpark Structured Streaming page more readable
b30a080398b is described below
commit b30a080398ba1092093ea3bbd62bdb3a3ce8de03
Author: itholic <ha...@databricks.com>
AuthorDate: Tue Jun 7 14:27:20 2022 +0900
[SPARK-39394][DOCS][SS] Improve PySpark Structured Streaming page more readable
### What changes were proposed in this pull request?
This PR proposes to improve the PySpark Structured Streaming API reference page to be more readable,
So far, the PySpark Structured Streaming API reference page is not-well organized so it's a bit uncomfortable to be read as below:
### Why are the changes needed?
The improvement of document readability will also improve the usability for PySpark Structured Streaming.
### Does this PR introduce _any_ user-facing change?
Yes, now the documentation is categorized by its class or their own purpose more clearly as below:
### How was this patch tested?
The existing doc build in CI should cover.
Closes #36782 from itholic/SPARK-39394.
Authored-by: itholic <ha...@databricks.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/docs/source/reference/index.rst | 2 +-
.../{index.rst => pyspark.ss/core_classes.rst} | 27 +++++-------
.../source/reference/{ => pyspark.ss}/index.rst | 23 ++++------
.../{pyspark.ss.rst => pyspark.ss/io.rst} | 50 ++--------------------
.../query_management.rst} | 50 +---------------------
5 files changed, 25 insertions(+), 127 deletions(-)
diff --git a/python/docs/source/reference/index.rst b/python/docs/source/reference/index.rst
index b16c614d34c..2f316924405 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/index.rst
@@ -29,7 +29,7 @@ Pandas API on Spark follows the API specifications of latest pandas release.
pyspark.sql/index
pyspark.pandas/index
- pyspark.ss
+ pyspark.ss/index
pyspark.ml
pyspark.streaming
pyspark.mllib
diff --git a/python/docs/source/reference/index.rst b/python/docs/source/reference/pyspark.ss/core_classes.rst
similarity index 68%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.ss/core_classes.rst
index b16c614d34c..10c2211ef1d 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.ss/core_classes.rst
@@ -16,22 +16,17 @@
under the License.
-=============
-API Reference
-=============
+============
+Core Classes
+============
-This page lists an overview of all public PySpark modules, classes, functions and methods.
+.. currentmodule:: pyspark.sql.streaming
-Pandas API on Spark follows the API specifications of latest pandas release.
+.. autosummary::
+ :toctree: api/
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql/index
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ DataStreamReader
+ DataStreamWriter
+ StreamingQuery
+ StreamingQueryManager
+ StreamingQueryListener
diff --git a/python/docs/source/reference/index.rst b/python/docs/source/reference/pyspark.ss/index.rst
similarity index 69%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.ss/index.rst
index b16c614d34c..2cb0b1216ef 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.ss/index.rst
@@ -16,22 +16,15 @@
under the License.
-=============
-API Reference
-=============
+====================
+Structured Streaming
+====================
-This page lists an overview of all public PySpark modules, classes, functions and methods.
-
-Pandas API on Spark follows the API specifications of latest pandas release.
+This page gives an overview of all public Structed Streaming API.
.. toctree::
- :maxdepth: 2
+ :maxdepth: 2
- pyspark.sql/index
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ core_classes
+ io
+ query_management
diff --git a/python/docs/source/reference/pyspark.ss.rst b/python/docs/source/reference/pyspark.ss/io.rst
similarity index 59%
copy from python/docs/source/reference/pyspark.ss.rst
copy to python/docs/source/reference/pyspark.ss/io.rst
index d55d46b9139..da476fb6fac 100644
--- a/python/docs/source/reference/pyspark.ss.rst
+++ b/python/docs/source/reference/pyspark.ss/io.rst
@@ -16,26 +16,9 @@
under the License.
-====================
-Structured Streaming
-====================
-
-Core Classes
-------------
-
-.. currentmodule:: pyspark.sql.streaming
-
-.. autosummary::
- :toctree: api/
-
- DataStreamReader
- DataStreamWriter
- StreamingQuery
- StreamingQueryManager
- StreamingQueryListener
-
-Input and Output
-----------------
+============
+Input/Output
+============
.. currentmodule:: pyspark.sql.streaming
@@ -62,30 +45,3 @@ Input and Output
DataStreamWriter.queryName
DataStreamWriter.start
DataStreamWriter.trigger
-
-Query Management
-----------------
-
-.. currentmodule:: pyspark.sql.streaming
-
-.. autosummary::
- :toctree: api/
-
- StreamingQuery.awaitTermination
- StreamingQuery.exception
- StreamingQuery.explain
- StreamingQuery.id
- StreamingQuery.isActive
- StreamingQuery.lastProgress
- StreamingQuery.name
- StreamingQuery.processAllAvailable
- StreamingQuery.recentProgress
- StreamingQuery.runId
- StreamingQuery.status
- StreamingQuery.stop
- StreamingQueryManager.active
- StreamingQueryManager.addListener
- StreamingQueryManager.awaitAnyTermination
- StreamingQueryManager.get
- StreamingQueryManager.removeListener
- StreamingQueryManager.resetTerminated
diff --git a/python/docs/source/reference/pyspark.ss.rst b/python/docs/source/reference/pyspark.ss/query_management.rst
similarity index 60%
rename from python/docs/source/reference/pyspark.ss.rst
rename to python/docs/source/reference/pyspark.ss/query_management.rst
index d55d46b9139..2a8a84d5ae7 100644
--- a/python/docs/source/reference/pyspark.ss.rst
+++ b/python/docs/source/reference/pyspark.ss/query_management.rst
@@ -16,55 +16,9 @@
under the License.
-====================
-Structured Streaming
-====================
-
-Core Classes
-------------
-
-.. currentmodule:: pyspark.sql.streaming
-
-.. autosummary::
- :toctree: api/
-
- DataStreamReader
- DataStreamWriter
- StreamingQuery
- StreamingQueryManager
- StreamingQueryListener
-
-Input and Output
-----------------
-
-.. currentmodule:: pyspark.sql.streaming
-
-.. autosummary::
- :toctree: api/
-
- DataStreamReader.csv
- DataStreamReader.format
- DataStreamReader.json
- DataStreamReader.load
- DataStreamReader.option
- DataStreamReader.options
- DataStreamReader.orc
- DataStreamReader.parquet
- DataStreamReader.schema
- DataStreamReader.text
- DataStreamWriter.foreach
- DataStreamWriter.foreachBatch
- DataStreamWriter.format
- DataStreamWriter.option
- DataStreamWriter.options
- DataStreamWriter.outputMode
- DataStreamWriter.partitionBy
- DataStreamWriter.queryName
- DataStreamWriter.start
- DataStreamWriter.trigger
-
+================
Query Management
-----------------
+================
.. currentmodule:: pyspark.sql.streaming
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org