You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2020/05/26 05:30:40 UTC
[spark] branch branch-3.0 updated: [SPARK-31792][SS][DOCS]
Introduce the structured streaming UI in the Web UI doc
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new cefdd09 [SPARK-31792][SS][DOCS] Introduce the structured streaming UI in the Web UI doc
cefdd09 is described below
commit cefdd09601d0a11cbfeec800dc00aa5c99b313e9
Author: Xingcan Cui <xc...@apache.org>
AuthorDate: Tue May 26 14:27:42 2020 +0900
[SPARK-31792][SS][DOCS] Introduce the structured streaming UI in the Web UI doc
### What changes were proposed in this pull request?
This PR adds the structured streaming UI introduction to the Web UI doc.
### Why are the changes needed?
The structured streaming web UI introduced before was missing from the Web UI documentation.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
N.A.
Closes #28609 from xccui/ss-ui-doc.
Authored-by: Xingcan Cui <xc...@apache.org>
Signed-off-by: HyukjinKwon <gu...@apache.org>
(cherry picked from commit 8ba2b47737f83bd27ebef75dea9e73b3649c2ec1)
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
docs/img/webui-structured-streaming-detail.png | Bin 0 -> 179135 bytes
docs/web-ui.md | 28 +++++++++++++++++++++++++
2 files changed, 28 insertions(+)
diff --git a/docs/img/webui-structured-streaming-detail.png b/docs/img/webui-structured-streaming-detail.png
new file mode 100644
index 0000000..f485052
Binary files /dev/null and b/docs/img/webui-structured-streaming-detail.png differ
diff --git a/docs/web-ui.md b/docs/web-ui.md
index 251e0a0..6653b00 100644
--- a/docs/web-ui.md
+++ b/docs/web-ui.md
@@ -406,6 +406,34 @@ Here is the list of SQL metrics:
</table>
+## Structured Streaming Tab
+When running Structured Streaming jobs in micro-batch mode, a Structured Streaming tab will be
+available on the Web UI. The overview page displays some brief statistics for running and completed
+queries. Also, you can check the latest exception of a failed query. For detailed statistics, please
+click a "run id" in the tables.
+
+<p style="text-align: center;">
+ <img src="img/webui-structured-streaming-detail.png" title="Structured Streaming Query Statistics" alt="Structured Streaming Query Statistics">
+</p>
+
+The statistics page displays some useful metrics for insight into the status of your streaming
+queries. Currently, it contains the following metrics.
+
+* **Input Rate.** The aggregate (across all sources) rate of data arriving.
+* **Process Rate.** The aggregate (across all sources) rate at which Spark is processing data.
+* **Input Rows.** The aggregate (across all sources) number of records processed in a trigger.
+* **Batch Duration.** The process duration of each batch.
+* **Operation Duration.** The amount of time taken to perform various operations in milliseconds.
+The tracked operations are listed as follows.
+ * addBatch: Adds result data of the current batch to the sink.
+ * getBatch: Gets a new batch of data to process.
+ * latestOffset: Gets the latest offsets for sources.
+ * queryPlanning: Generates the execution plan.
+ * walCommit: Writes the offsets to the metadata log.
+
+As an early-release version, the statistics page is still under development and will be improved in
+future releases.
+
## Streaming Tab
The web UI includes a Streaming tab if the application uses Spark streaming. This tab displays
scheduling delay and processing time for each micro-batch in the data stream, which can be useful
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org