You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/08/17 22:52:50 UTC
[spark] branch branch-3.4 updated: [SPARK-44859][SS] Fix incorrect property name in structured streaming doc
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 6c1107e449e [SPARK-44859][SS] Fix incorrect property name in structured streaming doc
6c1107e449e is described below
commit 6c1107e449edcb80fe6fc1d29c95c850aab587aa
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Thu Aug 17 15:52:00 2023 -0700
[SPARK-44859][SS] Fix incorrect property name in structured streaming doc
### What changes were proposed in this pull request?
We found that one structured streaming property for asynchronous progress tracking is not correct when comparing with codebase.
### Why are the changes needed?
Fix incorrect property name in structured streaming document.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
None, doc change only.
Closes #42544 from viirya/minor_doc_fix.
Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit 07f70227f8bf81928d98101a88fd2885784451f5)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
docs/structured-streaming-programming-guide.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/structured-streaming-programming-guide.md b/docs/structured-streaming-programming-guide.md
index 206b887ae74..bf8302b82de 100644
--- a/docs/structured-streaming-programming-guide.md
+++ b/docs/structured-streaming-programming-guide.md
@@ -3714,7 +3714,7 @@ The table below describes the configurations for this feature and default values
| Option | Value | Default | Description |
|-------------|-----------------|------------|---------------------|
|asyncProgressTrackingEnabled|true/false|false|enable or disable asynchronous progress tracking|
-|asyncProgressCheckpointingInterval|minutes|1|the interval in which we commit offsets and completion commits|
+|asyncProgressTrackingCheckpointIntervalMs|millisecond|1000|the interval in which we commit offsets and completion commits|
## Limitations
The initial version of the feature has the following limitations:
@@ -3735,7 +3735,7 @@ Also the following error message may be printed in the driver logs:
The offset log for batch x doesn't exist, which is required to restart the query from the latest batch x from the offset log. Please ensure there are two subsequent offset logs available for the latest batch via manually deleting the offset file(s). Please also ensure the latest batch for commit log is equal or one batch earlier than the latest batch for offset log.
```
-This is caused by the fact that when async progress tracking is enabled, the framework will not checkpoint progress for every batch as would be done if async progress tracking is not used. To solve this problem simply re-enable “asyncProgressTrackingEnabled” and set “asyncProgressCheckpointingInterval” to 0 and run the streaming query until at least two micro-batches have been processed. Async progress tracking can be now safely disabled and restarting query should proceed normally.
+This is caused by the fact that when async progress tracking is enabled, the framework will not checkpoint progress for every batch as would be done if async progress tracking is not used. To solve this problem simply re-enable “asyncProgressTrackingEnabled” and set “asyncProgressTrackingCheckpointIntervalMs” to 0 and run the streaming query until at least two micro-batches have been processed. Async progress tracking can be now safely disabled and restarting query should proceed normally.
# Continuous Processing
## [Experimental]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org