You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/16 00:49:50 UTC
[GitHub] [spark] viirya opened a new pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
viirya opened a new pull request #30789:
URL: https://github.com/apache/spark/pull/30789
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This updates SS documentation to document about State Store and task locality.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
During running some tests for structured streaming, I found state store locality becomes an issue sometimes and it is not very straightforward for end-users. It'd be great if we can document it.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No, only doc change.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
No, only doc change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745691941
cc @HyukjinKwon @HeartSaVioR @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r544094775
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,23 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations stores states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
Review comment:
`The stateful operations stores` -> `The stateful operations store`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745703005
Looks pretty good from my reading. cc @xuanyuanking too FYI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745700671
**[Test build #132844 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132844/testReport)** for PR 30789 at commit [`cec4be9`](https://github.com/apache/spark/commit/cec4be9dc58c82b3d242671f638890d16ea52f82).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r544698190
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
Review comment:
`batches` would also just work - looks like in further sentences you seem to use `batches` simply.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745720752
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132847/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r544110340
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,23 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations stores states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
Review comment:
Fixed. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747145115
**[Test build #132909 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132909/testReport)** for PR 30789 at commit [`d93f6dc`](https://github.com/apache/spark/commit/d93f6dc5628b7514cf2527d49dbf99e3d51b9be3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745733507
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37449/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745700775
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132844/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746828668
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37501/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745700775
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132844/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745728974
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37449/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746020445
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37480/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r543804826
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,12 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations stores states for events in state stores of executors. State stores occupies resources such as memory and disk space to store the states. So it is more efficient to keep a state store provider running in the same executor across different streaming batches. Changing the location of state store provider requires to load from checkpointed states from HDFS in the new executor. The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor. However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones. In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in previous batch will not be unloaded immediately. If in next batch the corresponding state store provider is scheduled on this ex
ecutor again, it could reuse the previous states and save the time of loading checkpointed state. Spark runs a maintenance task which checks and unloads the state store providers which are inactive on the executors.
Review comment:
Okay. Let me break it to few small pieces. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747164001
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37511/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746752710
**[Test build #132899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132899/testReport)** for PR 30789 at commit [`19de6fe`](https://github.com/apache/spark/commit/19de6fe2619f4f5fd1d5c2ebead512a06b10fa6d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745733507
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37449/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r543808416
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,12 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations stores states for events in state stores of executors. State stores occupies resources such as memory and disk space to store the states. So it is more efficient to keep a state store provider running in the same executor across different streaming batches. Changing the location of state store provider requires to load from checkpointed states from HDFS in the new executor. The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor. However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones. In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in previous batch will not be unloaded immediately. If in next batch the corresponding state store provider is scheduled on this ex
ecutor again, it could reuse the previous states and save the time of loading checkpointed state. Spark runs a maintenance task which checks and unloads the state store providers which are inactive on the executors.
+
+For some use cases like processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient. By changing the Spark configs related to task scheduling, for example `spark.locality.wait`, users can config Spark how long to wait to launch data-local task. For stateful operations in Structured Streaming, it can be used to let state store providers running on the same executors across batches.
Review comment:
maybe ...
- `For some use cases like processing` -> `For some use cases such as processing`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r543804073
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,12 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations stores states for events in state stores of executors. State stores occupies resources such as memory and disk space to store the states. So it is more efficient to keep a state store provider running in the same executor across different streaming batches. Changing the location of state store provider requires to load from checkpointed states from HDFS in the new executor. The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor. However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones. In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in previous batch will not be unloaded immediately. If in next batch the corresponding state store provider is scheduled on this ex
ecutor again, it could reuse the previous states and save the time of loading checkpointed state. Spark runs a maintenance task which checks and unloads the state store providers which are inactive on the executors.
Review comment:
Wondering if it's better to add some newlines here to make the diff look better next time ... :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r544685998
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires loading from checkpointed states from HDFS in the new executor.
Review comment:
Mentioning HDFS sounds slightly incorrect - even in built-in state store (named HDFS) you can store the checkpoint to the external storages including object stores.
Probably `loading from checkpoint` would just work - if it doesn't seem to emphasize the overhead, it's OK to explicitly mention the overhead, like `the overhead of loading state from checkpoint depends on the external storage and the size of the state, which tends to hurt the latency of micro-batch run`.
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
Review comment:
Probably explicitly mentioning `microbatches`, or if we still concern about continuous processing, then `during the lifetime of the query`?
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires loading from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers run in the previous batch will not be unloaded immediately.
+If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
+Spark runs a maintenance task which checks and unloads the state store providers that are inactive on the executors.
+
+For some use cases such as processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient.
+By changing the Spark configurations related to task scheduling, for example `spark.locality.wait`, users can configure Spark how long to wait to launch a data-local task.
+For stateful operations in Structured Streaming, it can be used to let state store providers running on the same executors across batches. Specially, users can check the state store metrics
Review comment:
The metric is only provided in built-in HDFS state store provider, so you may want to mention this explicitly, like `Specifically for built-in HDFS state store provider,` or just `for built-in HDFS state store provider,`.
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires loading from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers run in the previous batch will not be unloaded immediately.
+If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
+Spark runs a maintenance task which checks and unloads the state store providers that are inactive on the executors.
+
+For some use cases such as processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient.
Review comment:
Just 2 cents, probably we'd want to mention this near the earliest to explain rationalization about this section (`State Store and task locality`) before explaining details. Without this, end users might not notice why it's important to understand about the details, and simply ignore the section.
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires loading from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers run in the previous batch will not be unloaded immediately.
+If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
+Spark runs a maintenance task which checks and unloads the state store providers that are inactive on the executors.
+
+For some use cases such as processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient.
+By changing the Spark configurations related to task scheduling, for example `spark.locality.wait`, users can configure Spark how long to wait to launch a data-local task.
+For stateful operations in Structured Streaming, it can be used to let state store providers running on the same executors across batches. Specially, users can check the state store metrics
+such as `loadedMapCacheHitCount` and `loadedMapCacheMissCount`. Ideally, it is best if cache missing count is zero that means Spark won't spend time on loading checkpointed state.
+User can increase Spark locality waitting configurations to avoid loading state store providers in different executors across batches.
Review comment:
nit: waitting -> waiting
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires loading from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
Review comment:
I'd rearrange following sentences to explain happy case, and the bad case. Like...
> The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor. If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
>
> However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones. In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers run in the previous batch will not be unloaded immediately.
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires loading from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers run in the previous batch will not be unloaded immediately.
Review comment:
same here. let's avoid mentioning HDFS.
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,25 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires loading from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers run in the previous batch will not be unloaded immediately.
+If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
+Spark runs a maintenance task which checks and unloads the state store providers that are inactive on the executors.
+
+For some use cases such as processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient.
+By changing the Spark configurations related to task scheduling, for example `spark.locality.wait`, users can configure Spark how long to wait to launch a data-local task.
+For stateful operations in Structured Streaming, it can be used to let state store providers running on the same executors across batches. Specially, users can check the state store metrics
+such as `loadedMapCacheHitCount` and `loadedMapCacheMissCount`. Ideally, it is best if cache missing count is zero that means Spark won't spend time on loading checkpointed state.
Review comment:
I don't remember the case, but could you please do quick check that loadedMapCacheMissCount is also increasing for rerunning streaming query? (I roughly remember the value would increase for the case which is not abnormal) If then probably better to not mention `zero` but `minimized`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r544298547
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,23 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires to load from checkpointed states from HDFS in the new executor.
Review comment:
Maybe "requires loading checkpointed states from HDFS ..."
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,23 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires to load from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in the previous batch will not be unloaded immediately.
Review comment:
ran -> run
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,23 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires to load from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in the previous batch will not be unloaded immediately.
+If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
+Spark runs a maintenance task which checks and unloads the state store providers which are inactive on the executors.
Review comment:
nit: "that are inactive"
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,23 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires to load from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in the previous batch will not be unloaded immediately.
+If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
+Spark runs a maintenance task which checks and unloads the state store providers which are inactive on the executors.
+
+For some use cases such as processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient.
+By changing the Spark configurations related to task scheduling, for example `spark.locality.wait`, users can configure Spark how long to wait to launch a data-local task.
+For stateful operations in Structured Streaming, it can be used to let state store providers running on the same executors across batches.
Review comment:
This is good info but can we get more specific about the tuning? if you see what then you change spark.locality.wait to what, up or down, etc? what is too big, too small, etc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r543807072
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,12 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations stores states for events in state stores of executors. State stores occupies resources such as memory and disk space to store the states. So it is more efficient to keep a state store provider running in the same executor across different streaming batches. Changing the location of state store provider requires to load from checkpointed states from HDFS in the new executor. The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor. However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones. In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in previous batch will not be unloaded immediately. If in next batch the corresponding state store provider is scheduled on this ex
ecutor again, it could reuse the previous states and save the time of loading checkpointed state. Spark runs a maintenance task which checks and unloads the state store providers which are inactive on the executors.
+
+For some use cases like processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient. By changing the Spark configs related to task scheduling, for example `spark.locality.wait`, users can config Spark how long to wait to launch data-local task. For stateful operations in Structured Streaming, it can be used to let state store providers running on the same executors across batches.
Review comment:
typos:
- `Spark configs` -> `Spark configurations`
- `can config` -> `can configure`
- `launch data-local task.` -> `launch a data-local task.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745961679
**[Test build #132879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132879/testReport)** for PR 30789 at commit [`b5e7efa`](https://github.com/apache/spark/commit/b5e7efa3a25af12a297a7d8f3c3cd28f535d4ea6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745715636
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37446/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747136319
The PR looks OK to me. If not urgent, let's wait for a day to give others a chance to review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747831153
Thanks all!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746772017
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132899/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r544501647
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,23 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
+So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
+Changing the location of a state store provider requires to load from checkpointed states from HDFS in the new executor.
+
+The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+
+In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in the previous batch will not be unloaded immediately.
+If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
+Spark runs a maintenance task which checks and unloads the state store providers which are inactive on the executors.
+
+For some use cases such as processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient.
+By changing the Spark configurations related to task scheduling, for example `spark.locality.wait`, users can configure Spark how long to wait to launch a data-local task.
+For stateful operations in Structured Streaming, it can be used to let state store providers running on the same executors across batches.
Review comment:
Good point. I added few specific info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745716578
**[Test build #132847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132847/testReport)** for PR 30789 at commit [`23770a8`](https://github.com/apache/spark/commit/23770a8002807c9832fdfe177abbf59af1074d3e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747164018
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37511/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746886717
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37501/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r544731379
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1693,20 +1693,23 @@ end-to-end exactly once per query. Ensuring end-to-end exactly once for the last
The stateful operations store states for events in state stores of executors. State stores occupy resources such as memory and disk space to store the states.
So it is more efficient to keep a state store provider running in the same executor across different streaming batches.
-Changing the location of a state store provider requires loading from checkpointed states from HDFS in the new executor.
+Changing the location of a state store provider requires the extra overhead of loading checkpointed states. The overhead of loading state from checkpoint depends
+on the external storage and the size of the state, which tends to hurt the latency of micro-batch run. For some use cases such as processing very large state data,
+loading new state store providers from checkpointed states can be very time-consuming and inefficient.
The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor.
-However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
-
-In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers run in the previous batch will not be unloaded immediately.
If in the next batch the corresponding state store provider is scheduled on this executor again, it could reuse the previous states and save the time of loading checkpointed states.
+
+However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones.
+In this case, Spark will load state store providers from checkpointed states on new executors. The state store providers run in the previous batch will not be unloaded immediately.
Spark runs a maintenance task which checks and unloads the state store providers that are inactive on the executors.
-For some use cases such as processing very large state data, loading new state store providers from checkpointed states can be very time-consuming and inefficient.
By changing the Spark configurations related to task scheduling, for example `spark.locality.wait`, users can configure Spark how long to wait to launch a data-local task.
Review comment:
OFF TOPIC (for #30770): probably we may need to allow having different timeout between state RDD and others.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747153382
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37511/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746936775
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37501/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745716578
**[Test build #132847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132847/testReport)** for PR 30789 at commit [`23770a8`](https://github.com/apache/spark/commit/23770a8002807c9832fdfe177abbf59af1074d3e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746772017
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132899/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745980732
**[Test build #132879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132879/testReport)** for PR 30789 at commit [`b5e7efa`](https://github.com/apache/spark/commit/b5e7efa3a25af12a297a7d8f3c3cd28f535d4ea6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745981024
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132879/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747145226
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132909/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746014856
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37480/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747131852
Thanks @HeartSaVioR. Addressed the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746752710
**[Test build #132899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132899/testReport)** for PR 30789 at commit [`19de6fe`](https://github.com/apache/spark/commit/19de6fe2619f4f5fd1d5c2ebead512a06b10fa6d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #30789:
URL: https://github.com/apache/spark/pull/30789
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745715636
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37446/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745981024
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747815654
Thanks! Merged into master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746027035
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37480/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745720752
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132847/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745730243
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37449/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745697065
**[Test build #132844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132844/testReport)** for PR 30789 at commit [`cec4be9`](https://github.com/apache/spark/commit/cec4be9dc58c82b3d242671f638890d16ea52f82).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747141263
**[Test build #132909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132909/testReport)** for PR 30789 at commit [`d93f6dc`](https://github.com/apache/spark/commit/d93f6dc5628b7514cf2527d49dbf99e3d51b9be3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745697065
**[Test build #132844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132844/testReport)** for PR 30789 at commit [`cec4be9`](https://github.com/apache/spark/commit/cec4be9dc58c82b3d242671f638890d16ea52f82).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747814997
Given this is a pure doc change, I'll skip retriggering the build. Merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745815380
cc. @tdas @zsxwing @jose-torres @gaborgsomogyi
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746936775
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37501/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-747141263
**[Test build #132909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132909/testReport)** for PR 30789 at commit [`d93f6dc`](https://github.com/apache/spark/commit/d93f6dc5628b7514cf2527d49dbf99e3d51b9be3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745961679
**[Test build #132879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132879/testReport)** for PR 30789 at commit [`b5e7efa`](https://github.com/apache/spark/commit/b5e7efa3a25af12a297a7d8f3c3cd28f535d4ea6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-745720593
**[Test build #132847 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132847/testReport)** for PR 30789 at commit [`23770a8`](https://github.com/apache/spark/commit/23770a8002807c9832fdfe177abbf59af1074d3e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30789:
URL: https://github.com/apache/spark/pull/30789#issuecomment-746771379
**[Test build #132899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132899/testReport)** for PR 30789 at commit [`19de6fe`](https://github.com/apache/spark/commit/19de6fe2619f4f5fd1d5c2ebead512a06b10fa6d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30789: [SPARK-33797][SS][DOCS] Update SS doc about State Store and task locality
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30789:
URL: https://github.com/apache/spark/pull/30789#discussion_r543806306
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1689,6 +1689,12 @@ hence the number is not same as the number of original input rows. You'd like to
There's a known workaround: split your streaming query into multiple queries per stateful operator, and ensure
end-to-end exactly once per query. Ensuring end-to-end exactly once for the last query is optional.
+### State Store and task locality
+
+The stateful operations stores states for events in state stores of executors. State stores occupies resources such as memory and disk space to store the states. So it is more efficient to keep a state store provider running in the same executor across different streaming batches. Changing the location of state store provider requires to load from checkpointed states from HDFS in the new executor. The stateful operations in Structured Streaming queries rely on the preferred location feature of Spark's RDD to run the state store provider on the same executor. However, generally the preferred location is not a hard requirement and it is still possible that Spark schedules tasks to the executors other than the preferred ones. In this case, Spark will load state store providers from checkpointed states on HDFS to new executors. The state store providers ran in previous batch will not be unloaded immediately. If in next batch the corresponding state store provider is scheduled on this ex
ecutor again, it could reuse the previous states and save the time of loading checkpointed state. Spark runs a maintenance task which checks and unloads the state store providers which are inactive on the executors.
Review comment:
typos:
- `State stores occupies` -> `State stores occupy`
- `Changing the location of state store provider` -> `Changing the location of a state store provider`
- `in previous` -> `in the previous`
- `in next` -> `in the next`
- `the time of loading checkpointed state` -> `the time of loading checkpointed states`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org