You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anishshri-db (via GitHub)" <gi...@apache.org> on 2023/02/24 21:26:09 UTC
[GitHub] [spark] anishshri-db opened a new pull request, #40163: [SPARK-42567] Track load time for state store provider and log warning if it exceeds threshold
anishshri-db opened a new pull request, #40163:
URL: https://github.com/apache/spark/pull/40163
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Track load time for state store provider and log warning if it exceeds threshold
### Why are the changes needed?
We have seen that the initial state store provider load can be blocked by external factors such as filesystem initialization. This log enables us to track cases where this load takes too long and we log a warning in such cases.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Augmented some of the tests to verify the logging is working as expected.
Sample logs:
```
13:14:01.715 WARN org.apache.spark.sql.execution.streaming.state.StateStore: Took too long to load state store provider with storeId=StateStoreId[ checkpointRootLocation=file:/Users/anish.shrigondekar/spark/spark/target/tmp/streaming.metadata-a6feb4c9-6143-44ca-b0e6-b207ab397bb8/state, operatorId=0, partitionId=0, storeName=default ], queryRunId=0507dcb7-1c44-436a-95d2-78a8e11af149 and loadTimeMs=2005
13:14:03.723 WARN org.apache.spark.sql.execution.streaming.state.StateStore: Took too long to load state store provider with storeId=StateStoreId[ checkpointRootLocation=file:/Users/anish.shrigondekar/spark/spark/target/tmp/streaming.metadata-a6feb4c9-6143-44ca-b0e6-b207ab397bb8/state, operatorId=0, partitionId=2, storeName=default ], queryRunId=0507dcb7-1c44-436a-95d2-78a8e11af149 and loadTimeMs=2001
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #40163: [SPARK-42567][SS][SQL] Track load time for state store provider and log warning if it exceeds threshold
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #40163:
URL: https://github.com/apache/spark/pull/40163#issuecomment-1445041909
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on pull request #40163: [SPARK-42567][SS][SQL] Track load time for state store provider and log warning if it exceeds threshold
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on PR #40163:
URL: https://github.com/apache/spark/pull/40163#issuecomment-1444530972
@HeartSaVioR - please take a look. Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40163: [SPARK-42567][SS][SQL] Track load time for state store provider and log warning if it exceeds threshold
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #40163:
URL: https://github.com/apache/spark/pull/40163#discussion_r1117774538
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala:
##########
@@ -533,11 +539,22 @@ object StateStore extends Logging {
}
}
- val provider = loadedProviders.getOrElseUpdate(
- storeProviderId,
- StateStoreProvider.createAndInit(
- storeProviderId, keySchema, valueSchema, numColsPrefixKey, storeConf, hadoopConf)
- )
+ // SPARK-42567 - Track load time for state store provider and log warning if takes longer
+ // than 2s.
+ val (provider, loadTimeMs) = Utils.timeTakenMs {
+ loadedProviders.getOrElseUpdate(
+ storeProviderId,
+ StateStoreProvider.createAndInit(
+ storeProviderId, keySchema, valueSchema, numColsPrefixKey, storeConf, hadoopConf)
+ )
+ }
+
+ if (loadTimeMs > 2000L) {
+ logWarning(s"Took too long to load state store provider with " +
Review Comment:
nit: maybe it's not necessary to call out it's too long? we have similar informative warning log message in FlieStreamSource like below:
```
if (listingTimeMs > 2000) {
// Output a warning when listing files uses more than 2 seconds.
logWarning(s"Listed ${files.size} file(s) in $listingTimeMs ms")
} else {
logTrace(s"Listed ${files.size} file(s) in $listingTimeMs ms")
}
```
probably we can just replace "too long" with the elapsed time. The point in the message is "how long it took", and it's placed at the end of "a bit long" line of text.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on pull request #40163: [SPARK-42567][SS][SQL] Track load time for state store provider and log warning if it exceeds threshold
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on PR #40163:
URL: https://github.com/apache/spark/pull/40163#issuecomment-1444903424
@HeartSaVioR - looks like the tests finished fine. Not sure why the Actions result is not updated here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #40163: [SPARK-42567][SS][SQL] Track load time for state store provider and log warning if it exceeds threshold
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #40163:
URL: https://github.com/apache/spark/pull/40163#discussion_r1117787543
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala:
##########
@@ -533,11 +539,22 @@ object StateStore extends Logging {
}
}
- val provider = loadedProviders.getOrElseUpdate(
- storeProviderId,
- StateStoreProvider.createAndInit(
- storeProviderId, keySchema, valueSchema, numColsPrefixKey, storeConf, hadoopConf)
- )
+ // SPARK-42567 - Track load time for state store provider and log warning if takes longer
+ // than 2s.
+ val (provider, loadTimeMs) = Utils.timeTakenMs {
+ loadedProviders.getOrElseUpdate(
+ storeProviderId,
+ StateStoreProvider.createAndInit(
+ storeProviderId, keySchema, valueSchema, numColsPrefixKey, storeConf, hadoopConf)
+ )
+ }
+
+ if (loadTimeMs > 2000L) {
+ logWarning(s"Took too long to load state store provider with " +
Review Comment:
Sure, updated the message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #40163: [SPARK-42567][SS][SQL] Track load time for state store provider and log warning if it exceeds threshold
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #40163: [SPARK-42567][SS][SQL] Track load time for state store provider and log warning if it exceeds threshold
URL: https://github.com/apache/spark/pull/40163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org