You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anishshri-db (via GitHub)" <gi...@apache.org> on 2024/11/21 06:17:33 UTC
[PR] [SPARK-50378] Add custom metric for tracking spent for proc initial state in transformWithState [spark]
anishshri-db opened a new pull request, #48913:
URL: https://github.com/apache/spark/pull/48913
### What changes were proposed in this pull request?
Add custom metric for tracking spent for proc initial state in transformWithState
### Why are the changes needed?
Adds tracking for time spent in populating initial state
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests
```
[info] Run completed in 2 minutes, 38 seconds.
[info] Total number of tests run: 22
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 22, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #48913:
URL: https://github.com/apache/spark/pull/48913#discussion_r1862459541
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateInitialStateSuite.scala:
##########
@@ -395,6 +395,10 @@ class TransformWithStateInitialStateSuite extends StateStoreMetricsTest
AddData(inputData, InitInputRow("k2", "update", 40.0)),
AddData(inputData, InitInputRow("non-exist", "getOption", -1.0)),
CheckNewAnswer(("non-exist", "getOption", -1.0)),
+ Execute { q =>
+ assert(q.lastProgress
Review Comment:
This assertion seems flaky:
- https://github.com/apache/spark/actions/runs/12069501936/job/33656903689
- https://github.com/apache/spark/actions/runs/12060929700/job/33632163805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on PR #48913:
URL: https://github.com/apache/spark/pull/48913#issuecomment-2490166448
cc - @HeartSaVioR - PTAL, thx !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #48913:
URL: https://github.com/apache/spark/pull/48913#issuecomment-2492786192
I'll leave it to you; I won't block this PR for consistency.
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #48913: [SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState
URL: https://github.com/apache/spark/pull/48913
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #48913:
URL: https://github.com/apache/spark/pull/48913#discussion_r1872927183
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateInitialStateSuite.scala:
##########
@@ -395,6 +395,10 @@ class TransformWithStateInitialStateSuite extends StateStoreMetricsTest
AddData(inputData, InitInputRow("k2", "update", 40.0)),
AddData(inputData, InitInputRow("non-exist", "getOption", -1.0)),
CheckNewAnswer(("non-exist", "getOption", -1.0)),
+ Execute { q =>
+ assert(q.lastProgress
Review Comment:
The new failed link i n change pipeline:
- https://github.com/apache/spark/actions/runs/12192978771/job/34014487080
- https://github.com/apache/spark/actions/runs/12191406660/job/34010259589
- https://github.com/apache/spark/actions/runs/12185951744/job/33993383845
- https://github.com/apache/spark/actions/runs/12164293819/job/33925652391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org