You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/28 02:06:13 UTC

[GitHub] [spark] jerrypeng opened a new pull request #35985: [SPARK-38670] Add offset commit time to streaming query listener

jerrypeng opened a new pull request #35985:
URL: https://github.com/apache/spark/pull/35985


   
   
   ### What changes were proposed in this pull request?
   Add a metric called "commitOffsets" to the StreamingQueryListener
   
   
   ### Why are the changes needed?
   A good portion of the batch duration is committing offsets at the end of the micro-batch.  The timing for this operation is missing from the durationMs metrics.  Lets add this metric to have a more complete picture of where the time is going during the processing of a micro-batch
   
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   add a test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] jerrypeng commented on pull request #35985: [SPARK-38670][SS] Add offset commit time to streaming query listener

Posted by GitBox <gi...@apache.org>.
jerrypeng commented on pull request #35985:
URL: https://github.com/apache/spark/pull/35985#issuecomment-1081232830


   ![Screen Shot 2022-03-28 at 3 58 17 PM](https://user-images.githubusercontent.com/3613359/160500726-86029536-7cc9-408b-b38e-7946748bb39a.png)
   
   An image from the UI, the new metric shows up correctly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR closed pull request #35985: [SPARK-38670][SS] Add offset commit time to streaming query listener

Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #35985:
URL: https://github.com/apache/spark/pull/35985


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #35985: [SPARK-38670][SS] Add offset commit time to streaming query listener

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35985:
URL: https://github.com/apache/spark/pull/35985#issuecomment-1080285717


   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #35985: [SPARK-38670][SS] Add offset commit time to streaming query listener

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #35985:
URL: https://github.com/apache/spark/pull/35985#issuecomment-1081762663


   Thanks @jerrypeng for your contribution! I merged this to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #35985: [SPARK-38670][SS] Add offset commit time to streaming query listener

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #35985:
URL: https://github.com/apache/spark/pull/35985#issuecomment-1081761298


   Thanks! Merging to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on a change in pull request #35985: [SPARK-38670] Add offset commit time to streaming query listener

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #35985:
URL: https://github.com/apache/spark/pull/35985#discussion_r836009233



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
##########
@@ -316,6 +316,7 @@ class StreamingQuerySuite extends StreamTest with BeforeAndAfter with Logging wi
         assert(query.recentProgress.last.eq(query.lastProgress))
 
         val progress = query.lastProgress
+

Review comment:
       nit: unnecessary change




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] jerrypeng commented on a change in pull request #35985: [SPARK-38670][SS] Add offset commit time to streaming query listener

Posted by GitBox <gi...@apache.org>.
jerrypeng commented on a change in pull request #35985:
URL: https://github.com/apache/spark/pull/35985#discussion_r836919064



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
##########
@@ -316,6 +316,7 @@ class StreamingQuerySuite extends StreamTest with BeforeAndAfter with Logging wi
         assert(query.recentProgress.last.eq(query.lastProgress))
 
         val progress = query.lastProgress
+

Review comment:
       will revert




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] jerrypeng commented on pull request #35985: [SPARK-38670][SS] Add offset commit time to streaming query listener

Posted by GitBox <gi...@apache.org>.
jerrypeng commented on pull request #35985:
URL: https://github.com/apache/spark/pull/35985#issuecomment-1081233031


   @HeartSaVioR thanks for the review.  Can you please take another look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org