[GitHub] [spark] HeartSaVioR opened a new pull request #25987: [SPARK-29314][SS] Don't overwrite "updated" of state operator metric to 0 if empty batch is run

[GitBox <gi...@apache.org>]

HeartSaVioR opened a new pull request #25987: [SPARK-29314][SS] Don't overwrite "updated" of state operator metric to 0 if empty batch is run
URL: https://github.com/apache/spark/pull/25987
 
 
   ### What changes were proposed in this pull request?
   
   This patch fixes the behavior of ProgressReporter which always overwrite the value of "updated" of state operator to 0 if there's no new data. The behavior is correct only when we copy the state progress from "previous" executed plan, meaning no batch has been run. (Nonzero value of "updated" would be odd if batch didn't run, so it was correct.) 
   
   It was safe to assume no data is no batch, but SPARK-24156 enables empty data can run the batch if Spark needs to deal with watermark. After the patch, it only overwrites the value if both two conditions are met: 1) no data 2) no batch. 
   
   ### Why are the changes needed?
   
   It doesn't reflect correct metrics when empty batch is run.
   
   ### Does this PR introduce any user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Modified UT. Note that FlatMapGroupsWithState increases the value of "updated" when state rows are removed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org