[jira] [Commented] (SPARK-29312) Inconsistent behavior on metrics when dropping rows in state between FlatMapGroupWithState and other stateful operators

["Jungtaek Lim (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-29312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941693#comment-16941693 ] 

Jungtaek Lim commented on SPARK-29312:
--------------------------------------

Maybe it is the intention to represent users have removed the state rows, while other stateful operators are evicting state rows. I'll ask on the dev. mailing list to see whether we would like to change the behavior, and close this if we don't want to.

> Inconsistent behavior on metrics when dropping rows in state between FlatMapGroupWithState and other stateful operators
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-29312
>                 URL: https://issues.apache.org/jira/browse/SPARK-29312
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.0.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> Even rows are evicted from state, we don't increase the value of "updated" metrics. The rule applies on streaming aggregations as well as streaming join, but FlatMapGroupsWithState increases the value of "updated" metrics, allowing the metrics to be shown like (total = 0, updated = 1) which looks weird for who are familiar with metrics with others.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org