You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/10/01 09:03:00 UTC

[jira] [Created] (SPARK-29312) Inconsistent behavior on metrics when dropping rows in state between FlatMapGroupWithState and other stateful operators

Jungtaek Lim created SPARK-29312:
------------------------------------

             Summary: Inconsistent behavior on metrics when dropping rows in state between FlatMapGroupWithState and other stateful operators
                 Key: SPARK-29312
                 URL: https://issues.apache.org/jira/browse/SPARK-29312
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 3.0.0
            Reporter: Jungtaek Lim


Even rows are evicted from state, we don't increase the value of "updated" metrics. The rule applies on streaming aggregations as well as streaming join, but FlatMapGroupsWithState increases the value of "updated" metrics, allowing the metrics to be shown like (total = 0, updated = 1) which looks weird for who are familiar with metrics with others.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org