[SS] New numSavedStates metric for StateStoreRestoreExec for saved state?

[Jacek Laskowski <ja...@japila.pl>]

Hi,

Just reviewing StateStoreRestoreExec [1] and been wondering how to
know whether a state was available for a key. It has numOutputRows
metric [2], but that gives the number of aggregations from the child
operator only and seems to say nothing about whether state was
available for an aggregation.

What do you think about adding numSavedStates metric to
StateStoreRestoreExec? Or is there a way to find it out already
(perhaps in web UI)?

[1] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L186

[2] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L206

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Spark Structured Streaming (Apache Spark 2.2+)
https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org