[jira] [Created] (SPARK-24104) SQLAppStatusListener overwrites metrics onDriverAccumUpdates instead of updating them

["Juliusz Sompolski (JIRA)" <ji...@apache.org>]

Juliusz Sompolski created SPARK-24104:
-----------------------------------------

             Summary: SQLAppStatusListener overwrites metrics onDriverAccumUpdates instead of updating them
                 Key: SPARK-24104
                 URL: https://issues.apache.org/jira/browse/SPARK-24104
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.0
            Reporter: Juliusz Sompolski


SqlAppStatusListener does 
{code}
exec.driverAccumUpdates = accumUpdates.toMap
update(exec)
{code}
in onDriverAccumUpdates.
But postDriverMetricUpdates is called multiple time per query, e.g. from each FileSourceScanExec and BroadcastExchangeExec.

If the update does not really update it in the KV store (depending on liveUpdatePeriodNs), the previously posted metrics are lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org