You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2018/06/07 13:06:00 UTC

[jira] [Created] (SPARK-24485) Measure and log elapsed time for filesystem operations in HDFSBackedStateStoreProvider

Jungtaek Lim created SPARK-24485:
------------------------------------

             Summary: Measure and log elapsed time for filesystem operations in HDFSBackedStateStoreProvider
                 Key: SPARK-24485
                 URL: https://issues.apache.org/jira/browse/SPARK-24485
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.4.0
            Reporter: Jungtaek Lim


There're couple of operations which communicate with file system (mostly remote HDFS in production) in HDFSBackedStateStoreProvider, which contribute huge part of latency.

It would be better to measure the latency (elapsed time) and log to help debugging when there's unexpected huge latency on state store.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org