[jira] [Created] (SPARK-2127) Use application specific folders to dump metrics via CsvSink

["Rahul Singhal (JIRA)" <ji...@apache.org>]

Rahul Singhal created SPARK-2127:
------------------------------------

             Summary: Use application specific folders to dump metrics via CsvSink
                 Key: SPARK-2127
                 URL: https://issues.apache.org/jira/browse/SPARK-2127
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 1.0.0
            Reporter: Rahul Singhal
            Priority: Minor


Currently when using the CsvSink, all application's csv metrics are dumped in the root folder (configured via "*.sink.csv.director" in metrics.properties). Also, some files that have common names (e.g. "jvm.PS-MarkSweep.count.csv") are reused. And if one is running the same application multiple times, the metrics get appended to previously existing files.

This makes it harder to parse these files and extract the information that one might be looking for. I suggest that a unique folder is created every time an application is run and use it to dump the metrics from that particular run only. This unique folder could be created similar the one that is currently craeted for logging application events (e.g. "spark-pi-1402484928439").



--
This message was sent by Atlassian JIRA
(v6.2#6252)