You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Mike Sukmanowsky (JIRA)" <ji...@apache.org> on 2016/04/01 19:56:25 UTC

[jira] [Created] (SPARK-14332) Support Hadoop Input/OutputFormat Counters as Metrics

Mike Sukmanowsky created SPARK-14332:
----------------------------------------

             Summary: Support Hadoop Input/OutputFormat Counters as Metrics
                 Key: SPARK-14332
                 URL: https://issues.apache.org/jira/browse/SPARK-14332
             Project: Spark
          Issue Type: Improvement
          Components: Input/Output
            Reporter: Mike Sukmanowsky
            Priority: Minor


Many custom Hadoop Input/Output Formats register metrics via Hadoop's Counter infrastructure. For example, the Input/Output formats offered by es-hadoop register quite a few handy metrics https://www.elastic.co/guide/en/elasticsearch/hadoop/current/metrics.html#metrics.

It'd be very helpful for something like the `SparkHadoopWriter` class (not sure of the equivalent on the read side) to automatically discover and register all Hadoop counters as Spark metric sources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org