You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/08/23 09:16:20 UTC

[jira] [Commented] (SPARK-9004) Add s3 bytes read/written metrics

    [ https://issues.apache.org/jira/browse/SPARK-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432462#comment-15432462 ] 

Steve Loughran commented on SPARK-9004:
---------------------------------------

If you know the filesystem, you can get summary stats from {{FileSystem.getStatistics()}}; they'd have to be collected across all the executors

These counters are per-JVM, not isolated into individual jobs

> Add s3 bytes read/written metrics
> ---------------------------------
>
>                 Key: SPARK-9004
>                 URL: https://issues.apache.org/jira/browse/SPARK-9004
>             Project: Spark
>          Issue Type: Improvement
>          Components: Input/Output
>            Reporter: Abhishek Modi
>            Priority: Minor
>
> s3 read/write metrics can be pretty useful in finding the total aggregate data processed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org