You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinayakumar B (JIRA)" <ji...@apache.org> on 2016/12/12 09:49:58 UTC
[jira] [Comment Edited] (MAPREDUCE-6486) File{Input|Output}Format
counters not updated when using viewfs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741494#comment-15741494 ]
Vinayakumar B edited comment on MAPREDUCE-6486 at 12/12/16 9:49 AM:
--------------------------------------------------------------------
Since ViewFS is just a wrapper over real filesystems, it would be better just ignore these stats and dont show it in final counters list.
{{FileSystem.getAllStatistics()}} can filter stats for all filesystems with child filesystems, or just viewfs.
What do you think [~cnauroth]?
was (Author: vinayrpet):
Since ViewFS is just a wrapper over real filesystems, it would be better just ignore these stats and dont show it in final counters list.
Since these counters are already updated for resolved filesystems.
{{FileSystem.getAllStatistics()}} can filter stats for all filesystems with child filesystems, or just viewfs.
What do you think [~cnauroth]?
> File{Input|Output}Format counters not updated when using viewfs
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-6486
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6486
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
> Reporter: Laurent Goujon
>
> MapReduce tasks rely on filesystem stats to update FileInputFormatCounter and FileOutputFormatCounter bytes counters.
> But if the filesystem is viewfs, stats are not updated and counters are zero. However the resolved filesystem for the input and output paths have their stats updated, and should be used by the framework (which could be done by resolving the path first).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org