[jira] [Created] (MAPREDUCE-5453) File System Counters can NOT be updated in streaming application

["Danica Zhang (JIRA)" <ji...@apache.org>]

Danica Zhang created MAPREDUCE-5453:
---------------------------------------

             Summary: File System Counters can NOT be updated in streaming application
                 Key: MAPREDUCE-5453
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5453
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Danica Zhang


It seems that FILE_BYTES_READ/FILE_BYTES_WRITTEN can NOT be updated in streaming application by:

reporter:counter:<group>,<counter>,<amount> 

The syntax I tried is:
System.err.println("reporter:counter:"+ "FileSystemCounters" + ","
            					+ "FILE_BYTES_READ" + ","
            					+ number);

Since File System counters are added later into the framework by ticket
https://issues.apache.org/jira/browse/HADOOP-3001
 
and that's why it is NOT properly handled by streaming.jar.

The implementation of File System counter is a bit different, and it keeps track of the real information in FileSystem.Statistics object. When the task finished, it framework would read the info from 'FileSystem.Statistics' and set the File System counters accordingly.

Therefore, the regular way to update counters -- ' findCounter() and Counter.incrment() ' would NOT work for the File System counters as they would be overwritten by the info in 'FileSystem.Statistics ' eventually.
However, it seems that the 'reporter:counter:' scheme in streaming.jar is implemented by the regular way, and maybe that's why it is not working.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira