You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Varun Saxena (JIRA)" <ji...@apache.org> on 2016/06/20 19:40:58 UTC
[jira] [Updated] (MAPREDUCE-6720) Inconsistent values of counters
across tasks and job reported to timeline service.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Varun Saxena updated MAPREDUCE-6720:
------------------------------------
Description:
While testing found below issue. For some of the task counters, we do not have consistent values. This is not the case with every counter though.
Consider the case of counter "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
I found that its value for a flow I ran, was 936018 bytes. For the 3 apps associated with this flow run, the values were 312006 bytes each (which equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further down I found though that for one of the apps, the 4 tasks(2 mappers and 2 reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes each for the 2 mappers.
This means the value reported for the app should be (2 * 156003 + 2* 155918) or 623842 bytes but it is only 312006 bytes which indicates that only counter value of mappers is being picked up.
> Inconsistent values of counters across tasks and job reported to timeline service.
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster
> Affects Versions: YARN-2928
> Reporter: Varun Saxena
>
> While testing found below issue. For some of the task counters, we do not have consistent values. This is not the case with every counter though.
> Consider the case of counter "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps associated with this flow run, the values were 312006 bytes each (which equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further down I found though that for one of the apps, the 4 tasks(2 mappers and 2 reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) or 623842 bytes but it is only 312006 bytes which indicates that only counter value of mappers is being picked up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org