You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2008/12/12 01:23:44 UTC
[jira] Created: (HADOOP-4845) Shuffle counter issues
Shuffle counter issues
----------------------
Key: HADOOP-4845
URL: https://issues.apache.org/jira/browse/HADOOP-4845
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.20.0
Reporter: Chris Douglas
Fix For: 0.20.0
HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655865#action_12655865 ]
Zheng Shao commented on HADOOP-4845:
------------------------------------
@Yongqiang, is the bytesTransferred per mapper or for all mappers?
@Chris: Am I right in that the counter is the number of bytes that is fed into "reduce task"? For "reduce function" we only have a concept of records not bytes, since "reduce function" accepts objects not byte arrays/streams.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4845) Shuffle counter issues
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-4845:
------------------------------------
Release Note: (was: Change the REDUCE_INPUT_BYTES counter name to REDUCE_SHUFFLE_BYTE, merge counter variables. add a new human-readable value in Task_Counter.properties for REDUCE_SHUFFLE_BYTE)
No release note for "just a bug."
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
he yongqiang updated HADOOP-4845:
---------------------------------
Attachment: (was: Hadoop-4845-3.patch)
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656015#action_12656015 ]
he yongqiang commented on HADOOP-4845:
--------------------------------------
@Runping the compression ratio
long decompressedLength = Long.parseLong(connection.getHeaderField(RAW_MAP_OUTPUT_LENGTH));
long compressedLength = Long.parseLong(connection.getHeaderField(MAP_OUTPUT_LENGTH));
I think for each mapoutput, the comression ratio can be calculated from the variable decompressedLength and compressedLength.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655891#action_12655891 ]
Runping Qi commented on HADOOP-4845:
------------------------------------
We need to have a counter accounting the number of bytes FETCHED for each reduce the end of shuffling.
If the compression was turned on, that should be the number of bytes of the compressed data.
We should also estimate the compression ratio of the fetched compressed data and report it somehow.
We should also report the number of segments and number of bytes written to the local disks at the end of shuffling
At the end of reduce, we should know the number of records and bytes to the reduce.
That number of bytes may be different than the number of fetched bytes.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655912#action_12655912 ]
Runping Qi commented on HADOOP-4845:
------------------------------------
It's ok to record the counters for successful reduces.
If "shuffleInMemory", you should have the numbers of bytes for both the compressed data and uncompressed data.
We should always use the numbers for compressed data.
Estimate on the compression ratio is very useful.
It can be approximated based on the numbers for the ShuffleInMemory data.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656162#action_12656162 ]
Chris Douglas commented on HADOOP-4845:
---------------------------------------
{quote}
We should also estimate the compression ratio of the fetched compressed data and report it somehow.
We should also report the number of segments and number of bytes written to the local disks at the end of shuffling
At the end of reduce, we should know the number of records and bytes to the reduce.
That number of bytes may be different than the number of fetched bytes.
{quote}
These are all excellent ideas, but for this issue, the focus should remain on fixing the counter added in HADOOP-4749.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656167#action_12656167 ]
Arun C Murthy commented on HADOOP-4845:
---------------------------------------
{quote}
We should also estimate the compression ratio of the fetched compressed data and report it somehow.
We should also report the number of segments and number of bytes written to the local disks at the end of shuffling
At the end of reduce, we should know the number of records and bytes to the reduce.
That number of bytes may be different than the number of fetched bytes.
{quote}
+1. Though Counters don't seem like the right mechanism for these metrics - should we be using the metrics framework?
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4845) Shuffle counter issues
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4845:
----------------------------------
Status: Open (was: Patch Available)
I continue to agree with Zheng and Runping: the counter must be consistent. The most useful value will be the bytes transferred during the shuffle (i.e. the compressed bytes).
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845-3.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656171#action_12656171 ]
Zheng Shao commented on HADOOP-4845:
------------------------------------
Given Chris's comment, I agree we can use either SHUFFLE_BYTES or REDUCE_SHUFFLE_BYTES to avoid confusion.
I also agree let's postpone the new counters before we fix the problem in HADOOP-4749.
We still need to figure the story about compressed/uncompressed bytes. At least we need to make the counter consistent no matter compression is turned on or not.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4845) Shuffle counter issues
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4845:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
This passes all the mapred unit tests on my machine.
I just committed this. Thanks Yongqiang
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
he yongqiang updated HADOOP-4845:
---------------------------------
Status: Patch Available (was: Open)
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655824#action_12655824 ]
Chris Douglas commented on HADOOP-4845:
---------------------------------------
It might be more accurate if the counter referred to shuffled bytes instead of "reduce input bytes", since jobs running with a combiner will affect the number of bytes actually fed to the reduce.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655867#action_12655867 ]
he yongqiang commented on HADOOP-4845:
--------------------------------------
The local variable bytesTransferred in ReduceCopier.fetchOutputs record the same as reduceInputBytes. the only difference is that the reduceInputBytes is class member and bytesTransferred is only method-scope, I guess what Arun C Murthy said is to remove the bytesTransferred variable.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
he yongqiang updated HADOOP-4845:
---------------------------------
Attachment: Hadoop-4845.patch
make the size field of MapOutput count compressed bytes size for both shuffleToDisk and shuffleInMemory mapoutputs.
the size field was previously used to record the decompressed byte size for shuffleInMemory mapoutputs and had the same vaule as MapOutput.data.length, but this size field is only used one time to deteremine if the mapoutput's size is zero or not. I think it is ok to use the size filed to record the compressed size of the MapOutput and use the data.length as the size of decommpressed bytes.
For avoiding confusing, i changed the name size to compressedSize.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656920#action_12656920 ]
Chris Douglas commented on HADOOP-4845:
---------------------------------------
+1 This looks good.
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
{noformat}
The eclipse issue is dealt with in HADOOP-4878.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656189#action_12656189 ]
Zheng Shao commented on HADOOP-4845:
------------------------------------
{quote}
The local variable bytesTransferred in ReduceCopier.fetchOutputs record the same as reduceInputBytes. the only difference is that the reduceInputBytes is class member and bytesTransferred is only method-scope, I guess what Arun C Murthy said is to remove the bytesTransferred variable.
{quote}
@Yongqiang: Yep i agree we just need one variable as you did in the first version. Last time I thought that fetchOutputs() will be called multiple times, but apparently it's not. Please merge the 2 variables.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
he yongqiang updated HADOOP-4845:
---------------------------------
Release Note: Change the REDUCE_INPUT_BYTES counter name to REDUCE_SHUFFLE_BYTE, merge counter variables. add a new human-readable value in Task_Counter.properties for REDUCE_SHUFFLE_BYTE
Status: Patch Available (was: Open)
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845-3.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655823#action_12655823 ]
Arun C Murthy commented on HADOOP-4845:
---------------------------------------
It would also be nice to remove a local counter for a same in ReduceCopier.fetchOutputs which tracks the same value as the member reduceInputBytes...
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655871#action_12655871 ]
Arun C Murthy commented on HADOOP-4845:
---------------------------------------
bq. I guess what Arun C Murthy said is to remove the bytesTransferred variable.
Yes. Equivalently, we could just increment the counter in fetchOutputs rather than have a member 'reduceInputBytes'.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655873#action_12655873 ]
Arun C Murthy commented on HADOOP-4845:
---------------------------------------
bq. Equivalently, we could just increment the counter in fetchOutputs rather than have a member 'reduceInputBytes'.
To clarify, that should read as:
Equivalently, we could just increment the Counter (REDUCE_INPUT_BYTES) in fetchOutputs rather than have a member 'reduceInputBytes'.
Maybe that counter should be SHUFFLE_BYTES as Chris suggests...
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4845) Shuffle counter issues
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned HADOOP-4845:
-------------------------------------
Assignee: he yongqiang
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845-3.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655863#action_12655863 ]
he yongqiang commented on HADOOP-4845:
--------------------------------------
@Arun C Murthy, yes, we can remove the variable bytesTransferred.
@Chris Douglas, by shuffled bytes, you mean the number of recourds mappers output to a specific reducer or to all reducers?
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655869#action_12655869 ]
Chris Douglas commented on HADOOP-4845:
---------------------------------------
bq. Am I right in that the counter is the number of bytes that is fed into "reduce task"? For "reduce function" we only have a concept of records not bytes, since "reduce function" accepts objects not byte arrays/streams.
Good point, but an existing counter "map output bytes" doesn't follow these semantics. It counts serialized bytes of records out of the map function, not the bytes output from MapTask. It seems confusing to accept different terminology for the reduce side.
bq. by shuffled bytes, you mean the number of recourds mappers output to a specific reducer or to all reducers?
No, I mean the value this counter is half-tracking, i.e. the number of bytes fetched from all completed maps by the reduce.
As implemented, I'm not sure the counter will be correct when intermediate compression is on and a map output is too large to fetch into memory. When fetched into memory, the counter will be incremented by the size of the decompressed segment. When fetched to disk, it will be incremented by the compressed size.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656530#action_12656530 ]
he yongqiang commented on HADOOP-4845:
--------------------------------------
{quote}
Doesn't this still count a mix of compressed and uncompressed bytes?
{quote}
yes. For "shuffleInMemory", the counter records the size of decompressed data, and for shuffleToDisk, the counter records the size of the compressed data fetched.
If we need more accurate and the exact compressed size for both, it may have to introduce a filed in MapOutput for recording. Currently the patch uses the size field of MapOutput, which counts compressed bytes size for shuffleToDisk and uncompressed bytes size for shuffleInMemory.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845-3.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656506#action_12656506 ]
Chris Douglas commented on HADOOP-4845:
---------------------------------------
Doesn't this still count a mix of compressed and uncompressed bytes?
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845-3.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656225#action_12656225 ]
he yongqiang edited comment on HADOOP-4845 at 12/12/08 5:39 PM:
----------------------------------------------------------------
Change the REDUCE_INPUT_BYTES counter name to REDUCE_SHUFFLE_BYTE, and get rid of local variable byteTransfered and member field reducerInputBytes. Also add a new human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties for REDUCE_SHUFFLE_BYTE.
was (Author: he yongqiang):
Change the REDUCE_INPUT_BYTES counter name to REDUCE_SHUFFLE_BYTE, and get rid of local variable byteTransfered and member field reducerInputBytes.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845-3.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656369#action_12656369 ]
Hadoop QA commented on HADOOP-4845:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12395987/Hadoop-4845-3.patch
against trunk revision 726129.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3741/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3741/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3741/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3741/console
This message is automatically generated.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Assignee: he yongqiang
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845-3.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655902#action_12655902 ]
he yongqiang commented on HADOOP-4845:
--------------------------------------
@Arun
{quote}
we could just increment the Counter (REDUCE_INPUT_BYTES) in fetchOutputs rather than have a member 'reduceInputBytes'.
{quote}
thank you for the point. this would be easy to fix.
maybe we should change the name REDUCE_INPUT_BYTES to REDUCE_SHUFFLE_BYTES ?
@Runping
{quote}
We need to have a counter accounting the number of bytes FETCHED for each reduce the end of shuffling.
If the compression was turned on, that should be the number of bytes of the compressed data.
{quote}
by the number of bytes FETCHED, both sucess and failed? or just the successed copies?
currently, the added Counter REDUCE_INPUT_BYTES records the succssfully fetched from mappers. If "shuffleInMemory", the counter is for the decompressed data, otherwise, the counter is for the compressed data fetched.
{quote}
We should also estimate the compression ratio of the fetched compressed data and report it somehow.
We should also report the number of segments and number of bytes written to the local disks at the end of shuffling
{quote}
Does these are need ?
{quote}
At the end of reduce, we should know the number of records and bytes to the reduce.
{quote}
now we have REDUCE_INPUT_RECORDS for number of records to the reduce.
Maybe we should change the current name REDUCE_INPUT_BYTES to REDUCE_SHUFFLE_BYTES, and use the name REDUCE_INPUT_BYTES for bytes to the reduce?
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4845) Shuffle counter issues
Posted by "he yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
he yongqiang updated HADOOP-4845:
---------------------------------
Attachment: Hadoop-4845-3.patch
Change the REDUCE_INPUT_BYTES counter name to REDUCE_SHUFFLE_BYTE, and get rid of local variable byteTransfered and member field reducerInputBytes.
> Shuffle counter issues
> ----------------------
>
> Key: HADOOP-4845
> URL: https://issues.apache.org/jira/browse/HADOOP-4845
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Chris Douglas
> Fix For: 0.20.0
>
> Attachments: Hadoop-4845-3.patch
>
>
> HADOOP-4749 added a new counter tracking the bytes shuffled into the reduce. It adds an accumulator to ReduceCopier instead of simply incrementing the new counter and did not define a human-readable value in src/mapred/org/apache/hadoop/mapred/Task_Counter.properties.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.