You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2011/07/08 19:15:16 UTC
[jira] [Created] (MAPREDUCE-2664) MR 279: Implement JobCounters for
MRv2
MR 279: Implement JobCounters for MRv2
--------------------------------------
Key: MAPREDUCE-2664
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2664
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Fix For: 0.23.0
MRv2 is currently not setting any Job Counters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2664) MR 279: Implement JobCounters for
MRv2 + Fix for Map Data Locality
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-2664:
--------------------------------------
Attachment: MR2664.patch
Sending job counter update events via the eventHandler as and when they happen so that the counters can be queried while the job is running and to avoid any kind of sync issues in JobImpl.
Converting the split location to an IP address (Containers currently use IPs). This should enable node data locality.
> MR 279: Implement JobCounters for MRv2 + Fix for Map Data Locality
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2664
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2664
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.0
>
> Attachments: MR2664.patch
>
>
> MRv2 is currently not setting any Job Counters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2664) MR 279: Implement JobCounters for
MRv2 + Fix for Map Data Locality
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-2664:
--------------------------------------
Summary: MR 279: Implement JobCounters for MRv2 + Fix for Map Data Locality (was: MR 279: Implement JobCounters for MRv2)
> MR 279: Implement JobCounters for MRv2 + Fix for Map Data Locality
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2664
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2664
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.0
>
>
> MRv2 is currently not setting any Job Counters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2664) MR 279: Implement JobCounters
for MRv2 + Fix for Map Data Locality
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal resolved MAPREDUCE-2664.
---------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
I committed this MR-279 branch. Thanks Siddharth!
> MR 279: Implement JobCounters for MRv2 + Fix for Map Data Locality
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2664
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2664
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.0
>
> Attachments: MR2664.patch, MR2664_2.patch
>
>
> MRv2 is currently not setting any Job Counters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2664) MR 279: Implement JobCounters for
MRv2 + Fix for Map Data Locality
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-2664:
--------------------------------------
Attachment: MR2664_2.patch
Removed some TODOs
Added one to resolve host names in a better way.
Changed testSleepJob to not fail if the app-jar is missing (i.e. skip the test).
> MR 279: Implement JobCounters for MRv2 + Fix for Map Data Locality
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2664
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2664
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.0
>
> Attachments: MR2664.patch, MR2664_2.patch
>
>
> MRv2 is currently not setting any Job Counters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2664) MR 279: Implement JobCounters
for MRv2 + Fix for Map Data Locality
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070641#comment-13070641 ]
Siddharth Seth commented on MAPREDUCE-2664:
-------------------------------------------
Sharad, thanks for taking a look. The test cases do pass for me (patch applied to revision e3d9a2bcbcab817043b1c4c41efb7036ce00904f).
Could you please post details of which tests are failing along with the trace and env.
The Data locality fix in this patch is a temporary patch to allow testing. A more efficient patch incorporating MR-2489 will be required later.
> MR 279: Implement JobCounters for MRv2 + Fix for Map Data Locality
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2664
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2664
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.0
>
> Attachments: MR2664.patch
>
>
> MRv2 is currently not setting any Job Counters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2664) MR 279: Implement JobCounters
for MRv2 + Fix for Map Data Locality
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070391#comment-13070391 ]
Sharad Agarwal commented on MAPREDUCE-2664:
-------------------------------------------
The patch looks fine. minor - some TODOs and FIXME will go away from JobImpl after this patch.
@Sid - Do the testcases pass for you? Some testcases are failing for me with the patch.
> MR 279: Implement JobCounters for MRv2 + Fix for Map Data Locality
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2664
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2664
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 0.23.0
>
> Attachments: MR2664.patch
>
>
> MRv2 is currently not setting any Job Counters.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira