You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2008/09/08 07:20:44 UTC
[jira] Created: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Got ArrayOutOfBound exception while analyzing the job history
-------------------------------------------------------------
Key: HADOOP-4112
URL: https://issues.apache.org/jira/browse/HADOOP-4112
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Amar Kamat
HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Status: Open (was: Patch Available)
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Status: Open (was: Patch Available)
We need not have CleanupAttempt as separate TaskAttempt, making TASK_TYPE as CLEANUP will solve the problem
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629444#action_12629444 ]
Hadoop QA commented on HADOOP-4112:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389729/patch-4112.txt
against trunk revision 693404.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3216/console
This message is automatically generated.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630693#action_12630693 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4112:
------------------------------------------------
> All core tests and contrib tests passed on my mahcine except TestHighRAMJobs.
Is there any reason that TestHighRAMJobs fails? It also failed in my machine.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Status: Patch Available (was: Open)
test-patch result on trunk :
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
{noformat}
All core tests and contrib tests passed on my mahcine except TestHighRAMJobs.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Status: Patch Available (was: Open)
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Attachment: patch-4112.txt
Thanks for the review.
bq. logFailed() and logKilled() differ in one string. I assume you have done just to keep the code consistent. Someday we should collapse both the api's into something like logComplete(). There is a lot of duplicate/redundant code.
Yes. I raised HADOOP-4122 for the same.
bq. Why is job-failed log removed? Redundancy?
Yes. It is a redundant log. Moreover, JobInfo.logFailed log closes the log file. Then the cleanupAttempt logs will be missed, because killJob launches a cleanup task and after the completion of cleanup, job is marked Failed.
Patch incorporates other comments
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Status: Open (was: Patch Available)
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629391#action_12629391 ]
Amar Kamat commented on HADOOP-4112:
------------------------------------
Few comments
_JobHistory :_
1) The comment for {{CleanupAttempt}} class needs to be changed.
2) {{logFailed()}} and {{logKilled()}} differ in one string. I assume you have done just to keep the code consistent. Someday we should collapse both the api's into something like {{logComplete()}}. There is a lot of duplicate/redundant code.
_JobInProgress :_
1) Why is job-failed log removed? Redundancy?
2) I think there are some diffs because of formatting changes
Note that I have just checked {{JobHistory.java}} and {{JobInProgress.java}}.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630938#action_12630938 ]
Amareshwari Sriramadasu commented on HADOOP-4112:
-------------------------------------------------
bq. Is there any reason that TestHighRAMJobs fails? It also failed in my machine.
Due to HADOOP-4129
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Status: Patch Available (was: Open)
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Fix Version/s: 0.19.0
Affects Version/s: 0.19.0
Status: Patch Available (was: Open)
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629756#action_12629756 ]
Hadoop QA commented on HADOOP-4112:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389804/patch-4112.txt
against trunk revision 693705.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3229/console
This message is automatically generated.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630093#action_12630093 ]
Amar Kamat commented on HADOOP-4112:
------------------------------------
I think the refactoring of JobHistory apis should be done as a part of HADOOP-4122.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Attachment: patch-4112.txt
The patch removes refactoring done in earlier patch. A boolean parameter, isCleanup, is added to the methods in MapAttempt and ReduceAttempt and old methods are deprecated.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu reassigned HADOOP-4112:
-----------------------------------------------
Assignee: Amareshwari Sriramadasu
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Attachment: patch-4112.txt
Here is patch doing the following:
1. Adds CleanupAttempt to JobHistory. The cleanup tasks are logged in history as CleanupAttempts. And the TASK_TYPE is CLEANUP. This change also fixes ArrayOutOfBoundException in analyzeJob link.
2. Updates JobTasksParseListener about CleanupAttempt
3. JobInProgress is modified to log the cleanup task as CleanupAttempt. Also removes an extra JobFailed log from history.
4. Updates jobhistory.jsp and HistoryViewer.java to show cleanup task.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4112:
--------------------------------------------
Attachment: patch-4112.txt
Here is patch removing CleanupAttempt introduced in earlier patch. The cleanup tasks are logged with CLEANUP as TASK_TYPE. The patch adds the following methods to TaskAttempt
{noformat}
public static void logStarted(TaskAttemptID taskAttemptId, long startTime, String hostName, boolean isMap, boolean isCleanup)
public static void logFinished(TaskAttemptID taskAttemptId, long finishTime, String hostName, boolean isMap, boolean isCleanup)
public static void logKilled(TaskAttemptID taskAttemptId, long timestamp, String hostName, String error, boolean isFailure, boolean isMap, boolean isCleanup)
{noformat}
The methods in MapAttempt and ReduceAttempt are modified to use the new methods. And JobInProgress's updating history is modified to use the new methods.
There is no change for the JobTasksParseListener now. And there are no changes in jobhistory.jsp and HistoryViewer from the earlier patch.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-4112:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amareshwari!
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629712#action_12629712 ]
Amareshwari Sriramadasu commented on HADOOP-4112:
-------------------------------------------------
test-patch result on the trunk:
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec]
{noformat}
All unit tests except TestHighRAMJobs passed on my machine. TestHighRAMJobs failure is due to HADOOP-4129
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631143#action_12631143 ]
Hadoop QA commented on HADOOP-4112:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389987/patch-4112.txt
against trunk revision 695604.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3262/console
This message is automatically generated.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt, patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629410#action_12629410 ]
Hadoop QA commented on HADOOP-4112:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389725/patch-4112.txt
against trunk revision 693360.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3214/console
This message is automatically generated.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4112) Got ArrayOutOfBound exception while
analyzing the job history
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629436#action_12629436 ]
Amar Kamat commented on HADOOP-4112:
------------------------------------
Changes made to {{JobHistory}} and {{JobInProgress}} look fine to me.
> Got ArrayOutOfBound exception while analyzing the job history
> -------------------------------------------------------------
>
> Key: HADOOP-4112
> URL: https://issues.apache.org/jira/browse/HADOOP-4112
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4112.txt, patch-4112.txt
>
>
> HADOOP-3150 introduced 2 new type of tasks called the cleanup tasks. These are logged to history either as map tasks or reduce tasks. The number of maps/reducers is also logged to history. Since the number of maps will be less than the total number of map tasks logged to history (actual num maps + cleanup tasks), I think thats the reason for this exception. The important question is to investigate the effect of HADOOP-3150 on job history and code related to it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.