You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2009/03/19 08:07:50 UTC
[jira] Created: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Recovery duration shown on the jobtracker webpage is inaccurate
---------------------------------------------------------------
Key: HADOOP-5533
URL: https://issues.apache.org/jira/browse/HADOOP-5533
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Amar Kamat
Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5533:
-------------------------------
Attachment: HADOOP-5533-v1.0.patch
Attaching a straight forward fix. Result of test-patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Attachments: HADOOP-5533-v1.0.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700732#action_12700732 ]
Hudson commented on HADOOP-5533:
--------------------------------
Integrated in Hadoop-trunk #811 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/811/])
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch, HADOOP-5533-v1.3.patch, HADOOP-5533-v1.4.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683850#action_12683850 ]
Vinod K V commented on HADOOP-5533:
-----------------------------------
I think it will be useful to also show the start and end times of recovery.
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Attachments: HADOOP-5533-v1.0.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5533:
-------------------------------
Attachment: HADOOP-5533-v1.2.patch
Attaching a patch that provides more information about the recovery duration. Result of test-patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5533:
-------------------------------
Assignee: Amar Kamat
Status: Patch Available (was: Open)
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch, HADOOP-5533-v1.3.patch, HADOOP-5533-v1.4.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5533:
-------------------------------
Attachment: HADOOP-5533-v1.4.patch
Attaching a patch incorporating Amareshwari's super minor comment.
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch, HADOOP-5533-v1.3.patch, HADOOP-5533-v1.4.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-5533:
------------------------------------
Fix Version/s: (was: 0.21.0)
0.20.1
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.20.1
>
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch, HADOOP-5533-v1.3.patch, HADOOP-5533-v1.4.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699533#action_12699533 ]
Amareshwari Sriramadasu commented on HADOOP-5533:
-------------------------------------------------
One Minor nit :
In the code change,
{noformat}
@@ -1313,7 +1320,14 @@
}
}
- recoveryDuration = System.currentTimeMillis() - recoveryStartTime;
+ LOG.info("Took a total of "
+ + StringUtils.formatTime(System.currentTimeMillis()
+ - recoveryStartTime)
+ + " for parsing and recovering all the jobs from history.");
+
+ recoveryDuration = System.currentTimeMillis() - recoveryProcessStartTime;
{noformat}
Two calls to System.currentTimeMillis() can be combined into one.
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch, HADOOP-5533-v1.3.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5533:
-------------------------------
Attachment: HADOOP-5533-v1.3.patch
Attaching a patch that prints the total time spent in filename recovery and job recovery (read + parsing + replay). Result of test-patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch, HADOOP-5533-v1.3.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-5533:
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks Amar!
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch, HADOOP-5533-v1.3.patch, HADOOP-5533-v1.4.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5533) Recovery duration shown on the
jobtracker webpage is inaccurate
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699998#action_12699998 ]
Amareshwari Sriramadasu commented on HADOOP-5533:
-------------------------------------------------
+1
> Recovery duration shown on the jobtracker webpage is inaccurate
> ---------------------------------------------------------------
>
> Key: HADOOP-5533
> URL: https://issues.apache.org/jira/browse/HADOOP-5533
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-5533-v1.0.patch, HADOOP-5533-v1.2.patch, HADOOP-5533-v1.3.patch, HADOOP-5533-v1.4.patch
>
>
> Recovery manager starts the timer once all the job history filenames are recovered. Ideally the time to recover the filename should also be considered as part of the recovery duration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.