You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2012/10/04 23:33:47 UTC
[jira] [Created] (MAPREDUCE-4705) Historyserver links expire before
the history data does
Jason Lowe created MAPREDUCE-4705:
-------------------------------------
Summary: Historyserver links expire before the history data does
Key: MAPREDUCE-4705
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3
Reporter: Jason Lowe
Priority: Critical
The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472097#comment-13472097 ]
Siddharth Seth commented on MAPREDUCE-4705:
-------------------------------------------
+1. Committing this.
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471806#comment-13471806 ]
Jason Lowe commented on MAPREDUCE-4705:
---------------------------------------
The problem occurs because HistoryFileManager.scanOldDirsForJob is using a job's serial number directly to lookup the serial number index (which would require an entry in the map per job) rather than the result of JobHistoryUtils.serialNumberDirectoryComponent which carves off the lower digits of the serial number, effectively grouping them into buckets of 1000. I'll post a patch shortly.
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472122#comment-13472122 ]
Hudson commented on MAPREDUCE-4705:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2856/])
MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)
Result = FAILURE
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472389#comment-13472389 ]
Hudson commented on MAPREDUCE-4705:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1221 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/])
MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4705) Historyserver links expire before
the history data does
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-4705:
----------------------------------
Attachment: MAPREDUCE-4705.patch
Patch to fix scanOldDirsForJob to use a proper key for the serial number index lookup.
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472312#comment-13472312 ]
Hudson commented on MAPREDUCE-4705:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #399 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/399/])
merge MAPREDUCE-4705 from trunk. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395852)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395852
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469855#comment-13469855 ]
Siddharth Seth commented on MAPREDUCE-4705:
-------------------------------------------
The history server is supposed to be able to serve out older jobs by scanning through job directories ({{HistoryFileManager.scanOldDirsForJob}}) if a job is not found in the jobListcache. The number of directories it is aware of is controlled by "mapreduce.jobhistory.datestring.cache.size" - which defaults to a reasonably high value.
Maybe this is broken at the moment ?, or we're trying to access specific jobs using a list API of some kind.
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Priority: Critical
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472103#comment-13472103 ]
Hudson commented on MAPREDUCE-4705:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2895 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2895/])
MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4705) Historyserver links expire before
the history data does
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-4705:
----------------------------------
Target Version/s: 2.0.3-alpha, 0.23.5
Status: Patch Available (was: Open)
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472332#comment-13472332 ]
Hudson commented on MAPREDUCE-4705:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1190 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/])
MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash reassigned MAPREDUCE-4705:
---------------------------------------
Assignee: Jason Lowe (was: Ravi Prakash)
Jason got to it, before I could
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471871#comment-13471871 ]
Hadoop QA commented on MAPREDUCE-4705:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548297/MAPREDUCE-4705.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2916//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2916//console
This message is automatically generated.
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4705) Historyserver links expire before
the history data does
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-4705:
--------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.5
2.0.3-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk, branch-2 and branch-0.23. Thanks Jason
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472104#comment-13472104 ]
Hudson commented on MAPREDUCE-4705:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2833 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2833/])
MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Fix For: 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4705.patch
>
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4705) Historyserver links expire
before the history data does
Posted by "Ravi Prakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash reassigned MAPREDUCE-4705:
---------------------------------------
Assignee: Ravi Prakash
> Historyserver links expire before the history data does
> -------------------------------------------------------
>
> Key: MAPREDUCE-4705
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Ravi Prakash
> Priority: Critical
>
> The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.
> We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}}, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira