You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Wilfred Spiegelenburg (JIRA)" <ji...@apache.org> on 2016/06/20 05:51:05 UTC
[jira] [Commented] (MAPREDUCE-6718) add progress log to JHS during
startup
[ https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339036#comment-15339036 ]
Wilfred Spiegelenburg commented on MAPREDUCE-6718:
--------------------------------------------------
We still should have a progress report anything more than a couple of seconds could already cause a customer to say the server has not started.
What would happen if I have a cache in the history server setup for 150K jobs or more to be kept? Limiting the cache is OK and we already do that but customers increase the cache size because anything not in the cache can not be accessed. If they run 20K jobs a day and want 7 days to be accessible then the cache must be 150K.
Purge of the history is set to 7 days by default which could easily do this.
Not being able to find a history that is not in the cache is another issue which is far more difficult to fix.
> add progress log to JHS during startup
> --------------------------------------
>
> Key: MAPREDUCE-6718
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Priority: Minor
> Labels: supportability
>
> When the JHS starts up, it initializes the internal caches and storage via the HistoryFileManager. If we have a large number of existing finished jobs then we could spent minutes in this startup phase without logging progress:
> 2016-03-14 10:56:01,444 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://hadoopcdh.itnas01.ieee.org:8020]
> 2016-03-14 10:56:11,455 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing Jobs...
> 2016-03-14 12:01:36,926 INFO org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage Init
> This makes it really difficult to assess if things are working correctly (it looks hung). We can add logs to notify users of progress.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org