You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "zhoukang (Jira)" <ji...@apache.org> on 2020/02/02 14:45:00 UTC
[jira] [Updated] (YARN-10060) Historyserver may recover too slow
since JobHistory init too slow when there exist too many job
[ https://issues.apache.org/jira/browse/YARN-10060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhoukang updated YARN-10060:
----------------------------
Attachment: (was: YARN-10060.001.patch)
> Historyserver may recover too slow since JobHistory init too slow when there exist too many job
> -----------------------------------------------------------------------------------------------
>
> Key: YARN-10060
> URL: https://issues.apache.org/jira/browse/YARN-10060
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Reporter: zhoukang
> Assignee: zhoukang
> Priority: Major
> Attachments: YARN-10060-001.patch
>
>
> Like below it cost >7min to listen to the service port
> {code:java}
> 2019-12-24,20:01:37,272 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
> 2019-12-24,20:01:47,354 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing Jobs...
> 2019-12-24,20:08:29,589 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server xxx. Will not attempt to authenticate using SASL (unknown error)
> 2019-12-24,20:08:29,589 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to xxx, initiating session
> 2019-12-24,20:08:29,590 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server xxx, sessionid = 0x66d1a13e596ddc9, negotiated timeout = 5000
> 2019-12-24,20:08:29,593 INFO org.apache.zookeeper.ZooKeeper: Session: 0x66d1a13e596ddc9 closed
> 2019-12-24,20:08:29,593 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
> 2019-12-24,20:08:29,655 INFO org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage Init
> 2019-12-24,20:08:29,681 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
> 2019-12-24,20:08:29,715 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
> 2019-12-24,20:08:29,800 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2019-12-24,20:08:29,943 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2019-12-24,20:08:29,943 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobHistoryServer metrics system started
> 2019-12-24,20:08:29,950 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
> 2019-12-24,20:08:29,951 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
> 2019-12-24,20:08:29,952 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
> 2019-12-24,20:08:30,015 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.jobhistory is not defined
> 2019-12-24,20:08:30,025 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2019-12-24,20:08:30,027 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context jobhistory
> 2019-12-24,20:08:30,027 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
> 2019-12-24,20:08:30,030 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /jobhistory/*
> 2019-12-24,20:08:30,030 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
> 2019-12-24,20:08:30,057 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 20901
> 2019-12-24,20:08:30,939 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /jobhistory started at 20901
> 2019-12-24,20:08:31,177 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
> 2019-12-24,20:08:31,187 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
> 2019-12-24,20:08:31,187 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
> 2019-12-24,20:08:31,189 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.HSClientProtocolPB to the server
> 2019-12-24,20:08:31,216 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryClientService: Instantiated HistoryClientService at xxx
> 2019-12-24,20:08:31,344 INFO org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService: aggregated log deletion started.
> 2019-12-24,20:08:31,690 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=xxx sessionTimeout=5000 watcher=org
> {code}
> {code:java}
> protected void serviceInit(Configuration conf) throws Exception {
> LOG.info("JobHistory Init");
> this.conf = conf;
> this.appID = ApplicationId.newInstance(0, 0);
> this.appAttemptID = RecordFactoryProvider.getRecordFactory(conf)
> .newRecordInstance(ApplicationAttemptId.class);
> moveThreadInterval = conf.getLong(
> JHAdminConfig.MR_HISTORY_MOVE_INTERVAL_MS,
> JHAdminConfig.DEFAULT_MR_HISTORY_MOVE_INTERVAL_MS);
> hsManager = createHistoryFileManager();
> hsManager.init(conf);
> try {
> hsManager.initExisting();
> } catch (IOException e) {
> throw new YarnRuntimeException("Failed to initialize existing directories", e);
> }
> storage = createHistoryStorage();
>
> if (storage instanceof Service) {
> ((Service) storage).init(conf);
> }
> storage.setHistoryFileManager(hsManager);
> super.serviceInit(conf);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org