You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2022/10/04 13:52:00 UTC
[jira] [Updated] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead should close FS
[ https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated YARN-11039:
----------------------------------
Affects Version/s: 3.3.4
> LogAggregationFileControllerFactory::getFileControllerForRead should close FS
> ------------------------------------------------------------------------------
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
> Affects Versions: 3.3.4
> Reporter: Rajesh Balamohan
> Priority: Major
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
> try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator<FileStatus> getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org