You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Prabhu Joseph (Jira)" <ji...@apache.org> on 2022/10/20 11:23:00 UTC
[jira] [Resolved] (YARN-11285) LocalizedResources are leaked and its LocalPath are not cleared
[ https://issues.apache.org/jira/browse/YARN-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prabhu Joseph resolved YARN-11285.
----------------------------------
Resolution: Duplicate
> LocalizedResources are leaked and its LocalPath are not cleared
> ---------------------------------------------------------------
>
> Key: YARN-11285
> URL: https://issues.apache.org/jira/browse/YARN-11285
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.2.1
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
>
> LocalizedResources are leaked and its LocalPath are not cleared from NM Local Directories.
> Each container has separate LocalizedResource object and separate local path like below.
> {code}
> /mnt/yarn/usercache/hive/filecache/6/2552419:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552420:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552421:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552422:
> total 28456
> {code}
> NM logs will be filled with below
> {code}
> 2022-08-07 09:00:00,275 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource (IPC Server handler 4 on 8040): Resource hdfs://hdfscluster/user/svc_di_data_eng/.hiveJars/hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar(->/mnt/yarn/usercache/data_eng_user/filecache/2498262/hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar) transitioned from LOCALIZED to null
> 2022-08-07 09:00:00,340 INFO org.apache.hadoop.yarn.util.ProcfsBasedProcessTree (Container Monitor): SmapBasedCumulativeRssmem (bytes) : 0
> 2022-08-07 09:00:00,386 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource (IPC Server handler 9 on 8040): Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: LOCALIZED at LOCALIZED
> at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource.handle(LocalizedResource.java:198)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.handle(LocalResourcesTrackerImpl.java:186)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.handle(LocalResourcesTrackerImpl.java:58)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.processHeartbeat(ResourceLocalizationService.java:1048)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:722)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:356)
> at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:48)
> at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:63)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org