[jira] [Commented] (YARN-6032) scm cleaner task should rm InMemorySCMStore some cachedResources which does not exists in hdfs fs

["Zhaofei Meng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15782180#comment-15782180 ] 

Zhaofei Meng commented on YARN-6032:
------------------------------------

We should modify use interface in ClientProtocolService to verify fs if or not exist in hdfs.

public UseSharedCacheResourceResponse use(
      UseSharedCacheResourceRequest request) throws YarnException,
      IOException {

    UseSharedCacheResourceResponse response =
        recordFactory.newRecordInstance(UseSharedCacheResourceResponse.class);

    UserGroupInformation callerUGI;
    try {
      callerUGI = UserGroupInformation.getCurrentUser();
    } catch (IOException ie) {
      LOG.info("Error getting UGI ", ie);
      throw RPCUtil.getRemoteException(ie);
    }

    String fileName =
        this.store.addResourceReference(request.getResourceKey(),
            new SharedCacheResourceReference(request.getAppId(),
                callerUGI.getShortUserName()));

    if (fileName != null) {
      if(fs.exists(new Path(fs.getHomeDirectory(),fileName))){
        response
                .setPath(getCacheEntryFilePath(request.getResourceKey(), fileName));
        this.metrics.incCacheHitCount();
      }else{
        this.store.removeResource(request.getResourceKey());
      }
    } else {
      this.metrics.incCacheMissCount();
    }

    return response;
  }

>  scm cleaner task should rm InMemorySCMStore some cachedResources which does not exists in hdfs fs
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6032
>                 URL: https://issues.apache.org/jira/browse/YARN-6032
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.1
>            Reporter: Zhaofei Meng
>             Fix For: 2.7.1
>
>
> If cacheresources exist in scm but not exist in hdfs,the cacheresources  whill not rm from scm until restart scm.So we shoult add check funcion in cleaner task that  rm the cachedResources which does not exists in hdfs fs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org