[jira] [Commented] (YARN-5289) Use checksum in localization requests

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346075#comment-15346075 ] 

Steve Loughran commented on YARN-5289:
--------------------------------------

Note that this addresses some situations with filesystems and the localizer, where it's reporting FS timing differences even when the files are unchanged. Checking purely on timestamp , while simple, is a pretty brittle form of validation.

BTW, looking at the code of getFileChecksum in HDFS, it can throw {{UnsupportedFileSystemException}} if a symlink points to an FS without the feature, whereas the API says "return null". YARN code had better check for both when building requests; when localizing, skip checks if checksum == 0 

> Use checksum in localization requests
> -------------------------------------
>
>                 Key: YARN-5289
>                 URL: https://issues.apache.org/jira/browse/YARN-5289
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Yesha Vora
>
> Currently, timestamps is being used in  localization requests. Because using timestamps is always pretty brittle. ( we notice a clock jump in the VM.)
> For the filesystem supporting getFileChecksum(Path), the checksum can be used instead of timestamps in localisation requests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org