[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.

["zhihai xu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585365#comment-14585365 ] 

zhihai xu commented on YARN-3549:
---------------------------------

thanks [~vinodkv] and [~cnauroth]! it makes sense to do the optimization at {{loadPermissionInfo}}, so all users can benefit from it.
I did some profiling and I find It can speed up 3-4x using NativeIO.POSIX. {{checkLocalDir}} takes about 15ms with shell-based implementation currently, it only takes about 4 ms using NativeIO.POSIX. I will make this issue duplicate as HADOOP-11935.

> use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3549
>                 URL: https://issues.apache.org/jira/browse/YARN-3549
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>
> Use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
> As discussed in YARN-3491, shell-based implementation getPermission runs shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow).
> We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in RawLocalFileSystem to get rid of shell-based implementation for FileStatus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)