[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions

["Jonathan Eagles (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16411586#comment-16411586 ] 

Jonathan Eagles commented on HADOOP-14600:
------------------------------------------

[~myapachejira] Was taking a look at this exact bug I found this week while running 2.8. Filed HADOOP-15337 to fix, but this is likely the same exactly.  It would be great to have this feature fix in 2.8, but I see this JIRA took a vastly different approach that seems more work to backport. Did you compare the posix file system approach implementation approach for speeds and support? Without realizing you had already fixed this issue, I put a patch up in HADOOP-15337 that shows the alternative implementation approach. Sorry again for being so late to the conversation, not intending to step on toes. 

> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-14600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14600
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.7.3
>         Environment: file:// in a dir with many files
>            Reporter: Steve Loughran
>            Assignee: Ping Liu
>            Priority: Major
>             Fix For: 3.1.0
>
>         Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java, command_line_test_result__linux.txt, command_line_test_result__windows.txt
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws against the local FS, because {{FileStatus.getPemissions}} call forces  {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI values.
> That is: for every other FS, what's a field lookup or even a no-op, on the local FS it's a process exec/spawn, with all the costs. This gets expensive if you have many files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org