You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2009/08/18 07:47:14 UTC
[jira] Created: (HADOOP-6198) FileSystem filtering should work on
FileStatus rather than Path objects
FileSystem filtering should work on FileStatus rather than Path objects
-----------------------------------------------------------------------
Key: HADOOP-6198
URL: https://issues.apache.org/jira/browse/HADOOP-6198
Project: Hadoop Common
Issue Type: Improvement
Reporter: Chris Douglas
There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6198) FileSystem filtering should work on
FileStatus rather than Path objects
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744611#action_12744611 ]
Doug Cutting commented on HADOOP-6198:
--------------------------------------
> We'll have to keep PathFilter around for at least another release for backwards-compatibility, but I'd like to deprecate it.
+1 We've generally tried to switch FileSystem APIs from using Path to FileStatus whereever possible, deprecating and eventually removing Path-based methods, and this just seems like a case we've missed.
> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
> Key: HADOOP-6198
> URL: https://issues.apache.org/jira/browse/HADOOP-6198
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6198) FileSystem filtering should work on
FileStatus rather than Path objects
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744396#action_12744396 ]
Hong Tang commented on HADOOP-6198:
-----------------------------------
I am not sure why it would be a burden for FS implementors. listStatus would be implemented by FileSystem and FS implementors should only override them when the default implementation is less efficient (and the implementors have the incentive of making them fast).
> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
> Key: HADOOP-6198
> URL: https://issues.apache.org/jira/browse/HADOOP-6198
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6198) FileSystem filtering should work on
FileStatus rather than Path objects
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744454#action_12744454 ]
Chris Douglas commented on HADOOP-6198:
---------------------------------------
Is the proposal to retain two versions of globStatus, one taking a PathFilter and another taking e.g. StatusFilter? This seems like a premature optimization, given that we have no example of a FileSystem for which there is a significant performance difference, or any performance improvement. If one wants to implement a FileSystem that only supports filtering by Path, or that optimizes that particular call, then there are plenty of other ways to effect this without adding a separate method to the core API. Your point on possible performance gain is taken, but it doesn't seem like the right tradeoff, particularly without a concrete use case.
We'll have to keep PathFilter around for at least another release for backwards-compatibility, but I'd like to deprecate it.
> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
> Key: HADOOP-6198
> URL: https://issues.apache.org/jira/browse/HADOOP-6198
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6198) FileSystem filtering should work on
FileStatus rather than Path objects
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744384#action_12744384 ]
Hong Tang commented on HADOOP-6198:
-----------------------------------
I'd argue that we may want to support both interfaces. The fact that the current path-based filtering retrieves a FileStatus object first is an implementation detail and it is conceivable that a different FS implementation may be more efficient to support path-based filtering than file-status based filtering.
> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
> Key: HADOOP-6198
> URL: https://issues.apache.org/jira/browse/HADOOP-6198
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6198) FileSystem filtering should work on
FileStatus rather than Path objects
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744392#action_12744392 ]
Chris Douglas commented on HADOOP-6198:
---------------------------------------
Granted, but supporting two filtering interfaces will likely cause maintenance headaches and be more of a burden to a FS implementor; I'd rather pick an API and not support all possible variants. If a FileSystem is more efficient with path-based filtering, it can still work with FileStatus objects, either populating them lazily, filling them with defaults (what many shims do anyway), or even failing if a user queries unsupported data. Since globStatus returns FileStatus objects, any implementation will need to construct them for the set of accepted Paths, anyway. Given that the API seems biased toward FileStatus objects, I'd rather endure a penalty for the hypothetical FS that doesn't return this information, rather than maintain two separate filtering APIs.
> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
> Key: HADOOP-6198
> URL: https://issues.apache.org/jira/browse/HADOOP-6198
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.