You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jakob Homan (JIRA)" <ji...@apache.org> on 2009/08/11 21:36:14 UTC
[jira] Created: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
DistributedFileSystem::listStatus incorrectly returns null for empty result sets
--------------------------------------------------------------------------------
Key: HDFS-538
URL: https://issues.apache.org/jira/browse/HDFS-538
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Jakob Homan
Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan reassigned HDFS-538:
--------------------------------
Assignee: Jakob Homan
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HDFS-538:
-----------------------------
Release Note: Updated DistributedFileSystem::listStatus() method to throw a FileNotFoundException rather than return null in the case that the provided path does not exist. Updated code that relied on checking for null to catch FileNotFoundException. Client code that relied on checking for null should be updated to support the new semantics. (was: Solidify FileSystem contract for listStatus method when the provided path does not exist. The method is now guaranteed to throw a FileNotFoundException, whereas some FileSystem implementations had previously returned null in this case. Existing code that relies on checking for null should be updated to catch or propagate a FileNotFoundException.)
Adding correct release note.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.21.0
>
> Attachments: HDFS-538.patch
>
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749138#action_12749138 ]
Hudson commented on HDFS-538:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #65 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/])
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.21.0
>
> Attachments: HDFS-538.patch
>
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743362#action_12743362 ]
Jakob Homan commented on HDFS-538:
----------------------------------
bq. How do we distinguish between an empty directory and a non-existent file (or directory)?
The similar method {{getFileStatus()}} returns a FileNotFoundException, which seems the most reasonable and expected behavior.
The API documentation is exceedingly poor for this method: what happens if the passed-in path is not a directory?
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-538:
-----------------------------
Release Note: FileSystem.listStatus() previously returned null for empty or nonexistent directories. It has been changed to throw FileNotFoundException if the directory does not exist and to return an empty array if the directory is empty.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.21.0
>
> Attachments: HDFS-538.patch
>
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743365#action_12743365 ]
Jakob Homan commented on HDFS-538:
----------------------------------
Whoops. Looks like Konstantin and I were responding at the same time. As a more general point, it seems quite preferable for methods not to return null, but either return the object they promised to or throw an exception.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HDFS-538:
---------------------------------
Release Note: (was: Updated DistributedFileSystem::listStatus() method to throw a FileNotFoundException rather than return null in the case that the provided path does not exist. Updated code that relied on checking for null to catch FileNotFoundException. Client code that relied on checking for null should be updated to support the new semantics.)
Editorial pass over all release notes prior to publication of 0.21.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.21.0
>
> Attachments: HDFS-538.patch
>
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743140#action_12743140 ]
Tom White commented on HDFS-538:
--------------------------------
How do we distinguish between an empty directory and a non-existent file (or directory)? In the current implementation, an empty directory returns an empty array, and a non-existent file returns null. I think we should keep this distinction and change the documentation of FileSystem, and the implementation of LocalFileSystem to conform.
bq. it seems to me to be important to fix the semantics sooner than later.
I agree. This should be made clear in the release notes.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743358#action_12743358 ]
Konstantin Shvachko commented on HDFS-538:
------------------------------------------
> How do we distinguish between an empty directory and a non-existent file (or directory)?
Good point. Shouldn't we be throwing FileNotFoundException as other methods? We do not throw right now, but it would makes listStatus consistent with other methods, imo.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743488#action_12743488 ]
Sanjay Radia commented on HDFS-538:
-----------------------------------
Kons says:
>Good point. Shouldn't we be throwing FileNotFoundException as other methods?
Agreed. it should throw the exception if file is not found, instead of returning a null.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HDFS-538:
-----------------------------
Release Note: FileSystem.listStatus() previously returned null for empty or nonexistent directories; will now return empty array for empty directories and throw FileNotFoundException for non-existent directory. Client code should be updated for new semantics. (was: FileSystem.listStatus() previously returned null for empty or nonexistent directories. It has been changed to throw FileNotFoundException if the directory does not exist and to return an empty array if the directory is empty.)
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.21.0
>
> Attachments: HDFS-538.patch
>
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870092#action_12870092 ]
Jakob Homan commented on HDFS-538:
----------------------------------
@Todd - why the change in release note? The previous version better covered the change and provided a clue to clients at the change they need to make.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.21.0
>
> Attachments: HDFS-538.patch
>
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HDFS-538:
-----------------------------
Release Note: Solidify FileSystem contract for listStatus method when the provided path does not exist. The method is now guaranteed to throw a FileNotFoundException, whereas some FileSystem implementations had previously returned null in this case. Existing code that relies on checking for null should be updated to catch or propagate a FileNotFoundException.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.21.0
>
> Attachments: HDFS-538.patch
>
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743058#action_12743058 ]
Jakob Homan commented on HDFS-538:
----------------------------------
Tom-
I've actually run into an issue with this. Patch is done, but when running it through unit tests, it fails {{TestHDFSFileSystemContract}}, which includes a test called {{testListStatusReturnsNullForNonExistentFile()}}, which was added back in HADOOP-930. This test is evaluating an element of the contract that seems incorrect, particularly given the FileSystem API documention: "{{@return the statuses of the files/directories in the given patch}}," which does not mention returning null. It would be much better in this situation to just return a zero-length array. Otherwise, every call to listStatus has to include an extra null check, rather than just iterating over the array (which still succeeds with a zero-length array) or actually checking for zero files via the length of the array equaling zero.
This is borne out by listStatus' use in HDFS codebase, where there is only one check for null; the others assume an array will be returned and could potentially NPE at some point.
The rub is that the S3 FileSystem implementations do return nulls in this situation, so if we fix the contract, these will have an incompatible change introduced. I'm not sure how wide-felt that would be, but it seems to me to be important to fix the semantics sooner than later. The other option would be to change the API specification and continue with non-intuitive behavior, as well as changing HDFS to support it. Thoughts?
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus
incorrectly returns null for empty result sets
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870097#action_12870097 ]
Todd Lipcon commented on HDFS-538:
----------------------------------
Ah, I didn't know there was a prior release note - looking at the history, Rob seems to have deleted the old one on 10/09 without putting in a new one. The old release note also did not explain that listStatus now returns empty arrays for empty directories instead of null.
> DistributedFileSystem::listStatus incorrectly returns null for empty result sets
> --------------------------------------------------------------------------------
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.21.0
>
> Attachments: HDFS-538.patch
>
>
> Currently the listStatus method returns null if no files match the request. This differs from the Checksum/LocalFileSystem implementation, which returns an empty array, and the nontvery-explict prescription of the FileSystem interface: "{{@return the statuses of the files/directories in the given patch}}" It's better to return an empty collection than have to add extra null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.