You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Suresh Srinivas (JIRA)" <ji...@apache.org> on 2010/08/05 01:31:16 UTC
[jira] Created: (HADOOP-6900) FileSystem#listLocatedStatus API
needs changes to indicate complete set of error conditions
FileSystem#listLocatedStatus API needs changes to indicate complete set of error conditions
-------------------------------------------------------------------------------------------
Key: HADOOP-6900
URL: https://issues.apache.org/jira/browse/HADOOP-6900
Project: Hadoop Common
Issue Type: Bug
Reporter: Suresh Srinivas
HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Through the iterator application does not get all the error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896701#action_12896701 ]
Hairong Kuang commented on HADOOP-6900:
---------------------------------------
> Additionally I feel when a directory gets deleted in the middle of iteration, we should ignore it, or add provision to the iterator to ignore it.
Unfortunately not allowing concurrent modification on the input directories is the semantics that map/reduce has it now. I do not want to introduce an incompatible change for now.
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Attachments: listFilesIO.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897142#action_12897142 ]
Hudson commented on HADOOP-6900:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #356 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/356/])
HADOOP-6900. Make the iterator returned by FileSystem#listLocatedStatus to throw IOException rather than RuntimeException when there is an IO error fetching the next file. Contributed by Hairong Kuang.
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897092#action_12897092 ]
Sanjay Radia commented on HADOOP-6900:
--------------------------------------
>I do not want to introduce an incompatible change for now.
I don't understand the compatibility argument since this is new API.
We could add a parameter to indicate whether or not continue if a change in the sub-tree has occurred.
I can live with that as long as folks not interpret it to mean that the list returned is atomic - if a subdir's content has been
returned then the iterator will not detect it.
Hairong's patch can go though as Suresh has suggested and we can file another one to finalize this API.
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-6900:
----------------------------------
Attachment: listFilesIO1.patch
This patch adds more comments and changes some FileContext tests to use the new listStatus interface.
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang resolved HADOOP-6900.
-----------------------------------
Hadoop Flags: [Reviewed]
Fix Version/s: 0.22.0
Resolution: Fixed
I've just committed this!
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897088#action_12897088 ]
Suresh Srinivas commented on HADOOP-6900:
-----------------------------------------
+ 1 for the patch.
bq. Unfortunately not allowing concurrent modification on the input directories is the semantics that map/reduce has it now. I do not want to introduce an incompatible change for now.
Given that this is a new API, I am not sure handling concurrent modification and ignoring deleted directories is a real backward compatibility issue for mapreduce. Mapreduce folks can comment about what is more suitable. We can address this issue in a separate jira though.
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-6900:
------------------------------------
Summary: FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions (was: FileSystem#listLocatedStatus API needs changes to indicate complete set of error conditions)
Description: HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions. (was: HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Through the iterator application does not get all the error conditions.)
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-6900:
----------------------------------
Attachment: listFilesIO.patch
A patch for review.
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Attachments: listFilesIO.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895852#action_12895852 ]
Hairong Kuang commented on HADOOP-6900:
---------------------------------------
Suresh, good idea! I appreciate your thoughts on this.
I will create a new Iterator interface that throws IOException on next() and hasNext().
I will make listFiles, listLocatedStatus, AND listStatus to return this new iterator iterface. How does this sound to you?
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897099#action_12897099 ]
Hairong Kuang commented on HADOOP-6900:
---------------------------------------
> I don't understand the compatibility argument since this is new API.
Even though this is a new API, but mapreduce already has some logic in their code traversing the input directories and getting file status/block locations. Existing code does not allow any concurrent modification to input directories.
I agree that allowing concurrent modification could be an option, but lets do it in a different jira.
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus API
needs changes to indicate complete set of error conditions
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895470#action_12895470 ]
Suresh Srinivas commented on HADOOP-6900:
-----------------------------------------
Iterator#hasNext() has to indicate the following error conditions:
# next element exists - returns true
# next element does not exist - returns false
# could not get status for the next element, because a directory that was available before iteration started does not exist any more.
# could not talk to the remote server to get status.
# remote server throws IOException due to an error while getting the status.
For the last 3 conditions, currently RuntimeException is thrown. Throwing RTE forces application to catch it, to handle the error condition. This results in application catching other RTE as well. Worse still, an application that does not catch RTE might exit.
The problem is, Java Iterator interface is not suitable for implementation that use RPC underneath. We should have a different interface, which could indicate error conditions more clearly.
Additionally I feel when a directory gets deleted in the middle of iteration, we should ignore it, or add provision to the iterator to ignore it.
> FileSystem#listLocatedStatus API needs changes to indicate complete set of error conditions
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Through the iterator application does not get all the error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang reassigned HADOOP-6900:
-------------------------------------
Assignee: Hairong Kuang
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897437#action_12897437 ]
Hairong Kuang commented on HADOOP-6900:
---------------------------------------
Yes, I can not change HDFS in common but have to do it in HDFS-202. This is side effect of having fs in common. :(
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897132#action_12897132 ]
Hairong Kuang commented on HADOOP-6900:
---------------------------------------
Ant test-patch result:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 12 new or modified tests.
[exec]
[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
[exec]
[exec] +1 javac.
[exec] The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
Javadoc warnings are not related to this patch.
Ant test:
BUILD SUCCESSFUL
Total time: 11 minutes 2 seconds
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
not throw generic RuntimeException to indicate error conditions
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897389#action_12897389 ]
Konstantin Shvachko commented on HADOOP-6900:
---------------------------------------------
Hairong. You changed the prototype of {{AbstractFileSystem.listStatusIterator(Path)}} but did not change its overloaded version {{Hdfs.listStatusIterator(Path)}}. Could you please fix it, as current build is broken as it is.
> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6900
> URL: https://issues.apache.org/jira/browse/HADOOP-6900
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.