You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2010/02/17 23:57:27 UTC
[jira] Created: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
FileInputFormat to support multi-level/recursive directory listing
------------------------------------------------------------------
Key: MAPREDUCE-1501
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Zheng Shao
As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao reassigned MAPREDUCE-1501:
-------------------------------------
Assignee: Zheng Shao
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated MAPREDUCE-1501:
----------------------------------
Status: Open (was: Patch Available)
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Ian Soboroff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836858#action_12836858 ]
Ian Soboroff commented on MAPREDUCE-1501:
-----------------------------------------
I am one of the authors of the emails cited in the description. In my implementation (which I did not submit as a JIRA), I have path filters to make sure we don't add . and .. and other hidden directories. I haven't properly thought about this issue in months... does this patch need this kind of check?
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated MAPREDUCE-1501:
----------------------------------
Attachment: MAPREDUCE-1501.1.trunk.patch
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837030#action_12837030 ]
dhruba borthakur commented on MAPREDUCE-1501:
---------------------------------------------
That should be good enough, unless Ian has some other ideas.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-1501:
----------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks Zheng.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836948#action_12836948 ]
dhruba borthakur commented on MAPREDUCE-1501:
---------------------------------------------
I think Ian mentioned that you can enhance this feature by allowing the user to register a set of PathFilters. That will allow the job to process only a selected subset of the subdirectories.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated MAPREDUCE-1501:
----------------------------------
Status: Patch Available (was: Open)
There are 2 test failures but I don't think they are related. Resubmitting patch to get it tested again.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841042#action_12841042 ]
Hudson commented on MAPREDUCE-1501:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #257 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/257/])
. FileInputFormat supports multi-level, recursive
directory listing. (Zheng Shao via dhruba)
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841658#action_12841658 ]
Chris Douglas commented on MAPREDUCE-1501:
------------------------------------------
{noformat}
+import com.sun.org.apache.commons.logging.Log;
+import com.sun.org.apache.commons.logging.LogFactory;
{noformat}
Should these imports be {{org.apache.hadoop.commons.logging}}, not {{com.sun...}} ?
Is there a reason this feature was only added to a deprecated class, instead of the {{FileInputFormat}} in the {{mapreduce}} package?
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836936#action_12836936 ]
Zheng Shao commented on MAPREDUCE-1501:
---------------------------------------
Thanks for the feedback Ian.
I don't think FileSystem.listPath() returns "." or "..". If it does, I believe the current code in trunk will also break. The new unit test will also fail if that's the case.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated MAPREDUCE-1501:
----------------------------------
Attachment: MAPREDUCE-1501.1.branch-0.20.patch
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840122#action_12840122 ]
Hadoop QA commented on MAPREDUCE-1501:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12436481/MAPREDUCE-1501.1.trunk.patch
against trunk revision 916823.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/339/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/339/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/339/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/339/console
This message is automatically generated.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated MAPREDUCE-1501:
----------------------------------
Status: Patch Available (was: Open)
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836963#action_12836963 ]
Zheng Shao commented on MAPREDUCE-1501:
---------------------------------------
Thanks Dhruba. I missed the part "and other hidden directories". We do call PathFilter on the sub directories as well (see addInputPathRecursively(...)). Is that good enough or we want to split the PathFilters for files and the PathFilters for directories?
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836602#action_12836602 ]
Hadoop QA commented on MAPREDUCE-1501:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12436481/MAPREDUCE-1501.1.trunk.patch
against trunk revision 912471.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/469/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/469/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/469/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/469/console
This message is automatically generated.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao reopened MAPREDUCE-1501:
-----------------------------------
Reopened for Chris's comments.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840541#action_12840541 ]
dhruba borthakur commented on MAPREDUCE-1501:
---------------------------------------------
The failed unit test is TestMiniMRLocalFS.testWithLocal and is not related to this patch. I will commit this patch.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841428#action_12841428 ]
Hudson commented on MAPREDUCE-1501:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #248 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/248/])
. FileInputFormat supports multi-level, recursive
directory listing. (Zheng Shao via dhruba)
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao resolved MAPREDUCE-1501.
-----------------------------------
Resolution: Fixed
Will open a new one to address this issue.
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1501) FileInputFormat to support
multi-level/recursive directory listing
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-1501:
----------------------------------------
Fix Version/s: 0.21.1
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.21.1
>
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.