You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Nitin Khandelwal <ni...@germinait.com> on 2011/02/11 11:43:10 UTC

Custom Input format...

Hi,
I want to give a folder as input path to Map Red. Each Task should read one
file out of that folder at once . i was using it before in 0.19 using
multiFileSplit Format and my own Input format extending it. can u plz tell
how to do the same in 0.21.0.

Thanks,
-- 


Nitin Khandelwal

Re: Custom Input format...

Posted by David Rosenstrauch <da...@darose.net>.

On 02/11/2011 05:43 AM, Nitin Khandelwal wrote:
> Hi,
> I want to give a folder as input path to Map Red. Each Task should read one
> file out of that folder at once . i was using it before in 0.19 using
> multiFileSplit Format and my own Input format extending it. can u plz tell
> how to do the same in 0.21.0.
>
> Thanks,

No need for custom input format here.  In the code that creates/launches 
your job, iterate through each file in the directory using:

http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/fs/FileSystem.html#listStatus(org.apache.hadoop.fs.Path)

get the path of each file using:

http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/fs/FileStatus.html#getPath()

and then for each path call:

http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html#addInputPath(org.apache.hadoop.mapreduce.Job,%20org.apache.hadoop.fs.Path)

DR