You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by maha <ma...@umail.ucsb.edu> on 2010/11/17 21:36:34 UTC

repeat a job for different files

Hi,

   When I set my inputFileFormat to take an input directory with three files in, the job is processed on all three and the output is one containing the result from all of them. 

Instead I want the job to be repeated separately for each inputFile and hence a different output. 

    Eg.

         wordCount(input)   where input/ file1.txt  file2.txt ... fileN.txt
        
        This happens:   output/    outputFile.txt will contain all words from all the files along with their counts.
  
        I want: output /   outputFile1.txt  for file1.txt words    , ........ , outputFileN.txt for fileN.txt


       Thanks,
          Maha

Re: repeat a job for different files

Posted by Harsh J <qw...@gmail.com>.
On Fri, Nov 19, 2010 at 12:19 AM, maha <ma...@umail.ucsb.edu> wrote:
>  If I have only one MR code and pass one file at a time to it, the problem will be in the FileOutputFormat because it will say output file is already there!

Unless you remove it post-job; kind of like using it as a staging
directory and moving it out at the end of each job.

>
>  So I had to repeat the same code for MR three times with only difference in the output directory which is unreasonable because what if I have 100 files :(

Loop with a script, should be easy!

>
>  I guess another solution would be to know HOW to add an output file into an existing directory instead of creating a new one (using the FileOutputFormat) ?

Why don't you just fs.move() the output directory upon completion to a
set location, every job's end?

-- 
Harsh J
www.harshj.com

Re: repeat a job for different files

Posted by maha <ma...@umail.ucsb.edu>.
Hi Alex, 

  Thanks for the reply well this is what I did. But the problem then is that I have to write the MR code three times !!!

 If I have only one MR code and pass one file at a time to it, the problem will be in the FileOutputFormat because it will say output file is already there!

  So I had to repeat the same code for MR three times with only difference in the output directory which is unreasonable because what if I have 100 files :(

  I guess another solution would be to know HOW to add an output file into an existing directory instead of creating a new one (using the FileOutputFormat) ?

  Maha

On Nov 17, 2010, at 10:11 PM, Alex Baranau wrote:

> In case you need to process the files separately, use one MR job for each
> file. You can add a single file as input. I believe you'll need to iterate
> over all files in input dir and start job instance for each file. You can do
> this in java code or in script or... depending on your case.
> 
> Alex Baranau
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
> 
> On Wed, Nov 17, 2010 at 10:36 PM, maha <ma...@umail.ucsb.edu> wrote:
> 
>> Hi,
>> 
>>  When I set my inputFileFormat to take an input directory with three files
>> in, the job is processed on all three and the output is one containing the
>> result from all of them.
>> 
>> Instead I want the job to be repeated separately for each inputFile and
>> hence a different output.
>> 
>>   Eg.
>> 
>>        wordCount(input)   where input/ file1.txt  file2.txt ... fileN.txt
>> 
>>       This happens:   output/    outputFile.txt will contain all words
>> from all the files along with their counts.
>> 
>>       I want: output /   outputFile1.txt  for file1.txt words    ,
>> ........ , outputFileN.txt for fileN.txt
>> 
>> 
>>      Thanks,
>>          Maha
>> 
>>

Re: repeat a job for different files

Posted by Alex Baranau <al...@gmail.com>.
In case you need to process the files separately, use one MR job for each
file. You can add a single file as input. I believe you'll need to iterate
over all files in input dir and start job instance for each file. You can do
this in java code or in script or... depending on your case.

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase

On Wed, Nov 17, 2010 at 10:36 PM, maha <ma...@umail.ucsb.edu> wrote:

> Hi,
>
>   When I set my inputFileFormat to take an input directory with three files
> in, the job is processed on all three and the output is one containing the
> result from all of them.
>
> Instead I want the job to be repeated separately for each inputFile and
> hence a different output.
>
>    Eg.
>
>         wordCount(input)   where input/ file1.txt  file2.txt ... fileN.txt
>
>        This happens:   output/    outputFile.txt will contain all words
> from all the files along with their counts.
>
>        I want: output /   outputFile1.txt  for file1.txt words    ,
> ........ , outputFileN.txt for fileN.txt
>
>
>       Thanks,
>           Maha
>
>