One key per output file

[Florian Leibert <fl...@leibert.de>]

Hi,
for convenience reasons, I was wondering if there is a simple way to  
produce one output file per key in the Reducer?

Thanks,
Florian 

Re: One key per output file

[Florian Leibert <fl...@leibert.de>]

Great, thanks for that hint - for some reason I expected that behavior  
to be a feature of the MultipleTextOutputFormat class - doing so  
solved my problem! Thanks!!

Here my code (I wanted to specifically omit outputting the key however  
still having a file per key) if anyone is interested:

.. in my ToolRunner:

conf.setOutputFormat(KeyBasedMultipleTextOutputFormat.class);


...

static class KeyBasedMultipleTextOutputFormat extends  
MultipleTextOutputFormat<Text, Text> {
		
	@Override
	/** Ensures that no key is written to the file */
	protected Text generateActualKey(Text key, Text value) {
	      return null;
         }
	/** Generates a filename that reflects the value of the key */
	protected String generateFileNameForKeyValue(Text key, Text v, String  
name) {
	
	      return key.toString().substring(0, key.toString().length());
	}
}


~Florian

On Oct 29, 2008, at 12:12 AM, Mice wrote:

> Did you override generateFileNameForKeyValue?
>
> 2008/10/29 Florian Leibert <fl...@leibert.de>:
>> Thanks Mice,
>> tried using that already - however this doesn't yield the desired  
>> results -
>> upon output collection (using the OutputCollector), it still  
>> produces only
>> one output file (note, I only have one input file, not multiple  
>> input files,
>> but want a file per key for the output...)
>>
>> Thanks
>>
>> Code Fragment:
>> jobConfig.setOutputFormat(MultipleTextOutputFormat.class);
>>
>>
>> On Oct 28, 2008, at 9:09 PM, Mice wrote:
>>
>>> MultipleOutputFormat meets your need. It is in 0.18.1.
>>>
>>> 2008/10/29 Florian Leibert <fl...@leibert.de>:
>>>>
>>>> Hi,
>>>> for convenience reasons, I was wondering if there is a simple way  
>>>> to
>>>> produce
>>>> one output file per key in the Reducer?
>>>>
>>>> Thanks,
>>>> Florian
>>>>
>>
>>

Re: One key per output file

[Mice <mi...@gmail.com>]

Did you override generateFileNameForKeyValue?

2008/10/29 Florian Leibert <fl...@leibert.de>:
> Thanks Mice,
> tried using that already - however this doesn't yield the desired results -
> upon output collection (using the OutputCollector), it still produces only
> one output file (note, I only have one input file, not multiple input files,
> but want a file per key for the output...)
>
> Thanks
>
> Code Fragment:
> jobConfig.setOutputFormat(MultipleTextOutputFormat.class);
>
>
> On Oct 28, 2008, at 9:09 PM, Mice wrote:
>
>> MultipleOutputFormat meets your need. It is in 0.18.1.
>>
>> 2008/10/29 Florian Leibert <fl...@leibert.de>:
>>>
>>> Hi,
>>> for convenience reasons, I was wondering if there is a simple way to
>>> produce
>>> one output file per key in the Reducer?
>>>
>>> Thanks,
>>> Florian
>>>
>
>

Re: One key per output file

[Florian Leibert <fl...@leibert.de>]

Thanks Mice,
tried using that already - however this doesn't yield the desired  
results - upon output collection (using the OutputCollector), it still  
produces only one output file (note, I only have one input file, not  
multiple input files, but want a file per key for the output...)

Thanks

Code Fragment:
jobConfig.setOutputFormat(MultipleTextOutputFormat.class);


On Oct 28, 2008, at 9:09 PM, Mice wrote:

> MultipleOutputFormat meets your need. It is in 0.18.1.
>
> 2008/10/29 Florian Leibert <fl...@leibert.de>:
>> Hi,
>> for convenience reasons, I was wondering if there is a simple way  
>> to produce
>> one output file per key in the Reducer?
>>
>> Thanks,
>> Florian
>>

Re: One key per output file

[Mice <mi...@gmail.com>]

MultipleOutputFormat meets your need. It is in 0.18.1.

2008/10/29 Florian Leibert <fl...@leibert.de>:
> Hi,
> for convenience reasons, I was wondering if there is a simple way to produce
> one output file per key in the Reducer?
>
> Thanks,
> Florian
>