You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Kunsheng Chen <ke...@yahoo.com> on 2009/06/15 01:18:31 UTC
Could I output results into multiple files in hadoop ?
Hi, everyone,
Is there any way to output the results in Reduce to different files ?
thanks in advance,
-Kun
RE: Could I output results into multiple files in hadoop ?
Posted by Daniel Young <DY...@brandassetdigital.com>.
Er, MultipleOutputFormat. Sorry for the typo.
Dan
-----Original Message-----
From: Daniel Young [mailto:DYoung@brandassetdigital.com]
Sent: Monday, June 15, 2009 9:42 AM
To: core-user@hadoop.apache.org
Subject: RE: Could I output results into multiple files in hadoop ?
Or look into the MultipleFileOutput/MultipleTextOutputFormat classes. They provide a method, generateFileNameForKeyValue(), that allows the format to return arbitrary output filenames for given key/value pairs. It's deprecated in 0.20.0, but works well enough as far as I've seen.
Dan Young
-----Original Message-----
From: Jothi Padmanabhan [mailto:jothipn@yahoo-inc.com]
Sent: Monday, June 15, 2009 12:35 AM
To: core-user@hadoop.apache.org
Subject: Re: Could I output results into multiple files in hadoop ?
Would org.apache.hadoop.mapred.lib.MultipleOutputs help?
Jothi
> ________________________________
> From: Kunsheng Chen <ke...@yahoo.com>
> To: core-user@hadoop.apache.org
> Sent: Sunday, June 14, 2009 7:18:31 PM
> Subject: Could I output results into multiple files in hadoop ?
>
>
> Hi, everyone,
>
> Is there any way to output the results in Reduce to different files ?
>
>
> thanks in advance,
>
> -Kun
>
>
>
RE: Could I output results into multiple files in hadoop ?
Posted by Daniel Young <DY...@brandassetdigital.com>.
Or look into the MultipleFileOutput/MultipleTextOutputFormat classes. They provide a method, generateFileNameForKeyValue(), that allows the format to return arbitrary output filenames for given key/value pairs. It's deprecated in 0.20.0, but works well enough as far as I've seen.
Dan Young
-----Original Message-----
From: Jothi Padmanabhan [mailto:jothipn@yahoo-inc.com]
Sent: Monday, June 15, 2009 12:35 AM
To: core-user@hadoop.apache.org
Subject: Re: Could I output results into multiple files in hadoop ?
Would org.apache.hadoop.mapred.lib.MultipleOutputs help?
Jothi
> ________________________________
> From: Kunsheng Chen <ke...@yahoo.com>
> To: core-user@hadoop.apache.org
> Sent: Sunday, June 14, 2009 7:18:31 PM
> Subject: Could I output results into multiple files in hadoop ?
>
>
> Hi, everyone,
>
> Is there any way to output the results in Reduce to different files ?
>
>
> thanks in advance,
>
> -Kun
>
>
>
Re: Could I output results into multiple files in hadoop ?
Posted by Jothi Padmanabhan <jo...@yahoo-inc.com>.
Would org.apache.hadoop.mapred.lib.MultipleOutputs help?
Jothi
> ________________________________
> From: Kunsheng Chen <ke...@yahoo.com>
> To: core-user@hadoop.apache.org
> Sent: Sunday, June 14, 2009 7:18:31 PM
> Subject: Could I output results into multiple files in hadoop ?
>
>
> Hi, everyone,
>
> Is there any way to output the results in Reduce to different files ?
>
>
> thanks in advance,
>
> -Kun
>
>
>
Re: Could I output results into multiple files in hadoop ?
Posted by Zhengguo 'Mike' SUN <zh...@yahoo.com>.
I guess you could try to implement your own OutputFormat and RecordWriter. If that doesn't work, you could always write to HDFS directly using file system API.
________________________________
From: Kunsheng Chen <ke...@yahoo.com>
To: core-user@hadoop.apache.org
Sent: Sunday, June 14, 2009 7:18:31 PM
Subject: Could I output results into multiple files in hadoop ?
Hi, everyone,
Is there any way to output the results in Reduce to different files ?
thanks in advance,
-Kun