Could I write outputs in multiple directories?

[Jun Young Kim <ju...@gmail.com>]

Hi,

As I understand, a Hadoop can write multiple files in a directory.
but, it can't write output files in multiple directories. isn't it?


MultipleOutputs for generating multiple files.
FileInputFormat.addInputPaths for setting several input files 
simultaneously.

How could I do if I want to write outputs files in multiple directories 
depends on it's key?

for example)
A type key -> yyyyMMdd/A/output
B type Key -> yyyyMMdd/B/output
C type Key -> yyyyMMdd/C/output

thanks.

-- 
Junyoung Kim (juneng603@gmail.com)

Re: Could I write outputs in multiple directories?

[Matthew John <tm...@gmail.com>]

Hi Junyoung Kim,

You can try out MultipleOutputs.addNamedOutput() . The second
parameter u pass in is supposed to be the filename to be which you are
writing the reducer output. Therefore if your output folder is X
(using setOutputPath() ), you can try giving "A/output", "B/output",
"C/output" in the 2nd parameter space. It should write the
corresponding data to X/A/output , X/B/output and X/C/output
respectively I guess.

In the reducer, depending on the key , you can use getCollector() to
write it to different output paths.
For eg:
if (Key == A)
multipleoutputs.getCollector("A/output",reporter).collect(Key,Value);

Regards,
Matthew

On Mon, Feb 14, 2011 at 11:27 AM, Jun Young Kim <ju...@gmail.com> wrote:
>
> Hi,
>
> As I understand, a Hadoop can write multiple files in a directory.
> but, it can't write output files in multiple directories. isn't it?
>
>
> MultipleOutputs for generating multiple files.
> FileInputFormat.addInputPaths for setting several input files simultaneously.
>
> How could I do if I want to write outputs files in multiple directories depends on it's key?
>
> for example)
> A type key -> yyyyMMdd/A/output
> B type Key -> yyyyMMdd/B/output
> C type Key -> yyyyMMdd/C/output
>
> thanks.
>
> --
> Junyoung Kim (juneng603@gmail.com)
>