Combining pig output

[James Newhaven <ja...@gmail.com>]

Hi,

After running my pig script telling it to store my final relation, I know
it is normal for it to save the data into multiple parts

e.g.

part-m-00000
part-m-00001

Is it possible to configure pig to concatenate all of the parts into a
single file? I'd also like to rename the output to another name e.g.
results.csv

Thanks,
James

Re: Combining pig output

[Dmitriy Ryaboy <dv...@gmail.com>]

If you are just trying to get your data into a single file to copy to
local fs, you can use hadoop fs -getmerge
Pig does not give you such a hook explicitly.
If you really want to, you can encode it:

my_real_output = do_something;
single_file_output = foreach (group my_real_output all) generate
flatten(my_real_output);
store single_file_output into 'output_directory';

D

On Sun, Apr 15, 2012 at 8:49 AM, James Newhaven
<ja...@gmail.com> wrote:
> Hi,
>
> After running my pig script telling it to store my final relation, I know
> it is normal for it to save the data into multiple parts
>
> e.g.
>
> part-m-00000
> part-m-00001
>
> Is it possible to configure pig to concatenate all of the parts into a
> single file? I'd also like to rename the output to another name e.g.
> results.csv
>
> Thanks,
> James