Using the "fs" command in pig

[Andy Srine <an...@gmail.com>]

Hi Guys,

I have a list of file paths in a pig variable called "path" and how would I
iterate over it and run fs commands on it? I am specifically trying to copy
each file to another location in HDFS.

grunt> dump path;

(/user/andy/output_data/000000_0)

(/user/andy/tmp_j)

(/user/andy/tmp_l)


I want to do something like this.


B = FOREACH path 'fs -cp B /user/andy/new/'


Thanks,
Andy

Re: Using the "fs" command in pig

[Shahab Yunus <sh...@gmail.com>]

Not an exact solution but an idea.

You can use the STRSPLIT function first in Pig to parse your variable
'path'.
http://pig.apache.org/docs/r0.11.1/func.html#strsplit

Then for each split-ed path you can use the STORE command.
http://pig.apache.org/docs/r0.7.0/piglatin_ref1.html#Store+vs.+Dump

Regards,
Shahab


On Wed, Mar 19, 2014 at 8:18 PM, Andy Srine <an...@gmail.com> wrote:

> Hi Guys,
>
> I have a list of file paths in a pig variable called "path" and how would I
> iterate over it and run fs commands on it? I am specifically trying to copy
> each file to another location in HDFS.
>
> grunt> dump path;
>
> (/user/andy/output_data/000000_0)
>
> (/user/andy/tmp_j)
>
> (/user/andy/tmp_l)
>
>
> I want to do something like this.
>
>
> B = FOREACH path 'fs -cp B /user/andy/new/'
>
>
> Thanks,
> Andy
>