Re: Delete hdfs directory afterpig execution

[Rohini Palaniswamy <ro...@apache.org>]

> However the fs command  throws an error
 What error do you get? Is it "Could not find schema file" ?

> Also is  there a guarantee that the fs command will be executed in order ?
Yes. Whenever fs commands are encountered, pig executes the statements
prior to it, executes the fs command and then executes the statements after
it. So it is generally recommended to place the fs statements in the
beginning or end of the script to avoid generating a sub-optimal plans and
repetition of processing due to fs statements in the middle.




On Thu, Jan 31, 2019 at 8:34 AM Nishanth S <ni...@gmail.com> wrote:

> Hey folks,
> I have a  pig script as below. I want to delete a directory after  data is
> moved   from location x to y as you see below. I then want to delete the
> directory .However the fs command  throws an error .Is this the correct
> way  to delete the directory .Also is  there a guarantee that the fs
> command will be executed in order ?
>
> SET pig.maxCombinedSplitSize 268435456;
> A = load '$SOURCE' using org.apache.pig.piggybank.storage.avro.AvroStorage
> ();
> SET mapred.output.compress true
> SET mapred.output.compression.codec
> org.apache.hadoop.io.compress.SnappyCodec
> SET avro.output.codec snappy
> store A into '$TARGET' using
> org.apache.pig.piggybank.storage.avro.AvroStorage ();
> fs -rm -r '$SOURCE'
>
> Thanks,
> Nishanth
>