You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by Daan Gerits <da...@gmail.com> on 2012/03/05 10:33:26 UTC

Append Storage

Hello everyone,

Is Pig capable of appending to a file? I know that an exception is thrown when a file exists using PigStorage, but is there a way to get around this?

Thanks,

Daan.

Re: Append Storage

Posted by Alan Gates <ga...@hortonworks.com>.

Append is not currently supported by FileOutputFormat, so it isn't currently supported by Pig.  In general appending isn't safe in MR because it is not atomic, which means there can be issues if a task fails and must be retried, with things like speculative execution, etc. 

Alan.

On Mar 5, 2012, at 6:27 AM, Russell Jurney wrote:

> I'm curious how Pig will use append in the future?
> 
> Russell Jurney
> twitter.com/rjurney
> russell.jurney@gmail.com
> datasyndrome.com
> 
> On Mar 5, 2012, at 1:34 AM, Daan Gerits <da...@gmail.com> wrote:
> 
>> Hello everyone,
>> 
>> Is Pig capable of appending to a file? I know that an exception is thrown when a file exists using PigStorage, but is there a way to get around this?
>> 
>> Thanks,
>> 
>> Daan.

Re: Append Storage

Posted by Russell Jurney <ru...@gmail.com>.

I'm curious how Pig will use append in the future?

Russell Jurney
twitter.com/rjurney
russell.jurney@gmail.com
datasyndrome.com

On Mar 5, 2012, at 1:34 AM, Daan Gerits <da...@gmail.com> wrote:

> Hello everyone,
>
> Is Pig capable of appending to a file? I know that an exception is thrown when a file exists using PigStorage, but is there a way to get around this?
>
> Thanks,
>
> Daan.