You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Russ Weeks <rw...@newbrightidea.com> on 2015/01/06 23:43:16 UTC
AccumuloOutputFormat and speculative execution
Hi, all,
When I use AccumuloOutputFormat in my MR job, I see mutations are written
to the table before each individual map tasks completes. If speculative
execution is enabled on my cluster and a map task is killed, will these
mutations be rolled back? If not, should I explicitly disable speculative
execution whenever I use AccumuloOutputFormat?
Or does AccumuloOutputFormat do this internally somehow, and I'm worried
about nothing?
Thanks,
-Russ
Re: AccumuloOutputFormat and speculative execution
Posted by John Vines <vi...@apache.org>.
No, they are not rolled back. If you cannot tolerate duplication of your
mutations (i.e. you depend on versions / aggregates ) then you should not
use speculative execution.
On Tue, Jan 6, 2015 at 5:43 PM, Russ Weeks <rw...@newbrightidea.com> wrote:
> Hi, all,
>
> When I use AccumuloOutputFormat in my MR job, I see mutations are written
> to the table before each individual map tasks completes. If speculative
> execution is enabled on my cluster and a map task is killed, will these
> mutations be rolled back? If not, should I explicitly disable speculative
> execution whenever I use AccumuloOutputFormat?
>
> Or does AccumuloOutputFormat do this internally somehow, and I'm worried
> about nothing?
>
> Thanks,
> -Russ
>
Re: AccumuloOutputFormat and speculative execution
Posted by Russ Weeks <rw...@newbrightidea.com>.
Good to know. Thanks very much, Christopher and John.
-Russ
On Tue, Jan 6, 2015 at 2:51 PM, Christopher <ct...@apache.org> wrote:
> The answer is no, they won't be rolled back. And, yes, if you want this
> behavior, you either need to make your mutations idempotent or disable
> speculative execution.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Tue, Jan 6, 2015 at 5:43 PM, Russ Weeks <rw...@newbrightidea.com>
> wrote:
>
>> Hi, all,
>>
>> When I use AccumuloOutputFormat in my MR job, I see mutations are written
>> to the table before each individual map tasks completes. If speculative
>> execution is enabled on my cluster and a map task is killed, will these
>> mutations be rolled back? If not, should I explicitly disable speculative
>> execution whenever I use AccumuloOutputFormat?
>>
>> Or does AccumuloOutputFormat do this internally somehow, and I'm worried
>> about nothing?
>>
>> Thanks,
>> -Russ
>>
>
>
Re: AccumuloOutputFormat and speculative execution
Posted by Christopher <ct...@apache.org>.
The answer is no, they won't be rolled back. And, yes, if you want this
behavior, you either need to make your mutations idempotent or disable
speculative execution.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Tue, Jan 6, 2015 at 5:43 PM, Russ Weeks <rw...@newbrightidea.com> wrote:
> Hi, all,
>
> When I use AccumuloOutputFormat in my MR job, I see mutations are written
> to the table before each individual map tasks completes. If speculative
> execution is enabled on my cluster and a map task is killed, will these
> mutations be rolled back? If not, should I explicitly disable speculative
> execution whenever I use AccumuloOutputFormat?
>
> Or does AccumuloOutputFormat do this internally somehow, and I'm worried
> about nothing?
>
> Thanks,
> -Russ
>