You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Corey Nolet <cj...@gmail.com> on 2015/01/15 03:19:15 UTC

Accumulators

What are the limitations of using Accumulators to get a union of a bunch of
small sets?

Let's say I have an RDD[Map{String,Any} and i want to do:

rdd.map(accumulator += Set(_.get("entityType").get))


What implication does this have on performance? I'm assuming it's not
immediately aggregating each time I call the += on the Accumulator. Is it
doing a local combine and then occasionally sending the results on the
current partition back to the driver?

Re: Accumulators

Posted by Imran Rashid <im...@therashids.com>.
You're understanding is basically correct.  Each task creates it's own
local accumulator, and just those results get merged together.

However, there are some performance limitations to be aware of.  First you
need enough memory on the executors to build up whatever those intermediate
results are.  Second, all the work of *merging* the results from each task
are done by the *driver*.  So if there is a lot of stuff to merge, that can
be slow, as its not distributed at all.

Hope that helps a little

Imran
On Jan 14, 2015 6:21 PM, "Corey Nolet" <cj...@gmail.com> wrote:

> Just noticed an error in my wording.
>
> Should be " I'm assuming it's not immediately aggregating on the driver
> each time I call the += on the Accumulator."
>
> On Wed, Jan 14, 2015 at 9:19 PM, Corey Nolet <cj...@gmail.com> wrote:
>
>> What are the limitations of using Accumulators to get a union of a bunch
>> of small sets?
>>
>> Let's say I have an RDD[Map{String,Any} and i want to do:
>>
>> rdd.map(accumulator += Set(_.get("entityType").get))
>>
>>
>> What implication does this have on performance? I'm assuming it's not
>> immediately aggregating each time I call the += on the Accumulator. Is it
>> doing a local combine and then occasionally sending the results on the
>> current partition back to the driver?
>>
>>
>>
>>
>

Re: Accumulators

Posted by Corey Nolet <cj...@gmail.com>.
Just noticed an error in my wording.

Should be " I'm assuming it's not immediately aggregating on the driver
each time I call the += on the Accumulator."

On Wed, Jan 14, 2015 at 9:19 PM, Corey Nolet <cj...@gmail.com> wrote:

> What are the limitations of using Accumulators to get a union of a bunch
> of small sets?
>
> Let's say I have an RDD[Map{String,Any} and i want to do:
>
> rdd.map(accumulator += Set(_.get("entityType").get))
>
>
> What implication does this have on performance? I'm assuming it's not
> immediately aggregating each time I call the += on the Accumulator. Is it
> doing a local combine and then occasionally sending the results on the
> current partition back to the driver?
>
>
>
>