You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by SRK <sw...@gmail.com> on 2016/03/16 00:29:14 UTC

How to add an accumulator for a Set in Spark

Hi,

How do I add an accumulator for a Set in Spark?

Thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-add-an-accumulator-for-a-Set-in-Spark-tp26510.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: How to add an accumulator for a Set in Spark

Posted by Adrien Mogenet <ad...@contentsquare.com>.
Btw, here is a great article about accumulators and all their related
traps!
http://imranrashid.com/posts/Spark-Accumulators/ (I'm not the author)

On 16 March 2016 at 18:24, swetha kasireddy <sw...@gmail.com>
wrote:

> OK. I did take a look at them. So once I have an accumulater for a
> HashSet, how can I check if a particular key is already present in the
> HashSet accumulator? I don't see any .contains method there. My requirement
> is that I need to keep accumulating the keys in the HashSet across all the
> tasks in various nodes and use it to do a check if the key is already
> present in the HashSet.
>
> On Tue, Mar 15, 2016 at 9:56 PM, pppsunil <pp...@gmail.com> wrote:
>
>> Have you looked at using Accumulable interface,  Take a look at Spark
>> documentation at
>> http://spark.apache.org/docs/latest/programming-guide.html#accumulators
>> it
>> gives example of how to use vector type for accumalator, which might be
>> very
>> close to what you need
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-add-an-accumulator-for-a-Set-in-Spark-tp26510p26514.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>


-- 

*Adrien Mogenet*
Head of Backend/Infrastructure
adrien.mogenet@contentsquare.com
http://www.contentsquare.com
50, avenue Montaigne - 75008 Paris

Re: How to add an accumulator for a Set in Spark

Posted by swetha kasireddy <sw...@gmail.com>.
OK. I did take a look at them. So once I have an accumulater for a HashSet,
how can I check if a particular key is already present in the HashSet
accumulator? I don't see any .contains method there. My requirement is that
I need to keep accumulating the keys in the HashSet across all the tasks in
various nodes and use it to do a check if the key is already present in the
HashSet.

On Tue, Mar 15, 2016 at 9:56 PM, pppsunil <pp...@gmail.com> wrote:

> Have you looked at using Accumulable interface,  Take a look at Spark
> documentation at
> http://spark.apache.org/docs/latest/programming-guide.html#accumulators it
> gives example of how to use vector type for accumalator, which might be
> very
> close to what you need
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-add-an-accumulator-for-a-Set-in-Spark-tp26510p26514.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: How to add an accumulator for a Set in Spark

Posted by pppsunil <pp...@gmail.com>.
Have you looked at using Accumulable interface,  Take a look at Spark
documentation at
http://spark.apache.org/docs/latest/programming-guide.html#accumulators it
gives example of how to use vector type for accumalator, which might be very
close to what you need 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-add-an-accumulator-for-a-Set-in-Spark-tp26510p26514.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: How to add an accumulator for a Set in Spark

Posted by Ted Yu <yu...@gmail.com>.
Please take a look at:
core/src/test/scala/org/apache/spark/AccumulatorSuite.scala

FYI

On Tue, Mar 15, 2016 at 4:29 PM, SRK <sw...@gmail.com> wrote:

> Hi,
>
> How do I add an accumulator for a Set in Spark?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-add-an-accumulator-for-a-Set-in-Spark-tp26510.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>