You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Steve Lewis <lo...@gmail.com> on 2014/11/13 00:05:24 UTC
How (in Java) do I create an Accumulator of type Long
JavaSparkContext currentContext = ...;
Accumulator<Integer> accumulator = currentContext.accumulator(0,
"MyAccumulator");
will create an Accumulator of Integers. For many large Data problems
Integer is too small and Long is a better type.
I see a call like the following
AccumulatorParam<Long> param = ?? how do I get one of these;
// no compiler is unhappy with this call
Accumulator<Long> accumulatorLong = currentContext.accumulator(new
Long(0), acc ,param);
but NONE - ZERO documentation on its use, how to get a
AccumulatorParam<Long> or how to turn one into an Accumulator.
Any ideas
Re: How (in Java) do I create an Accumulator of type Long
Posted by Sean Owen <so...@cloudera.com>.
Look again, the type is AccumulatorParam, not AccumulableParam. But
yes that's what you do.
On Thu, Nov 13, 2014 at 4:32 AM, Steve Lewis <lo...@gmail.com> wrote:
> I see Javadoc Style documentation but nothing that looks like a code sample
> I tried the following before asking
>
> public static class LongAccumulableParam implements
> AccumulableParam<Long,Long>,Serializable {
> @Override
> public Long addAccumulator(final Long r, final Long t) {
> return r + t;
> }
> @Override
> public Long addInPlace(final Long r1, final Long r2) {
> return r1 + r2;
> }
> @Override
> public Long zero(final Long initialValue) {
> return 0L;
> }
> }
>
> sparkContext.currentContext.accumulator(0L, "myAccumulator, new
> LongAccumulableParam ());
>
> does not compile which is why I ask for code samples
>
> The Javados for accumulator says
> accumulator(T initialValue, AccumulatorParam<T> accumulatorParam)
> Create an Accumulator variable of a given type, which tasks can "add" values
> to using the add method.
> which is a LONG way from a working code sample
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How (in Java) do I create an Accumulator of type Long
Posted by Steve Lewis <lo...@gmail.com>.
I see Javadoc Style documentation but nothing that looks like a code sample
I tried the following before asking
public static class LongAccumulableParam implements
AccumulableParam<Long,Long>,Serializable {
@Override
public Long addAccumulator(final Long r, final Long t) {
return r + t;
}
@Override
public Long addInPlace(final Long r1, final Long r2) {
return r1 + r2;
}
@Override
public Long zero(final Long initialValue) {
return 0L;
}
}
sparkContext.currentContext.accumulator(0L, "myAccumulator,
new LongAccumulableParam ());
does not compile which is why I ask for code samples
The Javados for accumulator says
*accumulator
<https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/api/java/JavaSparkContext.html#accumulator(T,
org.apache.spark.AccumulatorParam)>*(T initialValue, AccumulatorParam
<https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/AccumulatorParam.html>
<T> accumulatorParam)
Create an Accumulator
<https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/Accumulator.html>
variable
of a given type, which tasks can "add" values to using the add method.
which is a LONG way from a working code sample
On Wed, Nov 12, 2014 at 8:18 PM, Sean Owen <so...@cloudera.com> wrote:
> It's the exact same API you've already found, and it's documented:
>
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.AccumulatorParam
>
> JavaSparkContext has helper methods for int and double but not long. You
> can just make your own little implementation of AccumulatorParam<Long>
> right? ... which would be nice to add to JavaSparkContext.
>
> On Wed, Nov 12, 2014 at 11:05 PM, Steve Lewis <lo...@gmail.com>
> wrote:
>
>> JavaSparkContext currentContext = ...;
>> Accumulator<Integer> accumulator = currentContext.accumulator(0,
>> "MyAccumulator");
>>
>> will create an Accumulator of Integers. For many large Data problems
>> Integer is too small and Long is a better type.
>> I see a call like the following
>>
>> AccumulatorParam<Long> param = ?? how do I get one of these;
>> // no compiler is unhappy with this call
>> Accumulator<Long> accumulatorLong =
>> currentContext.accumulator(new Long(0), acc ,param);
>>
>> but NONE - ZERO documentation on its use, how to get a
>> AccumulatorParam<Long> or how to turn one into an Accumulator.
>> Any ideas
>>
>>
>>
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: How (in Java) do I create an Accumulator of type Long
Posted by Sean Owen <so...@cloudera.com>.
It's the exact same API you've already found, and it's documented:
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.AccumulatorParam
JavaSparkContext has helper methods for int and double but not long. You
can just make your own little implementation of AccumulatorParam<Long>
right? ... which would be nice to add to JavaSparkContext.
On Wed, Nov 12, 2014 at 11:05 PM, Steve Lewis <lo...@gmail.com> wrote:
> JavaSparkContext currentContext = ...;
> Accumulator<Integer> accumulator = currentContext.accumulator(0,
> "MyAccumulator");
>
> will create an Accumulator of Integers. For many large Data problems
> Integer is too small and Long is a better type.
> I see a call like the following
>
> AccumulatorParam<Long> param = ?? how do I get one of these;
> // no compiler is unhappy with this call
> Accumulator<Long> accumulatorLong = currentContext.accumulator(new
> Long(0), acc ,param);
>
> but NONE - ZERO documentation on its use, how to get a
> AccumulatorParam<Long> or how to turn one into an Accumulator.
> Any ideas
>
>
>