You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Amit Sela <am...@gmail.com> on 2016/06/13 19:15:24 UTC
LegacyAccumulatorWrapper basically requires the Accumulator value to
implement equlas() or it will fail on isZero()
It seems that if you have an AccumulatorParam (or AccumulableParam) where
"R" is not a primitive, it will need to implement equlas() if the
implementation of the zero() creates a new instance (which I guess it will
in those cases).
This is where isZero applies the comparison:
https://github.com/apache/spark/blob/254bc8c34e70241508bdfc8ff42a65491f5280cd/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L462
I overcame this by providing null in zero and instantiating in
addAccumulator() but that's more of a hack, on the other hand, I don't see
a trivial solution, which is one of the reasons I'm writing this.
Anyone encounter this when upgrading to 2.0 ? with non-trivial
Accumulators of course..
Thanks,
Amit
Re: LegacyAccumulatorWrapper basically requires the Accumulator value
to implement equlas() or it will fail on isZero()
Posted by Amit Sela <am...@gmail.com>.
I thought so, and I agree. Still good to have this indexed here :)
On Mon, Jun 13, 2016 at 10:43 PM Sean Owen <so...@cloudera.com> wrote:
> I think that's right, but that seems as expected. If you're going to
> use this utility wrapper class, it can only determine if something is
> zero by comparing it to your 'zero' object, and that means defining
> equality. I suspect it's uncommon to accumulate things that aren't
> primitives or standard collections, which already define equality. But
> I'd expect I'd have to define equality for some custom user class in
> general if handing it over for a library to compare, add, clear, etc.
>
> On Mon, Jun 13, 2016 at 8:15 PM, Amit Sela <am...@gmail.com> wrote:
> > It seems that if you have an AccumulatorParam (or AccumulableParam) where
> > "R" is not a primitive, it will need to implement equlas() if the
> > implementation of the zero() creates a new instance (which I guess it
> will
> > in those cases).
> > This is where isZero applies the comparison:
> >
> https://github.com/apache/spark/blob/254bc8c34e70241508bdfc8ff42a65491f5280cd/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L462
> > I overcame this by providing null in zero and instantiating in
> > addAccumulator() but that's more of a hack, on the other hand, I don't
> see a
> > trivial solution, which is one of the reasons I'm writing this.
> > Anyone encounter this when upgrading to 2.0 ? with non-trivial
> Accumulators
> > of course..
> >
> >
> > Thanks,
> > Amit
>
Re: LegacyAccumulatorWrapper basically requires the Accumulator value
to implement equlas() or it will fail on isZero()
Posted by Sean Owen <so...@cloudera.com>.
I think that's right, but that seems as expected. If you're going to
use this utility wrapper class, it can only determine if something is
zero by comparing it to your 'zero' object, and that means defining
equality. I suspect it's uncommon to accumulate things that aren't
primitives or standard collections, which already define equality. But
I'd expect I'd have to define equality for some custom user class in
general if handing it over for a library to compare, add, clear, etc.
On Mon, Jun 13, 2016 at 8:15 PM, Amit Sela <am...@gmail.com> wrote:
> It seems that if you have an AccumulatorParam (or AccumulableParam) where
> "R" is not a primitive, it will need to implement equlas() if the
> implementation of the zero() creates a new instance (which I guess it will
> in those cases).
> This is where isZero applies the comparison:
> https://github.com/apache/spark/blob/254bc8c34e70241508bdfc8ff42a65491f5280cd/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L462
> I overcame this by providing null in zero and instantiating in
> addAccumulator() but that's more of a hack, on the other hand, I don't see a
> trivial solution, which is one of the reasons I'm writing this.
> Anyone encounter this when upgrading to 2.0 ? with non-trivial Accumulators
> of course..
>
>
> Thanks,
> Amit
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org