You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Antony Mayi <an...@yahoo.com> on 2017/05/11 23:21:11 UTC

weird serialization issue on beam 2.0.0rc2

I tried the 2.0.0rc2 and I started facing weird serialization issue that's not been happening on 0.6.0. I am able to reproduce it using the attached unit test - see first serialization attempt which is passing ok and second one slightly different (transform output type is generic) that's failing just on 2.0.0

Can anyone find a way around it? In my real case I depend on generics output types.

thanks,
antony.

Re: weird serialization issue on beam 2.0.0rc2

Posted by Lukasz Cwik <lc...@google.com>.

We were able to reproduce the issue and this is a regression in how we
serialize the SerializableCoder, we used to only save the class but now
save the class and type variable. The type variable is unnecessary and
being marked as transient.

On Thu, May 11, 2017 at 4:52 PM, Thomas Groh <tg...@google.com> wrote:

> Hey Antony;
>
> I've tried to update your code to compile in Java 7 and removed the
> SerializableUtils Dan mentioned, and I can't seem to reproduce the issue.
> Can you share more about the specific issue you're having? A stack trace
> would be really helpful.
>
> Thanks,
>
> Thomas
>
> On Thu, May 11, 2017 at 4:42 PM, Dan Halperin <dh...@apache.org> wrote:
>
>> Hi Anthony,
>>
>> I'm a little confused by your code snippet:
>>
>> SerializableUtils.serializeToByteArray(
>>     input.apply(ParDo.of(new NonGenericOutput<>())).apply("passing",
>> View.asSingleton())
>> );
>>
>> is serializing a PCollectionView object. Those are not necessarily
>> serializable and it's not clear that it makes sense to do this at all.
>>
>> Can you say more about what you're trying to do?
>>
>> Thanks,
>> Dan
>>
>> On Thu, May 11, 2017 at 4:21 PM, Antony Mayi <an...@yahoo.com>
>> wrote:
>>
>>> I tried the 2.0.0rc2 and I started facing weird serialization issue
>>> that's not been happening on 0.6.0. I am able to reproduce it using the
>>> attached unit test - see first serialization attempt which is passing ok
>>> and second one slightly different (transform output type is generic) that's
>>> failing just on 2.0.0
>>>
>>> Can anyone find a way around it? In my real case I depend on generics
>>> output types.
>>>
>>> thanks,
>>> antony.
>>
>>
>>
>

Re: weird serialization issue on beam 2.0.0rc2

Posted by Lukasz Cwik <lc...@google.com>.

Being tracked on https://issues.apache.org/jira/browse/BEAM-2275

On Thu, May 11, 2017 at 5:37 PM, Dan Halperin <dh...@google.com> wrote:

> Thanks Anthony -- I was wrong here when I said that PCollectionViews don't
> need to be serializable ; your original report is great and your new one is
> better.
>
> Great to catch this issue now!
>
> On Thu, May 11, 2017 at 5:16 PM, Antony Mayi <an...@yahoo.com> wrote:
>
>> please find another unit test with more realistic usecase. I am compiling
>> it using Java 8 and the stacktrace I get from the testGeneric is attached.
>>
>> thx,
>> a.
>>
>>
>> On Friday, 12 May 2017, 1:52, Thomas Groh <tg...@google.com> wrote:
>>
>>
>> Hey Antony;
>>
>> I've tried to update your code to compile in Java 7 and removed the
>> SerializableUtils Dan mentioned, and I can't seem to reproduce the issue.
>> Can you share more about the specific issue you're having? A stack trace
>> would be really helpful.
>>
>> Thanks,
>>
>> Thomas
>>
>> On Thu, May 11, 2017 at 4:42 PM, Dan Halperin <dh...@apache.org>
>> wrote:
>>
>> Hi Anthony,
>>
>> I'm a little confused by your code snippet:
>>
>> SerializableUtils.serializeToB yteArray(
>>     input.apply(ParDo.of(new NonGenericOutput<>())).apply(" passing",
>> View.asSingleton())
>> );
>>
>> is serializing a PCollectionView object. Those are not necessarily
>> serializable and it's not clear that it makes sense to do this at all.
>>
>> Can you say more about what you're trying to do?
>>
>> Thanks,
>> Dan
>>
>> On Thu, May 11, 2017 at 4:21 PM, Antony Mayi <an...@yahoo.com>
>> wrote:
>>
>> I tried the 2.0.0rc2 and I started facing weird serialization issue
>> that's not been happening on 0.6.0. I am able to reproduce it using the
>> attached unit test - see first serialization attempt which is passing ok
>> and second one slightly different (transform output type is generic) that's
>> failing just on 2.0.0
>>
>> Can anyone find a way around it? In my real case I depend on generics
>> output types.
>>
>> thanks,
>> antony.
>>
>>
>>
>>
>>
>>
>

Re: weird serialization issue on beam 2.0.0rc2

Posted by Dan Halperin <dh...@google.com>.

Thanks Anthony -- I was wrong here when I said that PCollectionViews don't
need to be serializable ; your original report is great and your new one is
better.

Great to catch this issue now!

On Thu, May 11, 2017 at 5:16 PM, Antony Mayi <an...@yahoo.com> wrote:

> please find another unit test with more realistic usecase. I am compiling
> it using Java 8 and the stacktrace I get from the testGeneric is attached.
>
> thx,
> a.
>
>
> On Friday, 12 May 2017, 1:52, Thomas Groh <tg...@google.com> wrote:
>
>
> Hey Antony;
>
> I've tried to update your code to compile in Java 7 and removed the
> SerializableUtils Dan mentioned, and I can't seem to reproduce the issue.
> Can you share more about the specific issue you're having? A stack trace
> would be really helpful.
>
> Thanks,
>
> Thomas
>
> On Thu, May 11, 2017 at 4:42 PM, Dan Halperin <dh...@apache.org> wrote:
>
> Hi Anthony,
>
> I'm a little confused by your code snippet:
>
> SerializableUtils.serializeToB yteArray(
>     input.apply(ParDo.of(new NonGenericOutput<>())).apply(" passing",
> View.asSingleton())
> );
>
> is serializing a PCollectionView object. Those are not necessarily
> serializable and it's not clear that it makes sense to do this at all.
>
> Can you say more about what you're trying to do?
>
> Thanks,
> Dan
>
> On Thu, May 11, 2017 at 4:21 PM, Antony Mayi <an...@yahoo.com> wrote:
>
> I tried the 2.0.0rc2 and I started facing weird serialization issue that's
> not been happening on 0.6.0. I am able to reproduce it using the attached
> unit test - see first serialization attempt which is passing ok and second
> one slightly different (transform output type is generic) that's failing
> just on 2.0.0
>
> Can anyone find a way around it? In my real case I depend on generics
> output types.
>
> thanks,
> antony.
>
>
>
>
>
>

Re: weird serialization issue on beam 2.0.0rc2

Posted by Antony Mayi <an...@yahoo.com>.

please find another unit test with more realistic usecase. I am compiling it using Java 8 and the stacktrace I get from the testGeneric is attached.
thx,a. 

    On Friday, 12 May 2017, 1:52, Thomas Groh <tg...@google.com> wrote:
 

 Hey Antony;
I've tried to update your code to compile in Java 7 and removed the SerializableUtils Dan mentioned, and I can't seem to reproduce the issue. Can you share more about the specific issue you're having? A stack trace would be really helpful.
Thanks,
Thomas
On Thu, May 11, 2017 at 4:42 PM, Dan Halperin <dh...@apache.org> wrote:

Hi Anthony,
I'm a little confused by your code snippet:
SerializableUtils.serializeToB yteArray(    input.apply(ParDo.of(new NonGenericOutput<>())).apply(" passing", View.asSingleton()));
is serializing a PCollectionView object. Those are not necessarily serializable and it's not clear that it makes sense to do this at all.
Can you say more about what you're trying to do?
Thanks,Dan
On Thu, May 11, 2017 at 4:21 PM, Antony Mayi <an...@yahoo.com> wrote:

I tried the 2.0.0rc2 and I started facing weird serialization issue that's not been happening on 0.6.0. I am able to reproduce it using the attached unit test - see first serialization attempt which is passing ok and second one slightly different (transform output type is generic) that's failing just on 2.0.0

Can anyone find a way around it? In my real case I depend on generics output types.

thanks,
antony.

Re: weird serialization issue on beam 2.0.0rc2

Posted by Thomas Groh <tg...@google.com>.

Hey Antony;

I've tried to update your code to compile in Java 7 and removed the
SerializableUtils Dan mentioned, and I can't seem to reproduce the issue.
Can you share more about the specific issue you're having? A stack trace
would be really helpful.

Thanks,

Thomas

On Thu, May 11, 2017 at 4:42 PM, Dan Halperin <dh...@apache.org> wrote:

> Hi Anthony,
>
> I'm a little confused by your code snippet:
>
> SerializableUtils.serializeToByteArray(
>     input.apply(ParDo.of(new NonGenericOutput<>())).apply("passing",
> View.asSingleton())
> );
>
> is serializing a PCollectionView object. Those are not necessarily
> serializable and it's not clear that it makes sense to do this at all.
>
> Can you say more about what you're trying to do?
>
> Thanks,
> Dan
>
> On Thu, May 11, 2017 at 4:21 PM, Antony Mayi <an...@yahoo.com> wrote:
>
>> I tried the 2.0.0rc2 and I started facing weird serialization issue
>> that's not been happening on 0.6.0. I am able to reproduce it using the
>> attached unit test - see first serialization attempt which is passing ok
>> and second one slightly different (transform output type is generic) that's
>> failing just on 2.0.0
>>
>> Can anyone find a way around it? In my real case I depend on generics
>> output types.
>>
>> thanks,
>> antony.
>
>
>

Re: weird serialization issue on beam 2.0.0rc2

Posted by Dan Halperin <dh...@apache.org>.

Hi Anthony,

I'm a little confused by your code snippet:

SerializableUtils.serializeToByteArray(
    input.apply(ParDo.of(new NonGenericOutput<>())).apply("passing",
View.asSingleton())
);

is serializing a PCollectionView object. Those are not necessarily
serializable and it's not clear that it makes sense to do this at all.

Can you say more about what you're trying to do?

Thanks,
Dan

On Thu, May 11, 2017 at 4:21 PM, Antony Mayi <an...@yahoo.com> wrote:

> I tried the 2.0.0rc2 and I started facing weird serialization issue that's
> not been happening on 0.6.0. I am able to reproduce it using the attached
> unit test - see first serialization attempt which is passing ok and second
> one slightly different (transform output type is generic) that's failing
> just on 2.0.0
>
> Can anyone find a way around it? In my real case I depend on generics
> output types.
>
> thanks,
> antony.