You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Matt <dr...@gmail.com> on 2016/12/19 20:56:44 UTC
Serializing NULLs
Hello list,
I'm getting this error:
*java.lang.RuntimeException: Could not forward element to next operator*
*...*
*Caused by: java.lang.NullPointerException: in com.entities.Sector in map
in double null of double of map in field properties of com.entities.Sector*
*...*
*Caused by: java.lang.NullPointerException*
The field mentioned is a HashMap<String, Double>, and some keys are mapped
to null values.
Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?
Regards,
Matt
Re: Serializing NULLs
Posted by Matt <dr...@gmail.com>.
Here is the code of a Double wrapper with null support [1].
[1] https://gist.github.com/a8e8aa377957d3d51eadf36fe5c92a9e
On Tue, Dec 20, 2016 at 4:26 PM, Anirudh Mallem <an...@247-inc.com>
wrote:
> If you are using Avro generated classes then you cannot have your values
> null.
> https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-Whyisn'
> teveryvalueinAvronullable?
>
> From: Stephan Ewen
> Reply-To: "user@flink.apache.org"
> Date: Tuesday, December 20, 2016 at 8:17 AM
> To: "user@flink.apache.org"
> Subject: Re: Serializing NULLs
>
> Thanks for sharing the stack trace.
>
> This seems not really Flink related, it is part of the specific Avro
> encoding logic.
> The Avro Generic Record Type apparently does not allow the map value to be
> null.
>
>
>
> On Tue, Dec 20, 2016 at 4:55 PM, Matt <dr...@gmail.com> wrote:
>
>> Here is the back trace: https://gist.github.com
>> /56af4818bcf5dee6b97c248fd9233c67
>>
>> In the meanwhile I've solved the issue by creating a POJO class where
>> null is just Long.MIN_VALUE, that with a custom equals() made the trick. I
>> guess it's not as fast as de/serializing Double though.
>>
>> If you need any other information let me know.
>>
>> Regards,
>> Matt
>>
>> On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> The "null" support in some types is not fully developed. However in that
>>> case I am wondering why it does not work. Can you share the stack trace, so
>>> we can take a look at the serializer?
>>>
>>>
>>>
>>> On Mon, Dec 19, 2016 at 9:56 PM, Matt <dr...@gmail.com> wrote:
>>>
>>>> Hello list,
>>>>
>>>> I'm getting this error:
>>>>
>>>>
>>>> *java.lang.RuntimeException: Could not forward element to next operator
>>>> *
>>>> *...*
>>>> *Caused by: java.lang.NullPointerException: in com.entities.Sector in
>>>> map in double null of double of map in field properties of
>>>> com.entities.Sector*
>>>> *...*
>>>> *Caused by: java.lang.NullPointerException*
>>>>
>>>> The field mentioned is a HashMap<String, Double>, and some keys are
>>>> mapped to null values.
>>>>
>>>> Why isn't it possible to forward/serialize those elements with null
>>>> values?
>>>> What do you do when your elements may contain nulls?
>>>>
>>>> Regards,
>>>> Matt
>>>>
>>>
>>>
>>
>
RE: Serializing NULLs
Posted by "Newport, Billy" <Bi...@gs.com>.
Map<union<null,double>> in your avro schema is what you want here if the map values are nullable.
From: Anirudh Mallem [mailto:anirudh.mallem@247-inc.com]
Sent: Tuesday, December 20, 2016 2:26 PM
To: user@flink.apache.org
Subject: Re: Serializing NULLs
If you are using Avro generated classes then you cannot have your values null.
https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-Whyisn'teveryvalueinAvronullable?<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_AVRO_FAQ-23FAQ-2DWhyisn-27teveryvalueinAvronullable-3F&d=DgMGaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=rlkM70D3djmDN7dGPzzbVKG26ShcTFDMKlX5AWucE5Q&m=lDkmaFdQUGxF0wCdZ6mHjtKO8fwbxwv7ya1D-utjZ3o&s=yBn5YzwBKzOYD2wOxud0pnwWyjLa2hMDoAME91wM7JA&e=>
From: Stephan Ewen
Reply-To: "user@flink.apache.org<ma...@flink.apache.org>"
Date: Tuesday, December 20, 2016 at 8:17 AM
To: "user@flink.apache.org<ma...@flink.apache.org>"
Subject: Re: Serializing NULLs
Thanks for sharing the stack trace.
This seems not really Flink related, it is part of the specific Avro encoding logic.
The Avro Generic Record Type apparently does not allow the map value to be null.
On Tue, Dec 20, 2016 at 4:55 PM, Matt <dr...@gmail.com>> wrote:
Here is the back trace: https://gist.github.com/56af4818bcf5dee6b97c248fd9233c67<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_56af4818bcf5dee6b97c248fd9233c67&d=DgMGaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=rlkM70D3djmDN7dGPzzbVKG26ShcTFDMKlX5AWucE5Q&m=lDkmaFdQUGxF0wCdZ6mHjtKO8fwbxwv7ya1D-utjZ3o&s=mUF_Xp5joSe3H2vF2CIUEom76khbr4_lcnvZHZ-H7bc&e=>
In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double though.
If you need any other information let me know.
Regards,
Matt
On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <se...@apache.org>> wrote:
The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?
On Mon, Dec 19, 2016 at 9:56 PM, Matt <dr...@gmail.com>> wrote:
Hello list,
I'm getting this error:
java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException
The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.
Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?
Regards,
Matt
Re: Serializing NULLs
Posted by Anirudh Mallem <an...@247-inc.com>.
If you are using Avro generated classes then you cannot have your values null.
https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-Whyisn'teveryvalueinAvronullable?
From: Stephan Ewen
Reply-To: "user@flink.apache.org<ma...@flink.apache.org>"
Date: Tuesday, December 20, 2016 at 8:17 AM
To: "user@flink.apache.org<ma...@flink.apache.org>"
Subject: Re: Serializing NULLs
Thanks for sharing the stack trace.
This seems not really Flink related, it is part of the specific Avro encoding logic.
The Avro Generic Record Type apparently does not allow the map value to be null.
On Tue, Dec 20, 2016 at 4:55 PM, Matt <dr...@gmail.com>> wrote:
Here is the back trace: https://gist.github.com/56af4818bcf5dee6b97c248fd9233c67
In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double though.
If you need any other information let me know.
Regards,
Matt
On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <se...@apache.org>> wrote:
The "null" support in some types is not fully developed. However in that case I am wondering why it does not work. Can you share the stack trace, so we can take a look at the serializer?
On Mon, Dec 19, 2016 at 9:56 PM, Matt <dr...@gmail.com>> wrote:
Hello list,
I'm getting this error:
java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException
The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.
Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?
Regards,
Matt
Re: Serializing NULLs
Posted by Stephan Ewen <se...@apache.org>.
Thanks for sharing the stack trace.
This seems not really Flink related, it is part of the specific Avro
encoding logic.
The Avro Generic Record Type apparently does not allow the map value to be
null.
On Tue, Dec 20, 2016 at 4:55 PM, Matt <dr...@gmail.com> wrote:
> Here is the back trace: https://gist.github.com/
> 56af4818bcf5dee6b97c248fd9233c67
>
> In the meanwhile I've solved the issue by creating a POJO class where null
> is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess
> it's not as fast as de/serializing Double though.
>
> If you need any other information let me know.
>
> Regards,
> Matt
>
> On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <se...@apache.org> wrote:
>
>> The "null" support in some types is not fully developed. However in that
>> case I am wondering why it does not work. Can you share the stack trace, so
>> we can take a look at the serializer?
>>
>>
>>
>> On Mon, Dec 19, 2016 at 9:56 PM, Matt <dr...@gmail.com> wrote:
>>
>>> Hello list,
>>>
>>> I'm getting this error:
>>>
>>>
>>> *java.lang.RuntimeException: Could not forward element to next operator*
>>> *...*
>>> *Caused by: java.lang.NullPointerException: in com.entities.Sector in
>>> map in double null of double of map in field properties of
>>> com.entities.Sector*
>>> *...*
>>> *Caused by: java.lang.NullPointerException*
>>>
>>> The field mentioned is a HashMap<String, Double>, and some keys are
>>> mapped to null values.
>>>
>>> Why isn't it possible to forward/serialize those elements with null
>>> values?
>>> What do you do when your elements may contain nulls?
>>>
>>> Regards,
>>> Matt
>>>
>>
>>
>
Re: Serializing NULLs
Posted by Deepak Sharma <de...@gmail.com>.
Hi Matt
Can you please share the code like how you are mapping null to long?
Thanks
Deepak
On Tue, Dec 20, 2016 at 9:25 PM, Matt <dr...@gmail.com> wrote:
> Here is the back trace: https://gist.github.com/
> 56af4818bcf5dee6b97c248fd9233c67
>
> In the meanwhile I've solved the issue by creating a POJO class where null
> is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess
> it's not as fast as de/serializing Double though.
>
> If you need any other information let me know.
>
> Regards,
> Matt
>
> On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <se...@apache.org> wrote:
>
>> The "null" support in some types is not fully developed. However in that
>> case I am wondering why it does not work. Can you share the stack trace, so
>> we can take a look at the serializer?
>>
>>
>>
>> On Mon, Dec 19, 2016 at 9:56 PM, Matt <dr...@gmail.com> wrote:
>>
>>> Hello list,
>>>
>>> I'm getting this error:
>>>
>>>
>>> *java.lang.RuntimeException: Could not forward element to next operator*
>>> *...*
>>> *Caused by: java.lang.NullPointerException: in com.entities.Sector in
>>> map in double null of double of map in field properties of
>>> com.entities.Sector*
>>> *...*
>>> *Caused by: java.lang.NullPointerException*
>>>
>>> The field mentioned is a HashMap<String, Double>, and some keys are
>>> mapped to null values.
>>>
>>> Why isn't it possible to forward/serialize those elements with null
>>> values?
>>> What do you do when your elements may contain nulls?
>>>
>>> Regards,
>>> Matt
>>>
>>
>>
>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net
Re: Serializing NULLs
Posted by Matt <dr...@gmail.com>.
Here is the back trace:
https://gist.github.com/56af4818bcf5dee6b97c248fd9233c67
In the meanwhile I've solved the issue by creating a POJO class where null
is just Long.MIN_VALUE, that with a custom equals() made the trick. I guess
it's not as fast as de/serializing Double though.
If you need any other information let me know.
Regards,
Matt
On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <se...@apache.org> wrote:
> The "null" support in some types is not fully developed. However in that
> case I am wondering why it does not work. Can you share the stack trace, so
> we can take a look at the serializer?
>
>
>
> On Mon, Dec 19, 2016 at 9:56 PM, Matt <dr...@gmail.com> wrote:
>
>> Hello list,
>>
>> I'm getting this error:
>>
>>
>> *java.lang.RuntimeException: Could not forward element to next operator*
>> *...*
>> *Caused by: java.lang.NullPointerException: in com.entities.Sector in map
>> in double null of double of map in field properties of com.entities.Sector*
>> *...*
>> *Caused by: java.lang.NullPointerException*
>>
>> The field mentioned is a HashMap<String, Double>, and some keys are
>> mapped to null values.
>>
>> Why isn't it possible to forward/serialize those elements with null
>> values?
>> What do you do when your elements may contain nulls?
>>
>> Regards,
>> Matt
>>
>
>
Re: Serializing NULLs
Posted by Stephan Ewen <se...@apache.org>.
The "null" support in some types is not fully developed. However in that
case I am wondering why it does not work. Can you share the stack trace, so
we can take a look at the serializer?
On Mon, Dec 19, 2016 at 9:56 PM, Matt <dr...@gmail.com> wrote:
> Hello list,
>
> I'm getting this error:
>
>
> *java.lang.RuntimeException: Could not forward element to next operator*
> *...*
> *Caused by: java.lang.NullPointerException: in com.entities.Sector in map
> in double null of double of map in field properties of com.entities.Sector*
> *...*
> *Caused by: java.lang.NullPointerException*
>
> The field mentioned is a HashMap<String, Double>, and some keys are mapped
> to null values.
>
> Why isn't it possible to forward/serialize those elements with null values?
> What do you do when your elements may contain nulls?
>
> Regards,
> Matt
>