You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Charith Wickramarachchi <ch...@gmail.com> on 2017/01/07 02:06:18 UTC
HashMap/HashSet Serialization Issue
Hi All,
I am using flink-gelly and using a custom POJO type as the
VertexValue/MessageType (I am using the vertex-centric model). The POJO
contains HashMap/HashSet as members. While executing the job I am getting
following Log message.
17:50:53,582 INFO org.apache.flink.api.java.typeutils.TypeExtractor
- No fields detected for class java.util.HashSet. Cannot be used as a
PojoType. Will be handled as GenericType
17:50:53,583 INFO org.apache.flink.api.java.typeutils.TypeExtractor
- class java.util.HashMap is not a valid POJO type
Is there a way to resolve this issue?
Thanks,
Charith
--
Charith Dhanushka Wickramaarachchi
Tel +1 213 447 4253
Blog http://charith.wickramaarachchi.org/
<http://charithwiki.blogspot.com/>
Twitter @charithwiki <https://twitter.com/charithwiki>
This communication may contain privileged or other confidential information
and is intended exclusively for the addressee/s. If you are not the
intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating
that fact and delete the copy you received and in addition, you should not
print, copy, retransmit, disseminate, or otherwise use the information
contained in this communication. Internet communications cannot be
guaranteed to be timely, secure, error or virus-free. The sender does not
accept liability for any errors or omissions
Re: HashMap/HashSet Serialization Issue
Posted by Charith Wickramarachchi <ch...@gmail.com>.
Thanks very much.
Regards,
Charith
On Sat, Jan 7, 2017 at 3:42 AM, Jark Wu <wu...@alibaba-inc.com> wrote:
> Hi Charith,
>
> The LOG message has nothing to worry about. The HashMap and HashSet
> members in your POJO is not a PojoType,
> so they will be handled as GenericType, i.e. will be
> serialized/deserialized by Kryo. But your custom POJO type will still
> recognized as PojoType, i.e. will be serialized/deserialized by Flink
> serialization framework.
>
> In a word, your custom POJO can be serialized/deserialized correctly.
> There is no issue in it. :)
>
> - Jark Wu
>
> 在 2017年1月7日，下午12:02，Charith Wickramarachchi <ch...@gmail.com>
> 写道：
>
> Thanks very much.
>
> Regards,
> Charith
>
> On Fri, Jan 6, 2017 at 7:44 PM, Chen Qin <qi...@gmail.com> wrote:
>
>> My understanding is HashMap doesn't work with Flink Native serialization
>> framework, though I might be wrong.
>>
>> This might worth reading
>> https://cwiki.apache.org/confluence/display/FLINK/Type+Syst
>> em,+Type+Extraction,+Serialization
>>
>> -Chen
>>
>> On Fri, Jan 6, 2017 at 6:06 PM, Charith Wickramarachchi <
>> charith.dhanushka@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I am using flink-gelly and using a custom POJO type as the
>>> VertexValue/MessageType (I am using the vertex-centric model). The POJO
>>> contains HashMap/HashSet as members. While executing the job I am getting
>>> following Log message.
>>>
>>> 17:50:53,582 INFO org.apache.flink.api.java.typeutils.TypeExtractor
>>> - No fields detected for class java.util.HashSet. Cannot be used
>>> as a PojoType. Will be handled as GenericType
>>> 17:50:53,583 INFO org.apache.flink.api.java.typeutils.
>>>
>>> TypeExtractor - class java.util.HashMap is not a valid POJO
>>> type
>>>
>>> Is there a way to resolve this issue?
>>>
>>> Thanks,
>>> Charith
>>>
>>>
>>> --
>>> Charith Dhanushka Wickramaarachchi
>>>
>>> Tel +1 213 447 4253
>>> Blog http://charith.wickramaarachchi.org/
>>> <http://charithwiki.blogspot.com/>
>>> Twitter @charithwiki <https://twitter.com/charithwiki>
>>>
>>> This communication may contain privileged or other confidential information
>>> and is intended exclusively for the addressee/s. If you are not the
>>> intended recipient/s, or believe that you may have
>>> received this communication in error, please reply to the sender indicating
>>> that fact and delete the copy you received and in addition, you should
>>> not print, copy, retransmit, disseminate, or otherwise use the
>>> information contained in this communication. Internet communications
>>> cannot be guaranteed to be timely, secure, error or virus-free. The
>>> sender does not accept liability for any errors or omissions
>>>
>>
>>
>
>
> --
> Charith Dhanushka Wickramaarachchi
>
> Tel +1 213 447 4253
> Blog http://charith.wickramaarachchi.org/
> <http://charithwiki.blogspot.com/>
> Twitter @charithwiki <https://twitter.com/charithwiki>
>
> This communication may contain privileged or other confidential information
> and is intended exclusively for the addressee/s. If you are not the
> intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating
> that fact and delete the copy you received and in addition, you should
> not print, copy, retransmit, disseminate, or otherwise use the
> information contained in this communication. Internet communications
> cannot be guaranteed to be timely, secure, error or virus-free. The
> sender does not accept liability for any errors or omissions
>
>
>
--
Charith Dhanushka Wickramaarachchi
Tel +1 213 447 4253
Blog http://charith.wickramaarachchi.org/
<http://charithwiki.blogspot.com/>
Twitter @charithwiki <https://twitter.com/charithwiki>
This communication may contain privileged or other confidential information
and is intended exclusively for the addressee/s. If you are not the
intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating
that fact and delete the copy you received and in addition, you should not
print, copy, retransmit, disseminate, or otherwise use the information
contained in this communication. Internet communications cannot be
guaranteed to be timely, secure, error or virus-free. The sender does not
accept liability for any errors or omissions
Re: HashMap/HashSet Serialization Issue
Posted by Jark Wu <wu...@alibaba-inc.com>.
Hi Charith,
The LOG message has nothing to worry about. The HashMap and HashSet members in your POJO is not a PojoType,
so they will be handled as GenericType, i.e. will be serialized/deserialized by Kryo. But your custom POJO type will still
recognized as PojoType, i.e. will be serialized/deserialized by Flink serialization framework.
In a word, your custom POJO can be serialized/deserialized correctly. There is no issue in it. :)
- Jark Wu
> 在 2017年1月7日，下午12:02，Charith Wickramarachchi <ch...@gmail.com> 写道：
>
> Thanks very much.
>
> Regards,
> Charith
>
> On Fri, Jan 6, 2017 at 7:44 PM, Chen Qin <qinnchen@gmail.com <ma...@gmail.com>> wrote:
> My understanding is HashMap doesn't work with Flink Native serialization framework, though I might be wrong.
>
> This might worth reading
> https://cwiki.apache.org/confluence/display/FLINK/Type+System,+Type+Extraction,+Serialization <https://cwiki.apache.org/confluence/display/FLINK/Type+System,+Type+Extraction,+Serialization>
>
> -Chen
>
> On Fri, Jan 6, 2017 at 6:06 PM, Charith Wickramarachchi <charith.dhanushka@gmail.com <ma...@gmail.com>> wrote:
> Hi All,
>
> I am using flink-gelly and using a custom POJO type as the VertexValue/MessageType (I am using the vertex-centric model). The POJO contains HashMap/HashSet as members. While executing the job I am getting following Log message.
>
> 17:50:53,582 INFO org.apache.flink.api.java.typeutils.TypeExtractor - No fields detected for class java.util.HashSet. Cannot be used as a PojoType. Will be handled as GenericType
> 17:50:53,583 INFO org.apache.flink.api.java.typeutils.TypeExtractor - class java.util.HashMap is not a valid POJO type
>
> Is there a way to resolve this issue?
>
> Thanks,
> Charith
>
>
> --
> Charith Dhanushka Wickramaarachchi
>
> Tel +1 213 447 4253 <>
> Blog http://charith.wickramaarachchi.org/ <http://charithwiki.blogspot.com/>
> Twitter @charithwiki <https://twitter.com/charithwiki>
>
> This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions
>
>
>
>
> --
> Charith Dhanushka Wickramaarachchi
>
> Tel +1 213 447 4253 <>
> Blog http://charith.wickramaarachchi.org/ <http://charithwiki.blogspot.com/>
> Twitter @charithwiki <https://twitter.com/charithwiki>
>
> This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions
Re: HashMap/HashSet Serialization Issue
Posted by Charith Wickramarachchi <ch...@gmail.com>.
Thanks very much.
Regards,
Charith
On Fri, Jan 6, 2017 at 7:44 PM, Chen Qin <qi...@gmail.com> wrote:
> My understanding is HashMap doesn't work with Flink Native serialization
> framework, though I might be wrong.
>
> This might worth reading
> https://cwiki.apache.org/confluence/display/FLINK/Type+
> System,+Type+Extraction,+Serialization
>
> -Chen
>
> On Fri, Jan 6, 2017 at 6:06 PM, Charith Wickramarachchi <
> charith.dhanushka@gmail.com> wrote:
>
>> Hi All,
>>
>> I am using flink-gelly and using a custom POJO type as the
>> VertexValue/MessageType (I am using the vertex-centric model). The POJO
>> contains HashMap/HashSet as members. While executing the job I am getting
>> following Log message.
>>
>> 17:50:53,582 INFO org.apache.flink.api.java.typeutils.TypeExtractor
>> - No fields detected for class java.util.HashSet. Cannot be used as
>> a PojoType. Will be handled as GenericType
>> 17:50:53,583 INFO org.apache.flink.api.java.typeutils.
>>
>> TypeExtractor - class java.util.HashMap is not a valid POJO
>> type
>>
>> Is there a way to resolve this issue?
>>
>> Thanks,
>> Charith
>>
>>
>> --
>> Charith Dhanushka Wickramaarachchi
>>
>> Tel +1 213 447 4253
>> Blog http://charith.wickramaarachchi.org/
>> <http://charithwiki.blogspot.com/>
>> Twitter @charithwiki <https://twitter.com/charithwiki>
>>
>> This communication may contain privileged or other confidential information
>> and is intended exclusively for the addressee/s. If you are not the
>> intended recipient/s, or believe that you may have
>> received this communication in error, please reply to the sender indicating
>> that fact and delete the copy you received and in addition, you should
>> not print, copy, retransmit, disseminate, or otherwise use the
>> information contained in this communication. Internet communications
>> cannot be guaranteed to be timely, secure, error or virus-free. The
>> sender does not accept liability for any errors or omissions
>>
>
>
--
Charith Dhanushka Wickramaarachchi
Tel +1 213 447 4253
Blog http://charith.wickramaarachchi.org/
<http://charithwiki.blogspot.com/>
Twitter @charithwiki <https://twitter.com/charithwiki>
This communication may contain privileged or other confidential information
and is intended exclusively for the addressee/s. If you are not the
intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating
that fact and delete the copy you received and in addition, you should not
print, copy, retransmit, disseminate, or otherwise use the information
contained in this communication. Internet communications cannot be
guaranteed to be timely, secure, error or virus-free. The sender does not
accept liability for any errors or omissions
Re: HashMap/HashSet Serialization Issue
Posted by Chen Qin <qi...@gmail.com>.
My understanding is HashMap doesn't work with Flink Native serialization
framework, though I might be wrong.
This might worth reading
https://cwiki.apache.org/confluence/display/FLINK/Type+System,+Type+Extraction,+Serialization
-Chen
On Fri, Jan 6, 2017 at 6:06 PM, Charith Wickramarachchi <
charith.dhanushka@gmail.com> wrote:
> Hi All,
>
> I am using flink-gelly and using a custom POJO type as the
> VertexValue/MessageType (I am using the vertex-centric model). The POJO
> contains HashMap/HashSet as members. While executing the job I am getting
> following Log message.
>
> 17:50:53,582 INFO org.apache.flink.api.java.typeutils.TypeExtractor
> - No fields detected for class java.util.HashSet. Cannot be used as
> a PojoType. Will be handled as GenericType
> 17:50:53,583 INFO org.apache.flink.api.java.typeutils.
>
> TypeExtractor - class java.util.HashMap is not a valid POJO
> type
>
> Is there a way to resolve this issue?
>
> Thanks,
> Charith
>
>
> --
> Charith Dhanushka Wickramaarachchi
>
> Tel +1 213 447 4253
> Blog http://charith.wickramaarachchi.org/
> <http://charithwiki.blogspot.com/>
> Twitter @charithwiki <https://twitter.com/charithwiki>
>
> This communication may contain privileged or other confidential information
> and is intended exclusively for the addressee/s. If you are not the
> intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating
> that fact and delete the copy you received and in addition, you should
> not print, copy, retransmit, disseminate, or otherwise use the
> information contained in this communication. Internet communications
> cannot be guaranteed to be timely, secure, error or virus-free. The
> sender does not accept liability for any errors or omissions
>