You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Yuta Morisawa <yu...@kddi-research.jp> on 2018/07/11 07:24:12 UTC
Some question about document
Hi all
Now, I'm reading Flink document and I have some points to feel difficult
to get an idea.
I'd appreciate if you tell it me.
1,TypeInformation
I understand TypeInformation is used for selecting relevant serializer
and comparator.
But, the ducument doesn't specify if it has another way to be used.
So, what I want to know is that what kinds of process gets benefit
from TypeInformation other than serializer and comparator.
2, Managed Memory
The word "Managed memory" is appeared several time in the document
but I can't find any detail description.
This is the only document I found
(https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)
If anyone has document that explains managed memory, please let me know.
3, Serializer
What do the words in the document "serializers we ship with Flink"
mean? I know Flink uses avro for POJOs, is it the same thing?
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html
Regards,
Yuta
--
--------------------------------------------
Business Vision ："Challenge for the future"
---------------------------------------------
〒356-8502
埼玉県ふじみ野市大原2丁目1番15号
株式会社 ＫＤＤＩ総合研究所（KDDI Research, Inc.）
コネクティッドカー1G
森澤 雄太
mail yu-morisawa@kddi-research.jp
tel 070-3871-8883
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
この電子メールおよび添付書類は、名宛人のための
特別な秘密情報を含んでおります。
そのため、名宛人以外の方による利用は認められて
おりません。
名宛人以外の方による通信内容公表、複写、転用等
は厳禁であり、違法となることがあります。
万が一、何らかの誤りによりこの電子メールを名宛
人以外の方が受信された場合は、お手数でも、直ち
に発信人にお知らせ頂くと同時に、当メールを削除
下さいますようお願い申し上げます。
Re: Some question about document
Posted by Yuta Morisawa <yu...@kddi-research.jp>.
Hi yang
Thank you for your comment.
I read document and got an idea.
Flink uses custom serializers on known types, and the fallback is kryo.
The other arbitary objects is serialized by kryo.
On 2018/07/12 12:14, vino yang wrote:
> Hi Yuta,
>
> It seems Chesnay is right. The "fallback" in flink's documentation is in
> terms of the types flink supported. But for all the other arbitrary
> types kryo is the first choice.
>
> 2018-07-12 9:55 GMT+08:00 Yuta Morisawa <yu-morisawa@kddi-research.jp
> <ma...@kddi-research.jp>>:
>
> Thank you for your answer.
>
> > For POJOs Flink has a custom serializer. For arbitrary objects we use
> > kryo, and can use Avro as a fallback.
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types>
>
> It may be the reverse.
> Kryo is for fallback, right?
>
>
> On 2018/07/11 19:00, Chesnay Schepler wrote:
>
> 1) TypeInformation are used to create serializers, comparators
> and to verify correctness of certain operations (like
> projections on tuple datasets).
>
> 2) see
> https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html
> <https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html>
>
> 3) Flink comes with a number of serializers for varying types as
> outlined here
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class>>.
> For POJOs Flink has a custom serializer. For arbitrary objects
> we use kryo, and can use Avro as a fallback.
>
> On 11.07.2018 09:24, Yuta Morisawa wrote:
>
> Hi all
>
> Now, I'm reading Flink document and I have some points to
> feel difficult to get an idea.
> I'd appreciate if you tell it me.
>
> 1,TypeInformation
> I understand TypeInformation is used for selecting
> relevant serializer and comparator.
> But, the ducument doesn't specify if it has another way to
> be used.
>
> So, what I want to know is that what kinds of process gets
> benefit from TypeInformation other than serializer and
> comparator.
>
> 2, Managed Memory
> The word "Managed memory" is appeared several time in the
> document but I can't find any detail description.
> This is the only document I found
> (https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi
> <https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi>)
>
> If anyone has document that explains managed memory,
> please let me know.
>
> 3, Serializer
> What do the words in the document "serializers we ship
> with Flink" mean? I know Flink uses avro for POJOs, is it
> the same thing?
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html>
>
>
>
> Regards,
> Yuta
>
>
>
>
Re: Some question about document
Posted by vino yang <ya...@gmail.com>.
Hi Yuta,
It seems Chesnay is right. The "fallback" in flink's documentation is in
terms of the types flink supported. But for all the other arbitrary types
kryo is the first choice.
2018-07-12 9:55 GMT+08:00 Yuta Morisawa <yu...@kddi-research.jp>:
> Thank you for your answer.
>
> > For POJOs Flink has a custom serializer. For arbitrary objects we use
> > kryo, and can use Avro as a fallback.
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/
> dev/types_serialization.html#serialization-of-pojo-types
>
> It may be the reverse.
> Kryo is for fallback, right?
>
>
> On 2018/07/11 19:00, Chesnay Schepler wrote:
>
>> 1) TypeInformation are used to create serializers, comparators and to
>> verify correctness of certain operations (like projections on tuple
>> datasets).
>>
>> 2) see https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-
>> and-Bytes.html
>>
>> 3) Flink comes with a number of serializers for varying types as outlined
>> here <https://ci.apache.org/projects/flink/flink-docs-release-1.
>> 5/dev/types_serialization.html#flinks-typeinformation-class>.
>> For POJOs Flink has a custom serializer. For arbitrary objects we use
>> kryo, and can use Avro as a fallback.
>>
>> On 11.07.2018 09:24, Yuta Morisawa wrote:
>>
>>> Hi all
>>>
>>> Now, I'm reading Flink document and I have some points to feel difficult
>>> to get an idea.
>>> I'd appreciate if you tell it me.
>>>
>>> 1,TypeInformation
>>> I understand TypeInformation is used for selecting relevant serializer
>>> and comparator.
>>> But, the ducument doesn't specify if it has another way to be used.
>>>
>>> So, what I want to know is that what kinds of process gets benefit from
>>> TypeInformation other than serializer and comparator.
>>>
>>> 2, Managed Memory
>>> The word "Managed memory" is appeared several time in the document but
>>> I can't find any detail description.
>>> This is the only document I found (https://www.slideshare.net/sb
>>> altagi/overview-of-apacheflinkbyslimbaltagi)
>>>
>>> If anyone has document that explains managed memory, please let me know.
>>>
>>> 3, Serializer
>>> What do the words in the document "serializers we ship with Flink"
>>> mean? I know Flink uses avro for POJOs, is it the same thing?
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/
>>> dev/types_serialization.html
>>>
>>>
>>> Regards,
>>> Yuta
>>>
>>>
>>
>
Re: Some question about document
Posted by Yuta Morisawa <yu...@kddi-research.jp>.
Thank you for your answer.
> For POJOs Flink has a custom serializer. For arbitrary objects we use
> kryo, and can use Avro as a fallback.
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types
It may be the reverse.
Kryo is for fallback, right?
On 2018/07/11 19:00, Chesnay Schepler wrote:
> 1) TypeInformation are used to create serializers, comparators and to
> verify correctness of certain operations (like projections on tuple
> datasets).
>
> 2) see
> https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html
>
> 3) Flink comes with a number of serializers for varying types as
> outlined here
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class>.
> For POJOs Flink has a custom serializer. For arbitrary objects we use
> kryo, and can use Avro as a fallback.
>
> On 11.07.2018 09:24, Yuta Morisawa wrote:
>> Hi all
>>
>> Now, I'm reading Flink document and I have some points to feel
>> difficult to get an idea.
>> I'd appreciate if you tell it me.
>>
>> 1,TypeInformation
>> I understand TypeInformation is used for selecting relevant
>> serializer and comparator.
>> But, the ducument doesn't specify if it has another way to be used.
>>
>> So, what I want to know is that what kinds of process gets benefit
>> from TypeInformation other than serializer and comparator.
>>
>> 2, Managed Memory
>> The word "Managed memory" is appeared several time in the document
>> but I can't find any detail description.
>> This is the only document I found
>> (https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)
>>
>> If anyone has document that explains managed memory, please let me know.
>>
>> 3, Serializer
>> What do the words in the document "serializers we ship with Flink"
>> mean? I know Flink uses avro for POJOs, is it the same thing?
>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html
>>
>>
>>
>> Regards,
>> Yuta
>>
>
Re: Some question about document
Posted by Chesnay Schepler <ch...@apache.org>.
1) TypeInformation are used to create serializers, comparators and to
verify correctness of certain operations (like projections on tuple
datasets).
2) see
https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html
3) Flink comes with a number of serializers for varying types as
outlined here
<https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class>.
For POJOs Flink has a custom serializer. For arbitrary objects we use
kryo, and can use Avro as a fallback.
On 11.07.2018 09:24, Yuta Morisawa wrote:
> Hi all
>
> Now, I'm reading Flink document and I have some points to feel
> difficult to get an idea.
> I'd appreciate if you tell it me.
>
> 1,TypeInformation
> I understand TypeInformation is used for selecting relevant
> serializer and comparator.
> But, the ducument doesn't specify if it has another way to be used.
>
> So, what I want to know is that what kinds of process gets benefit
> from TypeInformation other than serializer and comparator.
>
> 2, Managed Memory
> The word "Managed memory" is appeared several time in the document
> but I can't find any detail description.
> This is the only document I found
> (https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi)
>
> If anyone has document that explains managed memory, please let me know.
>
> 3, Serializer
> What do the words in the document "serializers we ship with Flink"
> mean? I know Flink uses avro for POJOs, is it the same thing?
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html
>
>
>
> Regards,
> Yuta
>