Size of state for any known production use case

[RKandoji <rk...@gmail.com>]

Hi Team,

I've done a POC using Flink and planning to give a presentation about my
learnings and share the benefits of using Flink.

I understand that companies are using Flink to handle Tera Bytes of state,
but it would be great if you could point me to any reference of a company
using Flink production for a known amount of state. Or any other related
links where I can get these details?

Basically I want to provide the known maximum limit of state that can be
stored. This is needed because my use case requires performing stream joins
on unbounded data (although data is unbounded, its not going to be super
huge like 10TB)


Thanks,
Reva

Re: Size of state for any known production use case

[RKandoji <rk...@gmail.com>]

hey Marta,

Thank you!
This is going to be pretty useful, let me go over these and get back to you
if I have any questions.

RK



On Thu, Feb 13, 2020 at 6:21 AM Marta Paes Moreira <ma...@ververica.com>
wrote:

> Hi, Reva.
>
> If you are looking for the maximum known state size, I believe Alibaba is
> using Flink at the largest scale in production [1].
>
> There are also other examples of variable scale scattered across Flink
> Forward talks [2]. In particular, this Netflix talk [3] should be
> interesting to you.
>
> Marta
>
> [1]
> https://www.itnextsummit.com/wp-content/uploads/2019/11/Stephan_Ewen_Stream_Processing_Beyond_Streaming.pdf (Slide
> 3)
> [2] https://www.youtube.com/channel/UCY8_lgiZLZErZPF47a2hXMA/videos
> [3] https://www.youtube.com/watch?v=2C44mUPlx5o
>
> On Wed, Feb 12, 2020 at 10:42 PM RKandoji <rk...@gmail.com> wrote:
>
>> Hi Team,
>>
>> I've done a POC using Flink and planning to give a presentation about my
>> learnings and share the benefits of using Flink.
>>
>> I understand that companies are using Flink to handle Tera Bytes of
>> state, but it would be great if you could point me to any reference of a
>> company using Flink production for a known amount of state. Or any other
>> related links where I can get these details?
>>
>> Basically I want to provide the known maximum limit of state that can be
>> stored. This is needed because my use case requires performing stream joins
>> on unbounded data (although data is unbounded, its not going to be super
>> huge like 10TB)
>>
>>
>> Thanks,
>> Reva
>>
>

Re: Size of state for any known production use case

[Marta Paes Moreira <ma...@ververica.com>]

Hi, Reva.

If you are looking for the maximum known state size, I believe Alibaba is
using Flink at the largest scale in production [1].

There are also other examples of variable scale scattered across Flink
Forward talks [2]. In particular, this Netflix talk [3] should be
interesting to you.

Marta

[1]
https://www.itnextsummit.com/wp-content/uploads/2019/11/Stephan_Ewen_Stream_Processing_Beyond_Streaming.pdf
(Slide
3)
[2] https://www.youtube.com/channel/UCY8_lgiZLZErZPF47a2hXMA/videos
[3] https://www.youtube.com/watch?v=2C44mUPlx5o

On Wed, Feb 12, 2020 at 10:42 PM RKandoji <rk...@gmail.com> wrote:

> Hi Team,
>
> I've done a POC using Flink and planning to give a presentation about my
> learnings and share the benefits of using Flink.
>
> I understand that companies are using Flink to handle Tera Bytes of state,
> but it would be great if you could point me to any reference of a company
> using Flink production for a known amount of state. Or any other related
> links where I can get these details?
>
> Basically I want to provide the known maximum limit of state that can be
> stored. This is needed because my use case requires performing stream joins
> on unbounded data (although data is unbounded, its not going to be super
> huge like 10TB)
>
>
> Thanks,
> Reva
>