Scaling Higher than 10k Nodes

[Joey Tran <jo...@schrodinger.com>]

Hi, I was looking at Apache Beam/Flink for some of our data processing
needs, but when reading about the resource managers
(YARN/mesos/Kubernetes), it seems like they all top out at around 10k
nodes. What are recommended solutions for scaling higher than this?

Thanks in advance,
Joey

Re: Scaling Higher than 10k Nodes

[Yuval Itzchakov <yu...@gmail.com>]

Hi Joey,

We are currently running around 2000+ small Flink clusters on top of k8s,
currently at around ~ 100 nodes. Do you see yourself scaling to 10k nodes,
given that each node can run a significant amount of Flink jobs inside of
it?

On Thu, Mar 4, 2021 at 10:51 AM Piotr Nowojski <pn...@apache.org> wrote:

> Maybe a stupid question Joey, but if the problem is in the resource
> managers, haven't you tried running standalone Flink clusters without any
> resource manager? Probably you would still hit the JobManager problems that
> Xintong mentioned, but those problems we can help addressing.
>
> Piotrek
>
> czw., 4 mar 2021 o 09:30 Xintong Song <to...@gmail.com> napisał(a):
>
>> Hi Joey,
>>
>> Quick question: by *nodes*, do you mean Flink task manager processes, or
>> physical/virtual machines (like ecs, yarn NM)?
>>
>> In our production, we run flink workloads on several Yarn/Kubernetes
>> clusters, where each cluster typically has 2k~5k machines. Most Flink
>> workloads are deployed in single-job mode, giving us thousands (sometimes
>> more than 10k) of flink instances concurrently running on each cluster. In
>> this way, the scale of each flink instance is usually not extremely large
>> (less than 1000 TMs), and we rely on the power of Yarn/Kubernetes to deal
>> with the large number of instances.
>>
>> There're also cases that a single flink job is extremely large. We had a
>> batch workload from last year's double-11 event, with 8k max per-stage
>> parallelism and up to 30k task managers running at the same time. At that
>> scale, we run into problems with the single-point JobManager process, such
>> as tremendous memory consumption, buzy rpc main thread, etc. To make that
>> case work, we did many optimizations on our internal flink version, which
>> we are trying to contribute to the community. See FLINK-21110 [1] for the
>> details.
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-21110
>>
>> On Thu, Mar 4, 2021 at 3:39 PM Piotr Nowojski <pn...@apache.org>
>> wrote:
>>
>>> Hi Joey,
>>>
>>> Sorry for not responding to your question sooner. As you can imagine
>>> there are not many users running Flink at such scale. As far as I know,
>>> Alibaba is running the largest/one of the largest clusters, I'm asking for
>>> someone who is familiar with those deployments to take a look at this
>>> conversation. I hope someone will respond here soon :)
>>>
>>> Best,
>>> Piotrek
>>>
>>> pon., 1 mar 2021 o 14:43 Joey Tran <jo...@schrodinger.com>
>>> napisał(a):
>>>
>>>> Hi, I was looking at Apache Beam/Flink for some of our data processing
>>>> needs, but when reading about the resource managers
>>>> (YARN/mesos/Kubernetes), it seems like they all top out at around 10k
>>>> nodes. What are recommended solutions for scaling higher than this?
>>>>
>>>> Thanks in advance,
>>>> Joey
>>>>
>>>

-- 
Best Regards,
Yuval Itzchakov.

Re: Scaling Higher than 10k Nodes

[Piotr Nowojski <pn...@apache.org>]

Maybe a stupid question Joey, but if the problem is in the resource
managers, haven't you tried running standalone Flink clusters without any
resource manager? Probably you would still hit the JobManager problems that
Xintong mentioned, but those problems we can help addressing.

Piotrek

czw., 4 mar 2021 o 09:30 Xintong Song <to...@gmail.com> napisał(a):

> Hi Joey,
>
> Quick question: by *nodes*, do you mean Flink task manager processes, or
> physical/virtual machines (like ecs, yarn NM)?
>
> In our production, we run flink workloads on several Yarn/Kubernetes
> clusters, where each cluster typically has 2k~5k machines. Most Flink
> workloads are deployed in single-job mode, giving us thousands (sometimes
> more than 10k) of flink instances concurrently running on each cluster. In
> this way, the scale of each flink instance is usually not extremely large
> (less than 1000 TMs), and we rely on the power of Yarn/Kubernetes to deal
> with the large number of instances.
>
> There're also cases that a single flink job is extremely large. We had a
> batch workload from last year's double-11 event, with 8k max per-stage
> parallelism and up to 30k task managers running at the same time. At that
> scale, we run into problems with the single-point JobManager process, such
> as tremendous memory consumption, buzy rpc main thread, etc. To make that
> case work, we did many optimizations on our internal flink version, which
> we are trying to contribute to the community. See FLINK-21110 [1] for the
> details.
>
> Thank you~
>
> Xintong Song
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-21110
>
> On Thu, Mar 4, 2021 at 3:39 PM Piotr Nowojski <pn...@apache.org>
> wrote:
>
>> Hi Joey,
>>
>> Sorry for not responding to your question sooner. As you can imagine
>> there are not many users running Flink at such scale. As far as I know,
>> Alibaba is running the largest/one of the largest clusters, I'm asking for
>> someone who is familiar with those deployments to take a look at this
>> conversation. I hope someone will respond here soon :)
>>
>> Best,
>> Piotrek
>>
>> pon., 1 mar 2021 o 14:43 Joey Tran <jo...@schrodinger.com>
>> napisał(a):
>>
>>> Hi, I was looking at Apache Beam/Flink for some of our data processing
>>> needs, but when reading about the resource managers
>>> (YARN/mesos/Kubernetes), it seems like they all top out at around 10k
>>> nodes. What are recommended solutions for scaling higher than this?
>>>
>>> Thanks in advance,
>>> Joey
>>>
>>

Re: Scaling Higher than 10k Nodes

[Xintong Song <to...@gmail.com>]

Hi Joey,

Quick question: by *nodes*, do you mean Flink task manager processes, or
physical/virtual machines (like ecs, yarn NM)?

In our production, we run flink workloads on several Yarn/Kubernetes
clusters, where each cluster typically has 2k~5k machines. Most Flink
workloads are deployed in single-job mode, giving us thousands (sometimes
more than 10k) of flink instances concurrently running on each cluster. In
this way, the scale of each flink instance is usually not extremely large
(less than 1000 TMs), and we rely on the power of Yarn/Kubernetes to deal
with the large number of instances.

There're also cases that a single flink job is extremely large. We had a
batch workload from last year's double-11 event, with 8k max per-stage
parallelism and up to 30k task managers running at the same time. At that
scale, we run into problems with the single-point JobManager process, such
as tremendous memory consumption, buzy rpc main thread, etc. To make that
case work, we did many optimizations on our internal flink version, which
we are trying to contribute to the community. See FLINK-21110 [1] for the
details.

Thank you~

Xintong Song


[1] https://issues.apache.org/jira/browse/FLINK-21110

On Thu, Mar 4, 2021 at 3:39 PM Piotr Nowojski <pn...@apache.org> wrote:

> Hi Joey,
>
> Sorry for not responding to your question sooner. As you can imagine there
> are not many users running Flink at such scale. As far as I know, Alibaba
> is running the largest/one of the largest clusters, I'm asking for someone
> who is familiar with those deployments to take a look at this conversation.
> I hope someone will respond here soon :)
>
> Best,
> Piotrek
>
> pon., 1 mar 2021 o 14:43 Joey Tran <jo...@schrodinger.com> napisał(a):
>
>> Hi, I was looking at Apache Beam/Flink for some of our data processing
>> needs, but when reading about the resource managers
>> (YARN/mesos/Kubernetes), it seems like they all top out at around 10k
>> nodes. What are recommended solutions for scaling higher than this?
>>
>> Thanks in advance,
>> Joey
>>
>

Re: Scaling Higher than 10k Nodes

[Piotr Nowojski <pn...@apache.org>]

Hi Joey,

Sorry for not responding to your question sooner. As you can imagine there
are not many users running Flink at such scale. As far as I know, Alibaba
is running the largest/one of the largest clusters, I'm asking for someone
who is familiar with those deployments to take a look at this conversation.
I hope someone will respond here soon :)

Best,
Piotrek

pon., 1 mar 2021 o 14:43 Joey Tran <jo...@schrodinger.com> napisał(a):

> Hi, I was looking at Apache Beam/Flink for some of our data processing
> needs, but when reading about the resource managers
> (YARN/mesos/Kubernetes), it seems like they all top out at around 10k
> nodes. What are recommended solutions for scaling higher than this?
>
> Thanks in advance,
> Joey
>