You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Pankaj Chand <pa...@gmail.com> on 2020/03/16 08:03:16 UTC
Flink on Kubernetes Vs Flink Natively on Kubernetes
Hi all,
I want to run Flink, Spark and other processing engines on a single
Kubernetes cluster.
From the Flink documentation, I did not understand the difference between:
(1) Running Flink on Kubernetes, Versus (2) Running Flink natively on
Kubernetes.
Could someone please explain the difference between the two, and when would
you use which option?
Thank you,
Pankaj
Re: Flink on Kubernetes Vs Flink Natively on Kubernetes
Posted by Pankaj Chand <pa...@gmail.com>.
Thank you, Yang and Xintong!
Best,
Pankaj
On Mon, Mar 16, 2020, 9:27 PM Yang Wang <da...@gmail.com> wrote:
> Hi Pankaj,
>
> Just like Xintong has said, the biggest difference of Flink on Kubernetes
> and native
> integration is dynamic resource allocation. Since the latter has en
> embedded K8s
> client and will communicate with K8s Api server directly to
> allocate/release JM/TM
> pods.
>
> Both for the two ways to run Flink on K8s, you do not need to reserve the
> whole
> cluster for Flink. Flink could run with other workloads(e.g. Spark,
> tensorflow, etc.).
> The K8s cluster could guarantee the isolation.
>
>
> Best,
> Yang
>
> Pankaj Chand <pa...@gmail.com> 于2020年3月16日周一 下午5:51写道：
>
>> Hi Xintong,
>>
>> Thank you for the explanation!
>>
>> If I run Flink "natively" on Kubernetes, will I also be able to run Spark
>> on the same Kubernetes cluster, or will it make the Kubernetes cluster be
>> reserved for Flink only?
>>
>> Thank you!
>>
>> Pankaj
>>
>> On Mon, Mar 16, 2020 at 5:41 AM Xintong Song <to...@gmail.com>
>> wrote:
>>
>>> Forgot to mention that "running Flink natively on Kubernetes" is newly
>>> introduced and is only available for Flink 1.10 and above.
>>>
>>>
>>> Thank you~
>>>
>>> Xintong Song
>>>
>>>
>>>
>>> On Mon, Mar 16, 2020 at 5:40 PM Xintong Song <to...@gmail.com>
>>> wrote:
>>>
>>>> Hi Pankaj,
>>>>
>>>> "Running Flink on Kubernetes" refers to the old way that basically
>>>> deploys a Flink standalone cluster on Kubernetes. We leverage scripts to
>>>> run Flink Master and TaskManager processes inside Kubernetes container. In
>>>> this way, Flink is not ware of whether it's running in containers or
>>>> directly on physical machines, and will not interact with the Kubernetes
>>>> Master. Flink Master reactively accept all registered TaskManagers, whose
>>>> number is decided by the Kubernetes replica.
>>>>
>>>> "Running Flink natively on Kubernetes" refers deploy Flink as a
>>>> Kubernetes Job. Flink Master will interact with Kubernetes Master, and
>>>> actively requests for pods/containers, like on Yarn/Mesos.
>>>>
>>>> Thank you~
>>>>
>>>> Xintong Song
>>>>
>>>>
>>>>
>>>> On Mon, Mar 16, 2020 at 4:03 PM Pankaj Chand <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I want to run Flink, Spark and other processing engines on a single
>>>>> Kubernetes cluster.
>>>>>
>>>>> From the Flink documentation, I did not understand the difference
>>>>> between:
>>>>> (1) Running Flink on Kubernetes, Versus (2) Running Flink natively on
>>>>> Kubernetes.
>>>>>
>>>>> Could someone please explain the difference between the two, and when
>>>>> would you use which option?
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Pankaj
>>>>>
>>>>
Re: Flink on Kubernetes Vs Flink Natively on Kubernetes
Posted by Yang Wang <da...@gmail.com>.
Hi Pankaj,
Just like Xintong has said, the biggest difference of Flink on Kubernetes
and native
integration is dynamic resource allocation. Since the latter has en
embedded K8s
client and will communicate with K8s Api server directly to
allocate/release JM/TM
pods.
Both for the two ways to run Flink on K8s, you do not need to reserve the
whole
cluster for Flink. Flink could run with other workloads(e.g. Spark,
tensorflow, etc.).
The K8s cluster could guarantee the isolation.
Best,
Yang
Pankaj Chand <pa...@gmail.com> 于2020年3月16日周一 下午5:51写道：
> Hi Xintong,
>
> Thank you for the explanation!
>
> If I run Flink "natively" on Kubernetes, will I also be able to run Spark
> on the same Kubernetes cluster, or will it make the Kubernetes cluster be
> reserved for Flink only?
>
> Thank you!
>
> Pankaj
>
> On Mon, Mar 16, 2020 at 5:41 AM Xintong Song <to...@gmail.com>
> wrote:
>
>> Forgot to mention that "running Flink natively on Kubernetes" is newly
>> introduced and is only available for Flink 1.10 and above.
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Mon, Mar 16, 2020 at 5:40 PM Xintong Song <to...@gmail.com>
>> wrote:
>>
>>> Hi Pankaj,
>>>
>>> "Running Flink on Kubernetes" refers to the old way that basically
>>> deploys a Flink standalone cluster on Kubernetes. We leverage scripts to
>>> run Flink Master and TaskManager processes inside Kubernetes container. In
>>> this way, Flink is not ware of whether it's running in containers or
>>> directly on physical machines, and will not interact with the Kubernetes
>>> Master. Flink Master reactively accept all registered TaskManagers, whose
>>> number is decided by the Kubernetes replica.
>>>
>>> "Running Flink natively on Kubernetes" refers deploy Flink as a
>>> Kubernetes Job. Flink Master will interact with Kubernetes Master, and
>>> actively requests for pods/containers, like on Yarn/Mesos.
>>>
>>> Thank you~
>>>
>>> Xintong Song
>>>
>>>
>>>
>>> On Mon, Mar 16, 2020 at 4:03 PM Pankaj Chand <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I want to run Flink, Spark and other processing engines on a single
>>>> Kubernetes cluster.
>>>>
>>>> From the Flink documentation, I did not understand the difference
>>>> between:
>>>> (1) Running Flink on Kubernetes, Versus (2) Running Flink natively on
>>>> Kubernetes.
>>>>
>>>> Could someone please explain the difference between the two, and when
>>>> would you use which option?
>>>>
>>>> Thank you,
>>>>
>>>> Pankaj
>>>>
>>>
Re: Flink on Kubernetes Vs Flink Natively on Kubernetes
Posted by Pankaj Chand <pa...@gmail.com>.
Hi Xintong,
Thank you for the explanation!
If I run Flink "natively" on Kubernetes, will I also be able to run Spark
on the same Kubernetes cluster, or will it make the Kubernetes cluster be
reserved for Flink only?
Thank you!
Pankaj
On Mon, Mar 16, 2020 at 5:41 AM Xintong Song <to...@gmail.com> wrote:
> Forgot to mention that "running Flink natively on Kubernetes" is newly
> introduced and is only available for Flink 1.10 and above.
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Mar 16, 2020 at 5:40 PM Xintong Song <to...@gmail.com>
> wrote:
>
>> Hi Pankaj,
>>
>> "Running Flink on Kubernetes" refers to the old way that basically
>> deploys a Flink standalone cluster on Kubernetes. We leverage scripts to
>> run Flink Master and TaskManager processes inside Kubernetes container. In
>> this way, Flink is not ware of whether it's running in containers or
>> directly on physical machines, and will not interact with the Kubernetes
>> Master. Flink Master reactively accept all registered TaskManagers, whose
>> number is decided by the Kubernetes replica.
>>
>> "Running Flink natively on Kubernetes" refers deploy Flink as a
>> Kubernetes Job. Flink Master will interact with Kubernetes Master, and
>> actively requests for pods/containers, like on Yarn/Mesos.
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Mon, Mar 16, 2020 at 4:03 PM Pankaj Chand <pa...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I want to run Flink, Spark and other processing engines on a single
>>> Kubernetes cluster.
>>>
>>> From the Flink documentation, I did not understand the difference
>>> between:
>>> (1) Running Flink on Kubernetes, Versus (2) Running Flink natively on
>>> Kubernetes.
>>>
>>> Could someone please explain the difference between the two, and when
>>> would you use which option?
>>>
>>> Thank you,
>>>
>>> Pankaj
>>>
>>
Re: Flink on Kubernetes Vs Flink Natively on Kubernetes
Posted by Xintong Song <to...@gmail.com>.
Forgot to mention that "running Flink natively on Kubernetes" is newly
introduced and is only available for Flink 1.10 and above.
Thank you~
Xintong Song
On Mon, Mar 16, 2020 at 5:40 PM Xintong Song <to...@gmail.com> wrote:
> Hi Pankaj,
>
> "Running Flink on Kubernetes" refers to the old way that basically deploys
> a Flink standalone cluster on Kubernetes. We leverage scripts to run Flink
> Master and TaskManager processes inside Kubernetes container. In this way,
> Flink is not ware of whether it's running in containers or directly on
> physical machines, and will not interact with the Kubernetes Master. Flink
> Master reactively accept all registered TaskManagers, whose number is
> decided by the Kubernetes replica.
>
> "Running Flink natively on Kubernetes" refers deploy Flink as a Kubernetes
> Job. Flink Master will interact with Kubernetes Master, and actively
> requests for pods/containers, like on Yarn/Mesos.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Mar 16, 2020 at 4:03 PM Pankaj Chand <pa...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I want to run Flink, Spark and other processing engines on a single
>> Kubernetes cluster.
>>
>> From the Flink documentation, I did not understand the difference between:
>> (1) Running Flink on Kubernetes, Versus (2) Running Flink natively on
>> Kubernetes.
>>
>> Could someone please explain the difference between the two, and when
>> would you use which option?
>>
>> Thank you,
>>
>> Pankaj
>>
>
Re: Flink on Kubernetes Vs Flink Natively on Kubernetes
Posted by Xintong Song <to...@gmail.com>.
Hi Pankaj,
"Running Flink on Kubernetes" refers to the old way that basically deploys
a Flink standalone cluster on Kubernetes. We leverage scripts to run Flink
Master and TaskManager processes inside Kubernetes container. In this way,
Flink is not ware of whether it's running in containers or directly on
physical machines, and will not interact with the Kubernetes Master. Flink
Master reactively accept all registered TaskManagers, whose number is
decided by the Kubernetes replica.
"Running Flink natively on Kubernetes" refers deploy Flink as a Kubernetes
Job. Flink Master will interact with Kubernetes Master, and actively
requests for pods/containers, like on Yarn/Mesos.
Thank you~
Xintong Song
On Mon, Mar 16, 2020 at 4:03 PM Pankaj Chand <pa...@gmail.com>
wrote:
> Hi all,
>
> I want to run Flink, Spark and other processing engines on a single
> Kubernetes cluster.
>
> From the Flink documentation, I did not understand the difference between:
> (1) Running Flink on Kubernetes, Versus (2) Running Flink natively on
> Kubernetes.
>
> Could someone please explain the difference between the two, and when
> would you use which option?
>
> Thank you,
>
> Pankaj
>