You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Navneeth Krishnan <re...@gmail.com> on 2020/04/15 01:57:05 UTC
Flink
Hi All,
I'm very new to EKS and trying to deploy a flink job in cluster mode. Are
there any good documentations on what are the steps to deploy on EKS?
From my understanding, with flink 1.10 running it on EKS will automatically
scale up and down with kubernetes integration based on the load. Is this
correct? Do I have to do enable some configs to support this feature?
How to use the lyft k8s operator when deploying on EKS?
Thanks a lot, appreciate all the help.
Re: Flink
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Navneeth,
I think there might be some misunderstanding here. Let me try to clarify.
1) The so-called native Kubernetes support [1], which was added as an
experimental feature in Flink 1.10, is not used by Ververica Platform CE
nor by the Lyft K8s Operator as far as I am aware of.
2) The native Kubernetes support only supports session clusters [1] so far.
Support for Job clusters work in progress for Flink 1.11. In case of a
session cluster, Flink will request additional resources from Kubernetes
whenever necessary to deploy a Flink Job upon submission. So, there is
automatic scaling of the *cluster, *but no automatic scaling of the *jobs*.
3) There are multiple efforts within the Flink ecosystem working on auto
scaling jobs.
I hope this helps and please reach out if you have further questions.
Best,
Konstantin
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/concepts/glossary.html#flink-session-cluster
On Thu, Apr 23, 2020 at 7:11 AM Navneeth Krishnan <re...@gmail.com>
wrote:
> Thanks a lot Timo. I will take a look at it. But does flink automatically
> scale up and down at this point with native integration?
>
> Thanks
>
> On Tue, Apr 14, 2020 at 11:27 PM Timo Walther <tw...@apache.org> wrote:
>
>> Hi Navneeth,
>>
>> it might be also worth to look into Ververica Plaform for this. The
>> community edition was published recently is free of charge. It provides
>> first class K8s support [1].
>>
>> There is also a tutorial how to deploy it on EKS [2] (not the most
>> recent one through).
>>
>> Regards,
>> Timo
>>
>> [1]
>>
>> https://www.ververica.com/blog/announcing-ververica-platform-community-edition?utm_campaign=Ververica%20Platform%20-%20Community%20Edition&utm_content=123140986&utm_medium=social&utm_source=twitter&hss_channel=tw-2581958070
>> [2]
>>
>> https://www.ververica.com/blog/how-to-get-started-with-data-artisans-platform-on-aws-eks
>>
>>
>>
>> On 15.04.20 03:57, Navneeth Krishnan wrote:
>> > Hi All,
>> >
>> > I'm very new to EKS and trying to deploy a flink job in cluster mode.
>> > Are there any good documentations on what are the steps to deploy on
>> EKS?
>> >
>> > From my understanding, with flink 1.10 running it on EKS will
>> > automatically scale up and down with kubernetes integration based on
>> the
>> > load. Is this correct? Do I have to do enable some configs to support
>> > this feature?
>> >
>> > How to use the lyft k8s operator when deploying on EKS?
>> >
>> > Thanks a lot, appreciate all the help.
>> >
>> >
>> >
>> >
>>
>>
--
Konstantin Knauf | Head of Product
+49 160 91394525
Follow us @VervericaData Ververica <https://www.ververica.com/>
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Tony) Cheng
Re: Flink
Posted by Navneeth Krishnan <re...@gmail.com>.
Thanks a lot Timo. I will take a look at it. But does flink automatically
scale up and down at this point with native integration?
Thanks
On Tue, Apr 14, 2020 at 11:27 PM Timo Walther <tw...@apache.org> wrote:
> Hi Navneeth,
>
> it might be also worth to look into Ververica Plaform for this. The
> community edition was published recently is free of charge. It provides
> first class K8s support [1].
>
> There is also a tutorial how to deploy it on EKS [2] (not the most
> recent one through).
>
> Regards,
> Timo
>
> [1]
>
> https://www.ververica.com/blog/announcing-ververica-platform-community-edition?utm_campaign=Ververica%20Platform%20-%20Community%20Edition&utm_content=123140986&utm_medium=social&utm_source=twitter&hss_channel=tw-2581958070
> [2]
>
> https://www.ververica.com/blog/how-to-get-started-with-data-artisans-platform-on-aws-eks
>
>
>
> On 15.04.20 03:57, Navneeth Krishnan wrote:
> > Hi All,
> >
> > I'm very new to EKS and trying to deploy a flink job in cluster mode.
> > Are there any good documentations on what are the steps to deploy on EKS?
> >
> > From my understanding, with flink 1.10 running it on EKS will
> > automatically scale up and down with kubernetes integration based on the
> > load. Is this correct? Do I have to do enable some configs to support
> > this feature?
> >
> > How to use the lyft k8s operator when deploying on EKS?
> >
> > Thanks a lot, appreciate all the help.
> >
> >
> >
> >
>
>
Re: Flink
Posted by Timo Walther <tw...@apache.org>.
Hi Navneeth,
it might be also worth to look into Ververica Plaform for this. The
community edition was published recently is free of charge. It provides
first class K8s support [1].
There is also a tutorial how to deploy it on EKS [2] (not the most
recent one through).
Regards,
Timo
[1]
https://www.ververica.com/blog/announcing-ververica-platform-community-edition?utm_campaign=Ververica%20Platform%20-%20Community%20Edition&utm_content=123140986&utm_medium=social&utm_source=twitter&hss_channel=tw-2581958070
[2]
https://www.ververica.com/blog/how-to-get-started-with-data-artisans-platform-on-aws-eks
On 15.04.20 03:57, Navneeth Krishnan wrote:
> Hi All,
>
> I'm very new to EKS and trying to deploy a flink job in cluster mode.
> Are there any good documentations on what are the steps to deploy on EKS?
>
> From my understanding, with flink 1.10 running it on EKS will
> automatically scale up and down with kubernetes integration based on the
> load. Is this correct? Do I have to do enable some configs to support
> this feature?
>
> How to use the lyft k8s operator when deploying on EKS?
>
> Thanks a lot, appreciate all the help.
>
>
>
>