Re: AWS Kinesis Flink vs K8s

[Puneet Duggal <pu...@gmail.com>]

Hi Jeremy,

Thank you for this detailed answer and yes this surely helps.. 

Regards,
Puneet

> On 16-Feb-2022, at 9:21 PM, Ber, Jeremy <jd...@amazon.com> wrote:
> 
> Hi Puneet,
> Amazon Kinesis Data Analytics for Apache Flink is a managed Apache Flink offering--it removes the need to setup your own checkpointing and snapshotting by providing a built-in mechanism for both.
> KDA Flink runs in session mode but only allows 1 application per cluster. We also offer Kinesis Data Analytics Studio, which is an Apache Zeppelin based notebook development experience for Apache Flink. This feature runs in session mode as well, but allows for multiple jobs per cluster. The Job Manager runs in high availability mode using Zookeeper across multiple Availability Zones. We use Amazon EKS behind the scenes to recover in the event of pod failures.
> You can read more about this here:https://docs.aws.amazon.com/kinesisanalytics/latest/java/disaster-recovery-resiliency.html <https://docs.aws.amazon.com/kinesisanalytics/latest/java/disaster-recovery-resiliency.html>
> If you decide to run your workloads on K8s, you would be responsible for implementing the above features, plus monitoring and deployments would require additional lift. The benefit of using K8s yourself would be access to the flink-conf and full control over cluster instance types and configuration which you do not have in KDA today.
>  
> Hope this helps,
> Jeremy
>  
>  
>  
>  
> From: Danny Cranmer <dannycranmer@apache.org <ma...@apache.org>>
> Date: Wednesday, February 16, 2022 at 9:27 AM
> To: Puneet Duggal <puneetduggal1795@gmail.com <ma...@gmail.com>>, "Ber, Jeremy" <jdber@amazon.com <ma...@amazon.com>>
> Cc: user <user@flink.apache.org <ma...@flink.apache.org>>
> Subject: RE: [EXTERNAL] AWS Kinesis Flink vs K8s
>  
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
>  
> +Jeremy who can help answer this question.
>  
> Thanks,
>  
> On Wed, Feb 16, 2022 at 10:26 AM Puneet Duggal <puneetduggal1795@gmail.com <ma...@gmail.com>> wrote:
> Hi,
> 
> Just wanted to ask the community various pros and cons of deploying flink using AWS Kinesis vs using K8s application mode. Currently we are deploying flink cluster in HA session standalone mode and planning to switch to application deployment mode.
> 
> Regards, 
> Puneet