Running flink in a Local Execution Environment for Production Workloads

[Joseph Lorenzini <JL...@gohealth.com>]

Hi all,



I plan to run flink jobs as docker containers in a AWS Elastic Container
Service. I will have checkpointing enabled where state is stored in a s3
bucket. Each deployment will run in a per-job mode.  Are there any non-obvious
downsides to running these jobs with a local execution environment so that the
deployment turns into deploying a single java application?



The obvious downside is that you don’t get any horizontal scalability. That’s
a given and I’d have to scale up not out in this mode. I’d like to discover if
there are any other negatives with this approach.



Thanks,  
Joe  

Privileged/Confidential Information may be contained in this message. If you
are not the addressee indicated in this message (or responsible for delivery
of the message to such person), you may not copy or deliver this message to
anyone. In such case, you should destroy this message and kindly notify the
sender by reply email. Please advise immediately if you or your employer does
not consent to Internet email for messages of this kind. Opinions, conclusions
and other information in this message that do not relate to the official
business of my firm shall be understood as neither given nor endorsed by it.

Re: Running flink in a Local Execution Environment for Production Workloads

[David Anderson <da...@alpinegizmo.com>]

Another factor to be aware of is that there's no cluster configuration file
to load (the mini-cluster doesn't know where to find flink-conf.yaml), and
by default you won't have a REST API endpoint or web UI available. But you
can do the configuration in the code, and/or provide configuration on the
command line. And both of these points can be easily overcome with
something like this:

String cwd = Paths.get(".").toAbsolutePath().normalize().toString();
Configuration conf = GlobalConfiguration.loadConfiguration(cwd);
env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

Regards,
David

On Thu, Oct 29, 2020 at 11:30 AM Matthias Pohl <ma...@ververica.com>
wrote:

> Hi Joseph,
> thanks for reaching out to us. There shouldn't be any downsides other than
> the one you already mentioned as far as I know.
>
> Best,
> Matthias
>
> On Fri, Oct 23, 2020 at 1:27 PM Joseph Lorenzini <JL...@gohealth.com>
> wrote:
>
>> Hi all,
>>
>>
>>
>> I plan to run flink jobs as docker containers in a AWS Elastic Container
>> Service. I will have checkpointing enabled where state is stored in a s3
>> bucket. Each deployment will run in a per-job mode.  Are there any
>> non-obvious downsides to running these jobs with a local execution
>> environment so that the deployment turns into deploying a single java
>> application?
>>
>>
>>
>> The obvious downside is that you don’t get any horizontal scalability.
>> That’s a given and I’d have to scale up not out in this mode. I’d like to
>> discover if there are any other negatives with this approach.
>>
>>
>>
>> Thanks,
>> Joe
>> Privileged/Confidential Information may be contained in this message. If
>> you are not the addressee indicated in this message (or responsible for
>> delivery of the message to such person), you may not copy or deliver this
>> message to anyone. In such case, you should destroy this message and kindly
>> notify the sender by reply email. Please advise immediately if you or your
>> employer does not consent to Internet email for messages of this kind.
>> Opinions, conclusions and other information in this message that do not
>> relate to the official business of my firm shall be understood as neither
>> given nor endorsed by it.
>>
>

Re: Running flink in a Local Execution Environment for Production Workloads

[Matthias Pohl <ma...@ververica.com>]

Hi Joseph,
thanks for reaching out to us. There shouldn't be any downsides other than
the one you already mentioned as far as I know.

Best,
Matthias

On Fri, Oct 23, 2020 at 1:27 PM Joseph Lorenzini <JL...@gohealth.com>
wrote:

> Hi all,
>
>
>
> I plan to run flink jobs as docker containers in a AWS Elastic Container
> Service. I will have checkpointing enabled where state is stored in a s3
> bucket. Each deployment will run in a per-job mode.  Are there any
> non-obvious downsides to running these jobs with a local execution
> environment so that the deployment turns into deploying a single java
> application?
>
>
>
> The obvious downside is that you don’t get any horizontal scalability.
> That’s a given and I’d have to scale up not out in this mode. I’d like to
> discover if there are any other negatives with this approach.
>
>
>
> Thanks,
> Joe
> Privileged/Confidential Information may be contained in this message. If
> you are not the addressee indicated in this message (or responsible for
> delivery of the message to such person), you may not copy or deliver this
> message to anyone. In such case, you should destroy this message and kindly
> notify the sender by reply email. Please advise immediately if you or your
> employer does not consent to Internet email for messages of this kind.
> Opinions, conclusions and other information in this message that do not
> relate to the official business of my firm shall be understood as neither
> given nor endorsed by it.
>