You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Maxim Parkachov <la...@gmail.com> on 2019/07/17 15:53:15 UTC

yarn-session vs cluster per job for streaming jobs

Hi,

I'm looking for advice on how to run flink streaming jobs on Yarn cluster
in production environment. I tried in testing environment both approaches
with HA mode, namely yarn session + multiple jobs vs cluster per job, both
seems to work for my cases, with slight preference of yarn session mode to
centrally manage credentials. I'm looking to run about 10 streaming jobs
mostly reading/writing from kafka + cassandra with following restictions:
1. yarn nodes will be hard rebooted quite often, roughly every 2 weeks. I
have a concern here what happens when Job manager dies in session mode.
2. there are often network interruptions/slowdowns.
3. I'm trying to minimise time to restart job to have as much as possible
continious processing.

Thanks in advance,
Maxim.

Re:Re: yarn-session vs cluster per job for streaming jobs

Posted by Haibo Sun <su...@163.com>.

HI, Maxim

As far as I understand, it's hard to draw a simple conclusion that who's faster. If the job is smaller (for example, the vertex number and the parallelism are very small), the session is usually faster than the per-job mode. I think the session has the advantage of sharing AM and TM, which saves some time for applying and starting containers. But because of the sharing, there will be some resource competition, such as network bandwidth in the submit-job phase. If it is very sensitive to speed, perhaps you can do a comparative test for your specific jobs and environment, and then decide which mode to use?

Best,
Haibo

At 2019-07-18 14:03:01, "Maxim Parkachov" <la...@gmail.com> wrote:

Hi Haibo,

thanks for tip, I almost forgot about max-attempts. I understood implication of running with one AM.

Maybe my question was incorrect, but what would be faster (with regards to downtime of each job):

1. In case of yarn-session: Parallel cancel all jobs with savepoints, restart yarn-session, parallel start all jobs from savepoints
2. In case of per-job mode Parallel cancel all jobs with savepoints, parallel start all jobs from savepoints.

I want to optimise standard situation where I deploy new version of all my jobs. My current impression that job starts faster in yarn-session mode.

Thanks,
Maxim.

On Thu, Jul 18, 2019 at 4:57 AM Haibo Sun <su...@163.com> wrote:

Hi, Maxim

For the concern talking on the first point：
If HA and checkpointing are enabled, AM (the application master, that is the job manager you said) will be restarted by YARN after it dies, and then the dispatcher will try to restore all the previously running jobs correctly. Note that the number of attempts be decided by the configurations "yarn.resourcemanager.am.max-attempts" and "yarn.application-attempts". The obvious difference between the session and per-job modes is that if a fatal error occurs on AM, it will affect all jobs running in it, while the per-job mode will only affect one job.

You can look at this document to see how to configure HA for the Flink cluster on YARN: https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/jobmanager_high_availability.html#yarn-cluster-high-availability .

Best,
Haibo

At 2019-07-17 23:53:15, "Maxim Parkachov" <la...@gmail.com> wrote:

Hi,

I'm looking for advice on how to run flink streaming jobs on Yarn cluster in production environment. I tried in testing environment both approaches with HA mode, namely yarn session + multiple jobs vs cluster per job, both seems to work for my cases, with slight preference of yarn session mode to centrally manage credentials. I'm looking to run about 10 streaming jobs mostly reading/writing from kafka + cassandra with following restictions:
1. yarn nodes will be hard rebooted quite often, roughly every 2 weeks. I have a concern here what happens when Job manager dies in session mode.

2. there are often network interruptions/slowdowns.
3. I'm trying to minimise time to restart job to have as much as possible continious processing.

Thanks in advance,
Maxim.

Re: yarn-session vs cluster per job for streaming jobs

Posted by Maxim Parkachov <la...@gmail.com>.

Hi Haibo,

thanks for tip, I almost forgot about max-attempts. I understood
implication of running with one AM.

Maybe my question was incorrect, but what would be faster (with regards to
downtime of each job):

1. In case of yarn-session: Parallel cancel all jobs with savepoints,
restart yarn-session, parallel start all jobs from savepoints
2. In case of per-job mode Parallel cancel all jobs with savepoints,
parallel start all jobs from savepoints.

I want to optimise standard situation where I deploy new version of all my
jobs. My current impression that job starts faster in yarn-session mode.

Thanks,
Maxim.


On Thu, Jul 18, 2019 at 4:57 AM Haibo Sun <su...@163.com> wrote:

> Hi, Maxim
>
> For the concern talking on the first point：
> If HA and checkpointing are enabled, AM (the application master, that is
> the job manager you said) will be restarted by YARN after it dies, and then
> the dispatcher will try to restore all the previously running jobs
> correctly. Note that the number of attempts be decided by the
> configurations "yarn.resourcemanager.am.max-attempts" and
> "yarn.application-attempts". The obvious difference between the session and
> per-job modes is that if a fatal error occurs on AM, it will affect all
> jobs running in it, while the per-job mode will only affect one job.
>
> You can look at this document to see how to configure HA for the Flink
> cluster on YARN:
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/jobmanager_high_availability.html#yarn-cluster-high-availability
>  .
>
> Best,
> Haibo
>
> At 2019-07-17 23:53:15, "Maxim Parkachov" <la...@gmail.com> wrote:
>
> Hi,
>
> I'm looking for advice on how to run flink streaming jobs on Yarn cluster
> in production environment. I tried in testing environment both approaches
> with HA mode, namely yarn session + multiple jobs vs cluster per job, both
> seems to work for my cases, with slight preference of yarn session mode to
> centrally manage credentials. I'm looking to run about 10 streaming jobs
> mostly reading/writing from kafka + cassandra with following restictions:
> 1. yarn nodes will be hard rebooted quite often, roughly every 2 weeks. I
> have a concern here what happens when Job manager dies in session mode.
> 2. there are often network interruptions/slowdowns.
> 3. I'm trying to minimise time to restart job to have as much as possible
> continious processing.
>
> Thanks in advance,
> Maxim.
>
>

Re:yarn-session vs cluster per job for streaming jobs

Posted by Haibo Sun <su...@163.com>.

Hi, Maxim

Best,
Haibo

At 2019-07-17 23:53:15, "Maxim Parkachov" <la...@gmail.com> wrote:

Hi,

2. there are often network interruptions/slowdowns.
3. I'm trying to minimise time to restart job to have as much as possible continious processing.

Thanks in advance,
Maxim.