How do failovers work on yarn?

[Niels Basjes <Ni...@basjes.nl>]

Hi,

When I start a yarn-session I found that a file is written with a
hostname+port number in a config file on my machine.
Apparently this is the place where the job manager van be found.

Question: What happens if that node on the cluster goes down?
I expect that Yarn will reallocate the job manager to a different node.
How will I be able to submit a job after that happened?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: How do failovers work on yarn?

[Maximilian Michels <mx...@apache.org>]

Hi Niels,

Very good question! The config file which is written serves as a hint
for the client. When the YARN session is started without high
availability mode, i.e. no high availability settings have been found
in the client's config, the client will try to look up the job manager
using the hostname and port found in the file. In the high
availability case, the client will perform a lookup with Zookeeper to
find the current active job manager.

Thus, submitting a new job after a job manager failover should work fine.

Cheers,
Max

On Mon, Dec 21, 2015 at 5:01 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> When I start a yarn-session I found that a file is written with a
> hostname+port number in a config file on my machine.
> Apparently this is the place where the job manager van be found.
>
> Question: What happens if that node on the cluster goes down?
> I expect that Yarn will reallocate the job manager to a different node.
> How will I be able to submit a job after that happened?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes