JobManager HA behind load balancer

[Shannon Carey <sc...@expedia.com>]

Is anyone running multiple JobManagers (in High Availability mode) behind a load balancer such as an AWS ELB or a software proxy such as HAProxy or Nginx?

Right now, it appears that server-side redirects that come from the JobManager Web UI use the internal IP address of the JobManager (from Akka). Therefore, if you're accessing your JobManager via a DNS name or a load balancing proxy, the redirect doesn't work properly. Has anyone created a workaround for this?

If there's no workaround, should we perhaps add a config setting to the JobManager to tell it what DNS name or root URL to use when sending redirect responses?

Also, it looks like at least some types of requests are not supported by non-master JobManagers, and therefore they respond with a redirect to the internal address of the master. Is it necessary to integrate the proxy with Zookeeper so that requests will only be proxied to the master JobManager? If the non-master nodes only send redirects, then including them as upstream servers in the proxy would be problematic.

Thanks for the info,
Shannon

Re: JobManager HA behind load balancer

[Aljoscha Krettek <al...@apache.org>]

Hi Shannon,

I think this will be reworked as part of the FLIP-6 efforts. Your problem comes up in Kubernetes where several JMs would sit behind a K8s service. I think the solution is for the JMs to act as proxies (they contact the JM leader, get the data and return to the client) instead of redirecting.

I'm also cc'ing Till, how is leading those efforts.

Best,
Aljoscha
> On 16. Aug 2017, at 17:24, Shannon Carey <sc...@expedia.com> wrote:
> 
> Is anyone running multiple JobManagers (in High Availability mode) behind a load balancer such as an AWS ELB or a software proxy such as HAProxy or Nginx?
> 
> Right now, it appears that server-side redirects that come from the JobManager Web UI use the internal IP address of the JobManager (from Akka). Therefore, if you're accessing your JobManager via a DNS name or a load balancing proxy, the redirect doesn't work properly. Has anyone created a workaround for this?
> 
> If there's no workaround, should we perhaps add a config setting to the JobManager to tell it what DNS name or root URL to use when sending redirect responses?
> 
> Also, it looks like at least some types of requests are not supported by non-master JobManagers, and therefore they respond with a redirect to the internal address of the master. Is it necessary to integrate the proxy with Zookeeper so that requests will only be proxied to the master JobManager? If the non-master nodes only send redirects, then including them as upstream servers in the proxy would be problematic.
> 
> Thanks for the info,
> Shannon