How to enforce ACLs on Flink JobManager/ApplicationMaster URL on Yarn

[Ethan Li <et...@gmail.com>]

Hi Team,

I am evaluating Flink on yarn. I can submit a flink job to a secured Yarn
cluster and the job can run correctly. But flink jobmanager UI seems
accessibly by everyone. Is there anyway in Flink or Yarn to secure it with
ACLs?

Thanks,
Ethan

Re: How to enforce ACLs on Flink JobManager/ApplicationMaster URL on Yarn

[Ethan Li <et...@gmail.com>]

Thanks for sharing! Aaron. Your comment is very helpful.

Our end goal is to support multi-tenancy and also share the yarn cluster
with MapReduce, Spark and other jobs. We probably need something else.

---

I wonder if there is any builtin functionalities in Flink or Yarn that
already supports ACL on JobManager/AM.

I also noticed is that flink doesn't have a way to set up ACLs for  yarn
containers so only the submitter or yarn admin can view the container logs.
A related PR (https://github.com/apache/flink/pull/8760) was closed due to
inactivity.  How do people deal with container ACLs? Do most of
flink-on-yarn dev/users use it without security? Or do we have to implement
our own solution outside of flink/yarn?  Please advise if anyone has any
idea about this. Thanks very much!

Best,
Ethan



On Mon, Mar 30, 2020 at 4:13 PM Aaron Langford <aa...@gmail.com>
wrote:

> I'd be curious to see how others have done this, but our setup restricts
> network access to machines in the YARN cluster to a jump box. Access to
> Flink job manager is limited to whoever can ssh to that box, and that is
> controlled with an Ansible playbook. Additionally, we have a list of users
> specific to the cluster who can ssh to the machines in the YARN cluster
> (also managed with Ansible). So the allowed users are the intersection of
> the jump server ACL and the YARN cluster ACL. Web access happens by using a
> local socks proxy along with the Foxy Proxy browser plugin. It's definitely
> pretty crude and doesn't scale super well as more teams need varying access
> policies to different YARN clusters/jobs, but it is satisfying our needs
> for now. One big simplifying assumption is that we don't support shared
> clusters. Amazon's EMR service allows teams to spin up clusters really
> easily, so we can get away with saying that the machine network rules can
> map to the actual access control rules any given job/team might need.
>
> Aaron
>
> On Mon, Mar 30, 2020 at 12:53 PM Ethan Li <et...@gmail.com>
> wrote:
>
>> Hi Team,
>>
>> I am evaluating Flink on yarn. I can submit a flink job to a secured Yarn
>> cluster and the job can run correctly. But flink jobmanager UI seems
>> accessibly by everyone. Is there anyway in Flink or Yarn to secure it with
>> ACLs?
>>
>> Thanks,
>> Ethan
>>
>

Re: How to enforce ACLs on Flink JobManager/ApplicationMaster URL on Yarn

[Aaron Langford <aa...@gmail.com>]

I'd be curious to see how others have done this, but our setup restricts
network access to machines in the YARN cluster to a jump box. Access to
Flink job manager is limited to whoever can ssh to that box, and that is
controlled with an Ansible playbook. Additionally, we have a list of users
specific to the cluster who can ssh to the machines in the YARN cluster
(also managed with Ansible). So the allowed users are the intersection of
the jump server ACL and the YARN cluster ACL. Web access happens by using a
local socks proxy along with the Foxy Proxy browser plugin. It's definitely
pretty crude and doesn't scale super well as more teams need varying access
policies to different YARN clusters/jobs, but it is satisfying our needs
for now. One big simplifying assumption is that we don't support shared
clusters. Amazon's EMR service allows teams to spin up clusters really
easily, so we can get away with saying that the machine network rules can
map to the actual access control rules any given job/team might need.

Aaron

On Mon, Mar 30, 2020 at 12:53 PM Ethan Li <et...@gmail.com> wrote:

> Hi Team,
>
> I am evaluating Flink on yarn. I can submit a flink job to a secured Yarn
> cluster and the job can run correctly. But flink jobmanager UI seems
> accessibly by everyone. Is there anyway in Flink or Yarn to secure it with
> ACLs?
>
> Thanks,
> Ethan
>