[jira] [Commented] (FLINK-17641) How to secure flink applications on yarn on multi-tenant environment

["Robert Metzger (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168775#comment-17168775 ] 

Robert Metzger commented on FLINK-17641:
----------------------------------------

Besides the approach you've already mentioned (using SSL), the Flink community recommends setting up a service in front of the Flink HTTP endpoints that controls access to it.
You could for example use nginx configured as a reverse proxy for that.
I agree that this solution is not very elegant in YARN, where the Flink sessions are probably rather short-lived, and you would need to dynamically figure out the HTTP endpoint of each Flink application.
Also, you need to either forbid users in your intranet to access the ports / ip-range of the Flink sessions OR you setup SSL between nginx and the Flink instances (can be the same certs).

This is a fairly frequent feature request for the Flink REST interfaces. In the past, we have rejected this because it out of the scope of Flink ("feature creep").

However, I see that it is fairly difficult to implement this for a YARN setup, in particular figuring out the right ip:port of the current JM leader (which relies on Zookeeper if you are using HA)
What I could imagine as an addition to Flink is adding a new command to the CLI frontend, that returns some cluster information, including the leader ip:port.
Ideally, nginx is able to query this information from the command line.


> How to secure flink applications on yarn on multi-tenant environment
> --------------------------------------------------------------------
>
>                 Key: FLINK-17641
>                 URL: https://issues.apache.org/jira/browse/FLINK-17641
>             Project: Flink
>          Issue Type: Wish
>          Components: Deployment / YARN
>            Reporter: Ethan Li
>            Priority: Major
>
> This is a question I wish to get some insights on. 
> We are trying to support and secure flink on shared yarn cluster. Besides the security provided by yarn side (queueACL, kerberos), what I noticed is that flink CLI can still interact with the flink job as long as it knows the jobmanager rpc port/hostname and rest.port, which can be obtained easily with yarn command. 
> Also on the UI side, on yarn cluster, users can visit flink job UI via yarn proxy using browser. As long as the user can authenticate and view yarn resourcemanager webpage, he/she can visit the flink UI without any problem. This basically means Flink UI is wide-open to corp internal users.
> On the internal connection side, I am aware of the support added in 1.10 to limit the mTLS connection by configuring security.ssl.internal.cert.fingerprint (https://ci.apache.org/projects/flink/flink-docs-stable/ops/security-ssl.html)
> This works but it is not very flexible. Users need to update the config if the cert changes before they submit a new job.
> I asked the similar question on the mailing list before. I am really interested in how other folks deal with this issue. Thanks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)