You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Fabian Hueske <fh...@gmail.com> on 2018/04/04 12:41:00 UTC

Re: SSL config on Kubernetes - Dynamic IP

Thank you Edward and Christophe!

2018-03-29 17:55 GMT+02:00 Edward Alexander Rojas Clavijo <
edward.rojascl@gmail.com>:

> Hi all,
>
> I did some tests based on the PR Christophe mentioned above and by making
> a change on the NettyClient to use CanonicalHostName instead of
> HostNameAddress to identify the server, the SSL validation works!!
>
> I created a PR with this change: https://github.com/apache/flink/pull/5789
>
> Regards,
> Edward
>
> 2018-03-28 17:22 GMT+02:00 Edward Alexander Rojas Clavijo <
> edward.rojascl@gmail.com>:
>
>> Hi Till,
>>
>> I just created the JIRA ticket: https://issues.apache.org/jira
>> /browse/FLINK-9103
>>
>> I added the JobManager and TaskManager logs, Hope this helps to resolve
>> the issue.
>>
>> Regards,
>> Edward
>>
>> 2018-03-27 17:48 GMT+02:00 Till Rohrmann <tr...@apache.org>:
>>
>>> Hi Edward,
>>>
>>> could you please file a JIRA issue for this problem. It might be as
>>> simple as that the TaskManager's network stack uses the IP instead of the
>>> hostname as you suggested. But we have to look into this to be sure. Also
>>> the logs of the JobManager as well as the TaskManagers could be helpful.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Tue, Mar 27, 2018 at 5:17 PM, Christophe Jolif <cj...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> I suspect this relates to: https://issues.apache.org/
>>>> jira/browse/FLINK-5030
>>>>
>>>> For which there was a PR at some point but nothing has been done so
>>>> far. It seems the current code explicitly uses the IP vs Hostname for Netty
>>>> SSL configuration.
>>>>
>>>> Without that I'm really wondering how people are reasonably using SSL
>>>> on a Kubernetes Flink-based cluster as every time a pod is (re-started) it
>>>> can theoretically take a different IP? Or do I miss something?
>>>>
>>>> --
>>>> Christophe
>>>>
>>>> On Tue, Mar 27, 2018 at 3:24 PM, Edward Alexander Rojas Clavijo <
>>>> edward.rojascl@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Currently I have a Flink 1.4 cluster running on kubernetes and with
>>>>> SSL configuration based on https://ci.apache.org/proje
>>>>> cts/flink/flink-docs-master/ops/security-ssl.html.
>>>>>
>>>>> However, as the IP of the nodes are dynamic (from the nature of
>>>>> kubernetes), we are using only the DNS which we can control using
>>>>> kubernetes services. So we add to the Subject Alternative Name(SAN) the
>>>>> flink-jobmanager DNS and also the DNS for the task managers
>>>>> *.flink-taskmanager-svc (each task manager has a DNS in the form
>>>>> flink-taskmanager-0.flink-taskmanager-svc).
>>>>>
>>>>> Additionally we set the jobmanager.rpc.address property on all the
>>>>> nodes and each task manager sets the taskmanager.host property, all
>>>>> matching the ones on the certificate.
>>>>>
>>>>> This is working well when using Job with Parallelism set to 1. The SSL
>>>>> validations are good and the Jobmanager can communicate with Task manager
>>>>> and vice versa.
>>>>>
>>>>> But when we set the parallelism to more than 1 we have exceptions on
>>>>> the SSL validation like this:
>>>>>
>>>>> Caused by: java.security.cert.CertificateException: No subject
>>>>> alternative names matching IP address 172.30.247.163 found
>>>>> at sun.security.util.HostnameChecker.matchIP(HostnameChecker.ja
>>>>> va:168)
>>>>> at sun.security.util.HostnameChecker.match(HostnameChecker.java:94)
>>>>> at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509Trus
>>>>> tManagerImpl.java:455)
>>>>> at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509Trus
>>>>> tManagerImpl.java:436)
>>>>> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509Trust
>>>>> ManagerImpl.java:252)
>>>>> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X50
>>>>> 9TrustManagerImpl.java:136)
>>>>> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHa
>>>>> ndshaker.java:1601)
>>>>> ... 21 more
>>>>>
>>>>>
>>>>> From the logs I see the Jobmanager is correctly registering the
>>>>> taskmanagers:
>>>>>
>>>>> org.apache.flink.runtime.instance.InstanceManager   - Registered
>>>>> TaskManager at flink-taskmanager-1 (akka.ssl.tcp://flink@taiga-fl
>>>>> ink-taskmanager-1.flink-taskmanager-svc.default.svc.cluster.local:6122/user/taskmanager)
>>>>> as 1a3f59693cec8b3929ed8898edcc2700. Current number of registered
>>>>> hosts is 3. Current number of alive task slots is 6.
>>>>>
>>>>> And also each taskmanager is correctly registered to use the hostname
>>>>> for communication:
>>>>>
>>>>> org.apache.flink.runtime.taskmanager.TaskManager   - TaskManager will
>>>>> use hostname/address 'flink-taskmanager-1.flink-tas
>>>>> kmanager-svc.default.svc.cluster.local' (172.30.247.163) for
>>>>> communication.
>>>>> ...
>>>>> akka.remote.Remoting   - Remoting started; listening on addresses
>>>>> :[akka.ssl.tcp://flink@flink-taskmanager-1.flink-taskmanager
>>>>> -svc.default.svc.cluster.local:6122]
>>>>> ...
>>>>> org.apache.flink.runtime.io.network.netty.NettyConfig   - NettyConfig
>>>>> [server address: flink-taskmanager-1.flink-task
>>>>> manager-svc.default.svc.cluster.local/172.30.247.163, server port:
>>>>> 6121, ssl enabled: true, memory segment size (bytes): 32768, transport
>>>>> type: NIO, number of server threads: 2 (manual), number of client threads:
>>>>> 2 (manual), server connect backlog: 0 (use Netty's default), client connect
>>>>> timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
>>>>> default)]
>>>>> ...
>>>>> org.apache.flink.runtime.taskmanager.TaskManager   - TaskManager data
>>>>> connection information: bf4a9b50e57c99c17049adb66d65f685 @
>>>>> flink-taskmanager-1.flink-taskmanager-svc.default.svc.cluster.local
>>>>> (dataPort=6121)
>>>>>
>>>>>
>>>>>
>>>>> But even with that, it seems like the taskmanagers are using the IP
>>>>> communicate between them and the SSL validation fails.
>>>>>
>>>>> Do you know if it's possible to make the taskmanagers to use the
>>>>> hostname to communicate instead of the IP ?
>>>>> or
>>>>> Do you have any advice to get the SSL configuration to work on this
>>>>> environment ?
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>> Regards,
>>>>> Edward
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Christophe
>>>>
>>>
>>>
>>
>>
>> --
>> *Edward Alexander Rojas Clavijo*
>>
>>
>>
>> *Software EngineerHybrid CloudIBM France*
>>
>
>

Re: SSL config on Kubernetes - Dynamic IP

Posted by Christophe Jolif <cj...@gmail.com>.
By the way Fabian, any chance this issue is looked into / the PR considered
for 1.5?

--
Christophe

On Wed, Apr 4, 2018 at 2:41 PM, Fabian Hueske <fh...@gmail.com> wrote:

> Thank you Edward and Christophe!
>
> 2018-03-29 17:55 GMT+02:00 Edward Alexander Rojas Clavijo <
> edward.rojascl@gmail.com>:
>
>> Hi all,
>>
>> I did some tests based on the PR Christophe mentioned above and by making
>> a change on the NettyClient to use CanonicalHostName instead of
>> HostNameAddress to identify the server, the SSL validation works!!
>>
>> I created a PR with this change: https://github.com/apa
>> che/flink/pull/5789
>>
>> Regards,
>> Edward
>>
>> 2018-03-28 17:22 GMT+02:00 Edward Alexander Rojas Clavijo <
>> edward.rojascl@gmail.com>:
>>
>>> Hi Till,
>>>
>>> I just created the JIRA ticket: https://issues.apache.org/jira
>>> /browse/FLINK-9103
>>>
>>> I added the JobManager and TaskManager logs, Hope this helps to resolve
>>> the issue.
>>>
>>> Regards,
>>> Edward
>>>
>>> 2018-03-27 17:48 GMT+02:00 Till Rohrmann <tr...@apache.org>:
>>>
>>>> Hi Edward,
>>>>
>>>> could you please file a JIRA issue for this problem. It might be as
>>>> simple as that the TaskManager's network stack uses the IP instead of the
>>>> hostname as you suggested. But we have to look into this to be sure. Also
>>>> the logs of the JobManager as well as the TaskManagers could be helpful.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Tue, Mar 27, 2018 at 5:17 PM, Christophe Jolif <cj...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> I suspect this relates to: https://issues.apache.org/
>>>>> jira/browse/FLINK-5030
>>>>>
>>>>> For which there was a PR at some point but nothing has been done so
>>>>> far. It seems the current code explicitly uses the IP vs Hostname for Netty
>>>>> SSL configuration.
>>>>>
>>>>> Without that I'm really wondering how people are reasonably using SSL
>>>>> on a Kubernetes Flink-based cluster as every time a pod is (re-started) it
>>>>> can theoretically take a different IP? Or do I miss something?
>>>>>
>>>>> --
>>>>> Christophe
>>>>>
>>>>> On Tue, Mar 27, 2018 at 3:24 PM, Edward Alexander Rojas Clavijo <
>>>>> edward.rojascl@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Currently I have a Flink 1.4 cluster running on kubernetes and with
>>>>>> SSL configuration based on https://ci.apache.org/proje
>>>>>> cts/flink/flink-docs-master/ops/security-ssl.html.
>>>>>>
>>>>>> However, as the IP of the nodes are dynamic (from the nature of
>>>>>> kubernetes), we are using only the DNS which we can control using
>>>>>> kubernetes services. So we add to the Subject Alternative Name(SAN) the
>>>>>> flink-jobmanager DNS and also the DNS for the task managers
>>>>>> *.flink-taskmanager-svc (each task manager has a DNS in the form
>>>>>> flink-taskmanager-0.flink-taskmanager-svc).
>>>>>>
>>>>>> Additionally we set the jobmanager.rpc.address property on all the
>>>>>> nodes and each task manager sets the taskmanager.host property, all
>>>>>> matching the ones on the certificate.
>>>>>>
>>>>>> This is working well when using Job with Parallelism set to 1. The
>>>>>> SSL validations are good and the Jobmanager can communicate with Task
>>>>>> manager and vice versa.
>>>>>>
>>>>>> But when we set the parallelism to more than 1 we have exceptions on
>>>>>> the SSL validation like this:
>>>>>>
>>>>>> Caused by: java.security.cert.CertificateException: No subject
>>>>>> alternative names matching IP address 172.30.247.163 found
>>>>>> at sun.security.util.HostnameChecker.matchIP(HostnameChecker.ja
>>>>>> va:168)
>>>>>> at sun.security.util.HostnameChecker.match(HostnameChecker.java:94)
>>>>>> at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509Trus
>>>>>> tManagerImpl.java:455)
>>>>>> at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509Trus
>>>>>> tManagerImpl.java:436)
>>>>>> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509Trust
>>>>>> ManagerImpl.java:252)
>>>>>> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X50
>>>>>> 9TrustManagerImpl.java:136)
>>>>>> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHa
>>>>>> ndshaker.java:1601)
>>>>>> ... 21 more
>>>>>>
>>>>>>
>>>>>> From the logs I see the Jobmanager is correctly registering the
>>>>>> taskmanagers:
>>>>>>
>>>>>> org.apache.flink.runtime.instance.InstanceManager   - Registered
>>>>>> TaskManager at flink-taskmanager-1 (akka.ssl.tcp://flink@taiga-fl
>>>>>> ink-taskmanager-1.flink-taskmanager-svc.default.svc.cluster.local:6122/user/taskmanager)
>>>>>> as 1a3f59693cec8b3929ed8898edcc2700. Current number of registered
>>>>>> hosts is 3. Current number of alive task slots is 6.
>>>>>>
>>>>>> And also each taskmanager is correctly registered to use the hostname
>>>>>> for communication:
>>>>>>
>>>>>> org.apache.flink.runtime.taskmanager.TaskManager   - TaskManager
>>>>>> will use hostname/address 'flink-taskmanager-1.flink-tas
>>>>>> kmanager-svc.default.svc.cluster.local' (172.30.247.163) for
>>>>>> communication.
>>>>>> ...
>>>>>> akka.remote.Remoting   - Remoting started; listening on addresses
>>>>>> :[akka.ssl.tcp://flink@flink-taskmanager-1.flink-taskmanager
>>>>>> -svc.default.svc.cluster.local:6122]
>>>>>> ...
>>>>>> org.apache.flink.runtime.io.network.netty.NettyConfig   -
>>>>>> NettyConfig [server address: flink-taskmanager-1.flink-task
>>>>>> manager-svc.default.svc.cluster.local/172.30.247.163, server port:
>>>>>> 6121, ssl enabled: true, memory segment size (bytes): 32768, transport
>>>>>> type: NIO, number of server threads: 2 (manual), number of client threads:
>>>>>> 2 (manual), server connect backlog: 0 (use Netty's default), client connect
>>>>>> timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
>>>>>> default)]
>>>>>> ...
>>>>>> org.apache.flink.runtime.taskmanager.TaskManager   - TaskManager
>>>>>> data connection information: bf4a9b50e57c99c17049adb66d65f685 @
>>>>>> flink-taskmanager-1.flink-taskmanager-svc.default.svc.cluster.local
>>>>>> (dataPort=6121)
>>>>>>
>>>>>>
>>>>>>
>>>>>> But even with that, it seems like the taskmanagers are using the IP
>>>>>> communicate between them and the SSL validation fails.
>>>>>>
>>>>>> Do you know if it's possible to make the taskmanagers to use the
>>>>>> hostname to communicate instead of the IP ?
>>>>>> or
>>>>>> Do you have any advice to get the SSL configuration to work on this
>>>>>> environment ?
>>>>>>
>>>>>> Thanks in advance.
>>>>>>
>>>>>> Regards,
>>>>>> Edward
>>>>>>
>>>>>