You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Wei-Chiu Chuang <we...@cloudera.com.INVALID> on 2020/03/05 02:27:07 UTC
Hadoop & TLS 1.3
Hi
We recently updated Jetty to 9.4.20 (HADOOP-16152
<https://issues.apache.org/jira/browse/HADOOP-16152>). According to the
Jetty release note
https://www.eclipse.org/lists/jetty-announce/msg00124.html Jetty 9.4.12 and
above, (I believe JDK 11.0.3 or above is required:
https://webtide.com/openjdk-11-and-tls-1-3-issues/
https://campustechnology.com/articles/2018/09/27/jdk-11-adds-support-for-tls-1.3-and-http2.aspx)
supports TLS 1.3.
I had a quick check and it appears it only requires:
hadoop.ssl.enabled.protocols = TLSv1.3
TLSv1.3 introduces new cipher suites () and legacy cipher suites are no
longer accepted.
Supported cipher suites in TLS 1.3:
TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
Fortunately, we use exclude list rather than accept list in Hadoop to
select ciphers, so we don't really need to change the cipher suite list in
the configuration.
I tried JDK 1.8.221 and JDK 11.0.5, the former failed and the latter
succeeded. So looks like JDK 11 is required (not sure if it's because of
the server or the client).
I am not familiar with the SSL connection in the cloud connectors so not
sure if this applies to them (or if cloud storage providers support TLS 1.3
at all)
Has anyone tried TLS 1.3 with Hadoop, and do you see the improved
performance as stated in
https://www.ssl.com/article/tls-1-3-is-here-to-stay/?
Cheers,
Wei-Chiu
Re: Hadoop & TLS 1.3
Posted by Akira Ajisaka <aa...@apache.org>.
Hi Wei-Chiu,
Now we are experimenting trunk with Java11 in our development cluster and
the HttpFS supports TLS 1.3 without any specific configuration.
We haven't done any performance tests.
-Akira
$ curl -I --tlsv1.3 --negotiate -u : "https://
<host>:<port>/webhdfs/v1/?op=liststatus"
HTTP/1.1 401 Authentication required
Date: Fri, 06 Mar 2020 07:15:12 GMT
Date: Fri, 06 Mar 2020 07:15:12 GMT
Pragma: no-cache
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Path=/; Secure; HttpOnly
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 271
HTTP/1.1 200 OK
Date: Fri, 06 Mar 2020 07:15:12 GMT
(snip)
On Thu, Mar 5, 2020 at 11:27 AM Wei-Chiu Chuang
<we...@cloudera.com.invalid> wrote:
> Hi
>
> We recently updated Jetty to 9.4.20 (HADOOP-16152
> <https://issues.apache.org/jira/browse/HADOOP-16152>). According to the
> Jetty release note
> https://www.eclipse.org/lists/jetty-announce/msg00124.html Jetty 9.4.12
> and
> above, (I believe JDK 11.0.3 or above is required:
> https://webtide.com/openjdk-11-and-tls-1-3-issues/
>
> https://campustechnology.com/articles/2018/09/27/jdk-11-adds-support-for-tls-1.3-and-http2.aspx
> )
> supports TLS 1.3.
>
> I had a quick check and it appears it only requires:
> hadoop.ssl.enabled.protocols = TLSv1.3
>
> TLSv1.3 introduces new cipher suites () and legacy cipher suites are no
> longer accepted.
>
> Supported cipher suites in TLS 1.3:
> TLS_AES_128_GCM_SHA256
> TLS_AES_256_GCM_SHA384
>
> Fortunately, we use exclude list rather than accept list in Hadoop to
> select ciphers, so we don't really need to change the cipher suite list in
> the configuration.
>
> I tried JDK 1.8.221 and JDK 11.0.5, the former failed and the latter
> succeeded. So looks like JDK 11 is required (not sure if it's because of
> the server or the client).
>
> I am not familiar with the SSL connection in the cloud connectors so not
> sure if this applies to them (or if cloud storage providers support TLS 1.3
> at all)
>
> Has anyone tried TLS 1.3 with Hadoop, and do you see the improved
> performance as stated in
> https://www.ssl.com/article/tls-1-3-is-here-to-stay/?
>
> Cheers,
> Wei-Chiu
>
Re: Hadoop & TLS 1.3
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
sorry, just checked the release notes of a JIRA, its 1.1.1 which breaks
wildfly-1.0.4.Final
"...to make abfs and adl connectors compatible with alpine linux and other
platforms which have libssl1.1-1.1.1b-r1 as their native openssl
implementation. see: HADOOP-16460. HADOOP-16438"
As well as editing the hadoop wildfly version, you need to move to a
version of azure-datalake-storage.jar which doesn' t have an unshaded copy
of the wildfly 1.0.4 classes.
On Thu, 5 Mar 2020 at 18:09, Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:
> >
> >
> > abfs and s3a can now go via wildfly to use any native openssl 1.1
> libraries
> > -if that supports TLS1.3 then maybe the stores will talk through it. No
> > idea if anyone has tried it.
> >
> > Warning: Do not attempt to use wildfly-1.0.4-Final with openssl 1.1; you
> > need to upgrade to 1.0.7 unless you like to see NPE stack traces
> >
>
> https://wiki.openssl.org/index.php/TLS1.3
> We will need OpenSSL 1.1.1 to support TLS 1.3.
> According to the wiki 1.1.1 is a drop in replacement of 1.1.0. So maybe
> Hadoop already supports it.
>
Re: Hadoop & TLS 1.3
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
>
>
> abfs and s3a can now go via wildfly to use any native openssl 1.1 libraries
> -if that supports TLS1.3 then maybe the stores will talk through it. No
> idea if anyone has tried it.
>
> Warning: Do not attempt to use wildfly-1.0.4-Final with openssl 1.1; you
> need to upgrade to 1.0.7 unless you like to see NPE stack traces
>
https://wiki.openssl.org/index.php/TLS1.3
We will need OpenSSL 1.1.1 to support TLS 1.3.
According to the wiki 1.1.1 is a drop in replacement of 1.1.0. So maybe
Hadoop already supports it.
Re: Hadoop & TLS 1.3
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Thu, 5 Mar 2020 at 02:27, Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:
> Hi
>
> I am not familiar with the SSL connection in the cloud connectors so not
> sure if this applies to them (or if cloud storage providers support TLS 1.3
> at all)
>
>
abfs and s3a can now go via wildfly to use any native openssl 1.1 libraries
-if that supports TLS1.3 then maybe the stores will talk through it. No
idea if anyone has tried it.
Warning: Do not attempt to use wildfly-1.0.4-Final with openssl 1.1; you
need to upgrade to 1.0.7 unless you like to see NPE stack traces
Re: Hadoop & TLS 1.3
Posted by Akira Ajisaka <aa...@apache.org>.
Hi Wei-Chiu,
Now we are experimenting trunk with Java11 in our development cluster and
the HttpFS supports TLS 1.3 without any specific configuration.
We haven't done any performance tests.
-Akira
$ curl -I --tlsv1.3 --negotiate -u : "https://
<host>:<port>/webhdfs/v1/?op=liststatus"
HTTP/1.1 401 Authentication required
Date: Fri, 06 Mar 2020 07:15:12 GMT
Date: Fri, 06 Mar 2020 07:15:12 GMT
Pragma: no-cache
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Path=/; Secure; HttpOnly
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 271
HTTP/1.1 200 OK
Date: Fri, 06 Mar 2020 07:15:12 GMT
(snip)
On Thu, Mar 5, 2020 at 11:27 AM Wei-Chiu Chuang
<we...@cloudera.com.invalid> wrote:
> Hi
>
> We recently updated Jetty to 9.4.20 (HADOOP-16152
> <https://issues.apache.org/jira/browse/HADOOP-16152>). According to the
> Jetty release note
> https://www.eclipse.org/lists/jetty-announce/msg00124.html Jetty 9.4.12
> and
> above, (I believe JDK 11.0.3 or above is required:
> https://webtide.com/openjdk-11-and-tls-1-3-issues/
>
> https://campustechnology.com/articles/2018/09/27/jdk-11-adds-support-for-tls-1.3-and-http2.aspx
> )
> supports TLS 1.3.
>
> I had a quick check and it appears it only requires:
> hadoop.ssl.enabled.protocols = TLSv1.3
>
> TLSv1.3 introduces new cipher suites () and legacy cipher suites are no
> longer accepted.
>
> Supported cipher suites in TLS 1.3:
> TLS_AES_128_GCM_SHA256
> TLS_AES_256_GCM_SHA384
>
> Fortunately, we use exclude list rather than accept list in Hadoop to
> select ciphers, so we don't really need to change the cipher suite list in
> the configuration.
>
> I tried JDK 1.8.221 and JDK 11.0.5, the former failed and the latter
> succeeded. So looks like JDK 11 is required (not sure if it's because of
> the server or the client).
>
> I am not familiar with the SSL connection in the cloud connectors so not
> sure if this applies to them (or if cloud storage providers support TLS 1.3
> at all)
>
> Has anyone tried TLS 1.3 with Hadoop, and do you see the improved
> performance as stated in
> https://www.ssl.com/article/tls-1-3-is-here-to-stay/?
>
> Cheers,
> Wei-Chiu
>