You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/07/07 00:34:00 UTC
[jira] [Resolved] (SPARK-28258) Incopatibility betweek spark docker
image and hadoop 3.2 and azure tools
[ https://issues.apache.org/jira/browse/SPARK-28258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-28258.
----------------------------------
Resolution: Not A Problem
Seems an issue in HADOOP-16405
> Incopatibility betweek spark docker image and hadoop 3.2 and azure tools
> ------------------------------------------------------------------------
>
> Key: SPARK-28258
> URL: https://issues.apache.org/jira/browse/SPARK-28258
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.3
> Reporter: Jose Luis Pedrosa
> Priority: Minor
>
> Currently the docker images generated by the distro uses openjdk8 based on alpine.
> This means that the version shipped of libssl is 1.1.1b-r1:
>
> {noformat}
> sh-4.4# apk list | grep ssl
> libssl1.1-1.1.1b-r1 x86_64 {openssl} (OpenSSL) [installed]
> {noformat}
> The hadoop distro ships wildfly-openssl-1.0.4.Final.jar, which is affected by [https://issues.jboss.org/browse/JBEAP-16425].
> This results on error running on the executor:
> {noformat}
> 2019-07-04 22:32:40,339 INFO openssl.SSL: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.1b 26 Feb 2019
> 2019-07-04 22:32:40,363 WARN streaming.FileStreamSink: Error while looking for metadata directory.
> Exception in thread "main" java.lang.NullPointerException
> at org.wildfly.openssl.CipherSuiteConverter.toJava(CipherSuiteConverter.java:284)
> {noformat}
> In my tests creating a Docker image with an updated version of wildfly, solves the issue: 1.0.7.Final
> Not sure if this is an Spark problem, if so, where would be the right place to solve it.
> It seems they may take care of it in Hadoop directly, but tickets are open.
> https://issues.apache.org/jira/browse/HADOOP-16410
> https://issues.apache.org/jira/browse/HADOOP-16405
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org