You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nikita ROUSSEAU (Jira)" <ji...@apache.org> on 2021/06/30 15:29:00 UTC
[jira] [Commented] (SPARK-35954) Upgrade Apache Curator Dependency
to 4.2.0
[ https://issues.apache.org/jira/browse/SPARK-35954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372079#comment-17372079 ]
Nikita ROUSSEAU commented on SPARK-35954:
-----------------------------------------
Since it is my first contribution, do not hesitate to point me guidelines about this issue :)
> Upgrade Apache Curator Dependency to 4.2.0
> ------------------------------------------
>
> Key: SPARK-35954
> URL: https://issues.apache.org/jira/browse/SPARK-35954
> Project: Spark
> Issue Type: Dependency upgrade
> Components: Deploy
> Affects Versions: 3.1.2
> Environment: * OS: Linux
> * JAVA: 1.8.0_292
> * {color:#FF0000}*hadoop-3.3.1*{color}
>
> Reporter: Nikita ROUSSEAU
> Priority: Major
>
> +Abstract :+ as a Spark Cluster Administrator, I want to connect spark masters deployed in HA/mode to Zookeeper over SSL/TLS, so that my network traffic is ciphered between y components. ([https://spark.apache.org/docs/latest/spark-standalone.html#standby-masters-with-zookeeper] )
>
> With the release of Hadoop 3.3.1, ZKFC libraries and their dependencies were updated : it is now possible to connect ZKFC to ZooKeeper over TLS.
>
> +Note:+ TLS is possible with Zookeeper Server Version >= 3.5.6 ([https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk] )
>
> Spark 3.2.0 aims to support Hadoop 3.3.1 ; this Hadoop release bundles the following shared libraries :
> * curator-client-4.2.0.jar
> * curator-framework-4.2.0.jar
> * curator-recipes-4.2.0.jar
> * zookeeper-3.5.6.jar
> * zookeeper-jute-3.5.6.jar
>
> Currently, Spark dependency is set to 2.13.0 for the Currator framework ([https://github.com/apache/spark/blob/master/pom.xml#L127).|https://github.com/apache/spark/blob/master/pom.xml#L127)]
>
> It would be great to update "curator-*" dependencies to 4.2.0 in order to be compatible with shared jars of the hadoop stack.
> Moreover, it will allow administrators to connect Spark Masters to ZooKeeper over TLS.
>
> Some patches will be required, such as :
> * [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkCuratorUtil.scala#L51]
>
> I will try to prepare a MR for this.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org