You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Norbert Kalmár (Jira)" <ji...@apache.org> on 2019/10/07 09:52:00 UTC
[jira] [Comment Edited] (HADOOP-16579) Upgrade to Apache Curator
4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945744#comment-16945744 ]
Norbert Kalmár edited comment on HADOOP-16579 at 10/7/19 9:51 AM:
------------------------------------------------------------------
Quick analysis: the ZK test failing creates a TestingServer which is implemented in Curator. Curator supposedly decides based on ZK version on the classpath if it runs in 3.5 or 3.4 mode. Somehow it looks like it decides to run in 3.5 mode even though there is a 3.4 ZK on the classpath (that's why ZK is excluded in curator's dependency).
This requires further analysis, but just a question: [~weichiu], is there a chance Hadoop would upgrade to ZooKeeper 3.5? Specifically I would suggest to wait for 3.5.6, as it fixes a CVE and also fixes some issues that came up, including upgrade from 3.4.x to 3.5(.5). It is already on vote to release.
was (Author: nkalmar):
Quick analysis: the ZK test failing creates a TestingServer which is implemented in Curator. Curator supposedly decides based on ZK version on the classpath if it runs in 3.5 or 3.4 mode. Somehow it looks like it decides to run in 3.5 mode even though there is a 3.4 ZK on the classpath (that's why ZK is excluded in curator's dependency).
This requires further analysis, but just a question: Is there a chance Hadoop would upgrade to ZooKeeper 3.5? Specifically I would suggest to wait for 3.5.6, as it fixes a CVE and also fixes some issues that came up, including upgrade from 3.4.x to 3.5(.5). It is already on vote to release.
> Upgrade to Apache Curator 4.2.0 in Hadoop
> -----------------------------------------
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Mate Szalay-Beko
> Assignee: Norbert Kalmár
> Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
> - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0)
> - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org