[GitHub] [incubator-livy] symat edited a comment on issue #262: [LIVY-717] introduce maven property to set ZooKeeper version

[GitBox <gi...@apache.org>]

symat edited a comment on issue #262: [LIVY-717] introduce maven property to set ZooKeeper version
URL: https://github.com/apache/incubator-livy/pull/262#issuecomment-558044949
 
 
   Thanks for the comments!
   
   >  even if we set the curator.version properly, the zk version from the dependency of curator maybe not what we really want.
   
   yes, that is true as well. The new Curator 4.2.0 pulls ZK 3.5.4-beta, and we don't want to use a beta version in production. (the last stable official ZK release is 3.5.6)
   The other problem is, that ZK comes not only from curator, but also from hadoop, so if we want to be sure that we only have a single ZooKeeper version on our classpath, then the best is to exclude it from both Curator and Hadoop and load our own version.
   
   
   > can it really be worked after version is changed
   
   it depends on the combination of versions. You are right, that it won't work if you only change ZooKeeper version. Curator 2.x is only compatible with ZooKeeper 3.4.x, while Curator 3.x is compatible only with ZooKeeper 3.5.x. The new Curator 4.x is [said to be compatible](https://curator.apache.org/zk-compatibility.html) with both ZooKeeper versions, but we found during testing, that it is not necessarily works with the old ZooKeeper. (also one thing is compile time compatibility and an other the run-time one) So we found, that the best is to use the latest Curator only with the new ZooKeeper. (it is good to know, that on the Hadoop trunk we already have Curator 4.2.0 and ZooKeeper 3.5.6, but I am not aware of any released Hadoop version yet that contains this change)
   
   So the trick is to make sure to use proper (compatible) Curator-Hadoop-ZooKeeper versions. 
   - E.g. the current combination used by Livy: Curator 2.7.1, ZooKeeper 3.4.6, Hadoop 2.7.3 
   - We are testing with the following versions: Curator 4.2.0, ZooKeeper 3.5.6, Hadoop 3.0.2
   
   When you try to build Livy with the new Curator/ZooKeeper, then you will face an other small issue (one mock object needs to be changed in the integration tests), but other than that, Livy is working for us. Assuming you exclude the old ZooKeeper that comes from Hadoop.
   
   I am mainly a ZooKeeper contributor, so I don't want to change the default Curator / ZooKeeper version in Livy, I think that is something the Livy community should decide / schedule. But I think it is an important feature to have the ability to explicitly set the ZooKeeper and Curator versions in Livy. So at least the community can test / experiment with the new third party versions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services