You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Hunter Lee <na...@gmail.com> on 2020/01/21 22:45:47 UTC

Separating ZkClient into a separate module

It seems that it would be beneficial to separate ZkClient and its
supporting classes from IOItec into a separate module. This work includes
doing away with IOItec library altogether. We've made some bug fixes to
their implementation and noticed that some of their newer versions aren't
backward-compatible.

This may require an addition of another module, say helix-common, to
resolve circular dependencies among the existing modules.

This will allow for more modularity and remove deployment cadence
dependency among ZK-related classes and helix-core classes. Additionally,
those who wish to use Helix's ZK-related features without explicitly using
helix-core or Task Framework could do so by simply importing zookeeper-api
module.

I'll be starting this work in a PR and let me know if you have any feedback!

Hunter

Re: Separating ZkClient into a separate module

Posted by Hunter Lee <na...@gmail.com>.
Let me also add that this is also about weaning Helix off of IOItec's
ZkClient library. We've made some bug fixes and also noticed some
backward-compatibility issues in that library, so the new module will keep
all Helix's ZkClient and its supporting classes, and remove the 101tec
import.

Hunter

On Thu, Jan 23, 2020 at 6:29 PM Hunter Lee <na...@gmail.com> wrote:

> Hey Kishore,
>
> 1. How we will achieve this is still an area to explore. What was meant by
> the description is that it will allow for the possibility - basically, if
> we were to achieve that, it's a lot easier to reason about if we have two
> different modules as opposed to some package in the same module. Creating
> separate releases might be one option.
> 2. That is correct. But Helix's ZkClient has some additional connectors to
> make it easier to emit MBean metrics via JMX, and this is a common use case
> in a few different production settings (e.g. LinkedIn, Uber, etc.). So
> there is value in using Helix's ZooKeeper API if that fits users' use case
> better.
>
> The new modules have been created in a pull request: zookeeper-api and
> helix-common. It is currently under review, and all Maven-based
> unit/integration tests came back green. Just moving files into a separate
> package is a trivial exercise, so upon discussing with a few of the PMC
> members/committers, I decided to go ahead with creating new modules.
>
> Care has been taken to move as little as possible, meaning the classes
> that were moved are bare minimum, and most of them have been left as a
> deprecated link (by way of subclassing) in helix-core to maintain as much
> backward-compatibility as possible. I am creating a more detailed and
> thorough test plan while the PR is being reviewed.
>
> Hunter
>
> On Thu, Jan 23, 2020 at 5:58 PM kishore g <g....@gmail.com> wrote:
>
>> I like the idea of removing the dependency on zkclient but have few
>> questions on the approach.
>>
>> "This will allow for more modularity and remove deployment cadence
>> dependency among ZK-related classes and helix-core classes. Additionally,
>> those who wish to use Helix's ZK-related features without explicitly using
>> helix-core or Task Framework could do so by simply importing zookeeper-api
>> module."
>>
>> - How can we achieve this? Are we planning to make release these modules
>> independent of each other.
>> - Users should use Curator if they want ZK related features (make it easy
>> to use ZK) Helix has a different goal (making it easy to build distributed
>> systems).
>>
>> - Why not just pull the code into a separate package within helix-core as
>> the first step and think of moving it to a separate module at a later time.
>>
>> Thanks
>> Kishore G
>>
>> On Thu, Jan 23, 2020 at 5:38 PM Hunter Lee <na...@gmail.com> wrote:
>>
>>> I added a short wiki describing this in GitHub:
>>> https://github.com/apache/helix/wiki/New-Modules-for-Apache-Helix
>>>
>>>
>>> On Tue, Jan 21, 2020 at 2:45 PM Hunter Lee <na...@gmail.com> wrote:
>>>
>>>> It seems that it would be beneficial to separate ZkClient and its
>>>> supporting classes from IOItec into a separate module. This work includes
>>>> doing away with IOItec library altogether. We've made some bug fixes to
>>>> their implementation and noticed that some of their newer versions aren't
>>>> backward-compatible.
>>>>
>>>> This may require an addition of another module, say helix-common, to
>>>> resolve circular dependencies among the existing modules.
>>>>
>>>> This will allow for more modularity and remove deployment cadence
>>>> dependency among ZK-related classes and helix-core classes. Additionally,
>>>> those who wish to use Helix's ZK-related features without explicitly using
>>>> helix-core or Task Framework could do so by simply importing zookeeper-api
>>>> module.
>>>>
>>>> I'll be starting this work in a PR and let me know if you have any
>>>> feedback!
>>>>
>>>> Hunter
>>>>
>>>

Re: Separating ZkClient into a separate module

Posted by Hunter Lee <na...@gmail.com>.
Hey Kishore,

1. How we will achieve this is still an area to explore. What was meant by
the description is that it will allow for the possibility - basically, if
we were to achieve that, it's a lot easier to reason about if we have two
different modules as opposed to some package in the same module. Creating
separate releases might be one option.
2. That is correct. But Helix's ZkClient has some additional connectors to
make it easier to emit MBean metrics via JMX, and this is a common use case
in a few different production settings (e.g. LinkedIn, Uber, etc.). So
there is value in using Helix's ZooKeeper API if that fits users' use case
better.

The new modules have been created in a pull request: zookeeper-api and
helix-common. It is currently under review, and all Maven-based
unit/integration tests came back green. Just moving files into a separate
package is a trivial exercise, so upon discussing with a few of the PMC
members/committers, I decided to go ahead with creating new modules.

Care has been taken to move as little as possible, meaning the classes that
were moved are bare minimum, and most of them have been left as a
deprecated link (by way of subclassing) in helix-core to maintain as much
backward-compatibility as possible. I am creating a more detailed and
thorough test plan while the PR is being reviewed.

Hunter

On Thu, Jan 23, 2020 at 5:58 PM kishore g <g....@gmail.com> wrote:

> I like the idea of removing the dependency on zkclient but have few
> questions on the approach.
>
> "This will allow for more modularity and remove deployment cadence
> dependency among ZK-related classes and helix-core classes. Additionally,
> those who wish to use Helix's ZK-related features without explicitly using
> helix-core or Task Framework could do so by simply importing zookeeper-api
> module."
>
> - How can we achieve this? Are we planning to make release these modules
> independent of each other.
> - Users should use Curator if they want ZK related features (make it easy
> to use ZK) Helix has a different goal (making it easy to build distributed
> systems).
>
> - Why not just pull the code into a separate package within helix-core as
> the first step and think of moving it to a separate module at a later time.
>
> Thanks
> Kishore G
>
> On Thu, Jan 23, 2020 at 5:38 PM Hunter Lee <na...@gmail.com> wrote:
>
>> I added a short wiki describing this in GitHub:
>> https://github.com/apache/helix/wiki/New-Modules-for-Apache-Helix
>>
>>
>> On Tue, Jan 21, 2020 at 2:45 PM Hunter Lee <na...@gmail.com> wrote:
>>
>>> It seems that it would be beneficial to separate ZkClient and its
>>> supporting classes from IOItec into a separate module. This work includes
>>> doing away with IOItec library altogether. We've made some bug fixes to
>>> their implementation and noticed that some of their newer versions aren't
>>> backward-compatible.
>>>
>>> This may require an addition of another module, say helix-common, to
>>> resolve circular dependencies among the existing modules.
>>>
>>> This will allow for more modularity and remove deployment cadence
>>> dependency among ZK-related classes and helix-core classes. Additionally,
>>> those who wish to use Helix's ZK-related features without explicitly using
>>> helix-core or Task Framework could do so by simply importing zookeeper-api
>>> module.
>>>
>>> I'll be starting this work in a PR and let me know if you have any
>>> feedback!
>>>
>>> Hunter
>>>
>>

Re: Separating ZkClient into a separate module

Posted by kishore g <g....@gmail.com>.
I like the idea of removing the dependency on zkclient but have few
questions on the approach.

"This will allow for more modularity and remove deployment cadence
dependency among ZK-related classes and helix-core classes. Additionally,
those who wish to use Helix's ZK-related features without explicitly using
helix-core or Task Framework could do so by simply importing zookeeper-api
module."

- How can we achieve this? Are we planning to make release these modules
independent of each other.
- Users should use Curator if they want ZK related features (make it easy
to use ZK) Helix has a different goal (making it easy to build distributed
systems).

- Why not just pull the code into a separate package within helix-core as
the first step and think of moving it to a separate module at a later time.

Thanks
Kishore G

On Thu, Jan 23, 2020 at 5:38 PM Hunter Lee <na...@gmail.com> wrote:

> I added a short wiki describing this in GitHub:
> https://github.com/apache/helix/wiki/New-Modules-for-Apache-Helix
>
>
> On Tue, Jan 21, 2020 at 2:45 PM Hunter Lee <na...@gmail.com> wrote:
>
>> It seems that it would be beneficial to separate ZkClient and its
>> supporting classes from IOItec into a separate module. This work includes
>> doing away with IOItec library altogether. We've made some bug fixes to
>> their implementation and noticed that some of their newer versions aren't
>> backward-compatible.
>>
>> This may require an addition of another module, say helix-common, to
>> resolve circular dependencies among the existing modules.
>>
>> This will allow for more modularity and remove deployment cadence
>> dependency among ZK-related classes and helix-core classes. Additionally,
>> those who wish to use Helix's ZK-related features without explicitly using
>> helix-core or Task Framework could do so by simply importing zookeeper-api
>> module.
>>
>> I'll be starting this work in a PR and let me know if you have any
>> feedback!
>>
>> Hunter
>>
>

Re: Separating ZkClient into a separate module

Posted by Hunter Lee <na...@gmail.com>.
I added a short wiki describing this in GitHub:
https://github.com/apache/helix/wiki/New-Modules-for-Apache-Helix


On Tue, Jan 21, 2020 at 2:45 PM Hunter Lee <na...@gmail.com> wrote:

> It seems that it would be beneficial to separate ZkClient and its
> supporting classes from IOItec into a separate module. This work includes
> doing away with IOItec library altogether. We've made some bug fixes to
> their implementation and noticed that some of their newer versions aren't
> backward-compatible.
>
> This may require an addition of another module, say helix-common, to
> resolve circular dependencies among the existing modules.
>
> This will allow for more modularity and remove deployment cadence
> dependency among ZK-related classes and helix-core classes. Additionally,
> those who wish to use Helix's ZK-related features without explicitly using
> helix-core or Task Framework could do so by simply importing zookeeper-api
> module.
>
> I'll be starting this work in a PR and let me know if you have any
> feedback!
>
> Hunter
>