Helix V2 0.7.0

[kishore g <g....@gmail.com>]

Hi,

We have made lot of good changes in 0.7.0 and improved the api's. However,
I think it is not  easy/intuitive for a new user. One of the problems with
0.7.0 code is that we tried to maintain complete backward compatibility
with respect to both logical api's and physical layout on zookeeper.

Trying to maintain the logical api backward compatibility has definitely
caused some pain and did not allow us to do the right thing in 0.7.0 and it
has made our code base huge. The reasoning here is - when we built Helix
((almost 3 years back), we did not anticipate Helix being used  in other
systems. So our main focus was minimal code and to make sure it works for
the use case we had. We did not gather much feedback from users.

However, we are seeing the usage grow and while everyone agrees that the
high level concepts are good, it is apparent that api's are making people
shy away from Helix. I would even say some of the terminologies are
confusing until you spend quite some time with Helix.

I want to see what others think about this.

We have two options going forward

Option1: Continue to maintain backward compatibility and improving the api's
Option2: Break the api compatibility and call it Helix V2. We redesign our
api's and make it more intuitive and easier/flexible to use.

I think the core functionality and design is great and don't see much
change needed in the architecture (Do let us know if you think we need any
change). What is lacking is documentation and a simple set of api's that
are intuitive.

While Option 1 is great for existing users, I prefer Option2. We will
redesign the 0.7.0 api's without maintaining backward compatibility. Lot of
work has already been done in 0.7.0, so we are not that far. This also
gives chance to the community to contribute and provide
suggestions/feedback/ideas.

For existing users, we will continue to maintain 0.6.2 and continue to make
critical bug fixes. But no new features will be added.

Thoughts ?

thanks,
Kishore G

RE: Helix V2 0.7.0

[Kanak Biscuitwala <ka...@hotmail.com>]

Sorry, I meant module.

----------------------------------------
> From: kanak.b@hotmail.com
> To: dev@helix.apache.org
> CC: user@helix.apache.org
> Subject: RE: Helix V2 0.7.0
> Date: Sat, 22 Feb 2014 10:45:50 -0800
>
> +1 for helix-api as a package that only contains interfaces.
>
> ----------------------------------------
>> Date: Sat, 22 Feb 2014 10:44:19 -0800
>> Subject: Re: Helix V2 0.7.0
>> From: g.kishore@gmail.com
>> To: dev@helix.apache.org
>> CC: user@helix.apache.org
>>
>> Good suggestion, I think Vinayak had suggestions on similar lines but I
>> think he suggestion writing high level api's on top of existing api's, as
>> opposed to changing the changing the underlying implementation. What that
>> means is we leave the existing api's as is but add high level api. What
>> this means is we can keep the core as is but add helix-api module that
>> provides high level interfaces. We can re-use existing implementation under
>> the hood. I like the idea of creating a separate module helix-api that is
>> decoupled from implementation. This will enforce us to have no dependency
>> on zookeeper in api and allows one to plug other forms of storage. For
>> example many usecases dont need the level of consistency we get from
>> Zookeeper.
>>
>> Looks like we might have to make decisions on a case by case basis.
>>
>> Can we make a decision of having a helix-api package.
>>
>> thanks,
>> Kishore G
>>
>>
>> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
>>
>>> The biggest problem is that we pretty much have to backport bug fixes
>>> forever if we completely break compatibility. Here's what I think we could
>>> do:
>>>
>>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>>> stuff)
>>> 2) Keep the interfaces for the most common currently used APIs
>>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
>>> REST), but use our new implementations underneath (HelixConnection,
>>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>>> This means there will be breakages for people who used the implementation
>>> classes directly, but the changes would be minor.
>>> 3) Provide adapters between common new and old APIs
>>> (HelixConnectionAdapter is an example)
>>> 4) Maintain the model package as-is
>>>
>>> This way, there's a clear transition path from one branch to another and
>>> we can eventually end support for the old branch.
>>> ________________________________
>>>> Date: Sat, 22 Feb 2014 07:33:10 -0800
>>>> Subject: Helix V2 0.7.0
>>>> From: g.kishore@gmail.com
>>>> To: dev@helix.apache.org; user@helix.apache.org
>>>>
>>>> Hi,
>>>>
>>>> We have made lot of good changes in 0.7.0 and improved the api's.
>>>> However, I think it is not easy/intuitive for a new user. One of the
>>>> problems with 0.7.0 code is that we tried to maintain complete backward
>>>> compatibility with respect to both logical api's and physical layout on
>>>> zookeeper.
>>>>
>>>> Trying to maintain the logical api backward compatibility has
>>>> definitely caused some pain and did not allow us to do the right thing
>>>> in 0.7.0 and it has made our code base huge. The reasoning here is -
>>>> when we built Helix ((almost 3 years back), we did not anticipate Helix
>>>> being used in other systems. So our main focus was minimal code and to
>>>> make sure it works for the use case we had. We did not gather much
>>>> feedback from users.
>>>>
>>>> However, we are seeing the usage grow and while everyone agrees that
>>>> the high level concepts are good, it is apparent that api's are making
>>>> people shy away from Helix. I would even say some of the terminologies
>>>> are confusing until you spend quite some time with Helix.
>>>>
>>>> I want to see what others think about this.
>>>>
>>>> We have two options going forward
>>>>
>>>> Option1: Continue to maintain backward compatibility and improving the
>>> api's
>>>> Option2: Break the api compatibility and call it Helix V2. We redesign
>>>> our api's and make it more intuitive and easier/flexible to use.
>>>>
>>>> I think the core functionality and design is great and don't see much
>>>> change needed in the architecture (Do let us know if you think we need
>>>> any change). What is lacking is documentation and a simple set of api's
>>>> that are intuitive.
>>>>
>>>> While Option 1 is great for existing users, I prefer Option2. We will
>>>> redesign the 0.7.0 api's without maintaining backward compatibility.
>>>> Lot of work has already been done in 0.7.0, so we are not that far.
>>>> This also gives chance to the community to contribute and provide
>>>> suggestions/feedback/ideas.
>>>>
>>>> For existing users, we will continue to maintain 0.6.2 and continue to
>>>> make critical bug fixes. But no new features will be added.
>>>>
>>>> Thoughts ?
>>>>
>>>> thanks,
>>>> Kishore G
>>> 		 	   		  

RE: Helix V2 0.7.0

[Kanak Biscuitwala <ka...@hotmail.com>]

Sorry, I meant module.

----------------------------------------
> From: kanak.b@hotmail.com
> To: dev@helix.apache.org
> CC: user@helix.apache.org
> Subject: RE: Helix V2 0.7.0
> Date: Sat, 22 Feb 2014 10:45:50 -0800
>
> +1 for helix-api as a package that only contains interfaces.
>
> ----------------------------------------
>> Date: Sat, 22 Feb 2014 10:44:19 -0800
>> Subject: Re: Helix V2 0.7.0
>> From: g.kishore@gmail.com
>> To: dev@helix.apache.org
>> CC: user@helix.apache.org
>>
>> Good suggestion, I think Vinayak had suggestions on similar lines but I
>> think he suggestion writing high level api's on top of existing api's, as
>> opposed to changing the changing the underlying implementation. What that
>> means is we leave the existing api's as is but add high level api. What
>> this means is we can keep the core as is but add helix-api module that
>> provides high level interfaces. We can re-use existing implementation under
>> the hood. I like the idea of creating a separate module helix-api that is
>> decoupled from implementation. This will enforce us to have no dependency
>> on zookeeper in api and allows one to plug other forms of storage. For
>> example many usecases dont need the level of consistency we get from
>> Zookeeper.
>>
>> Looks like we might have to make decisions on a case by case basis.
>>
>> Can we make a decision of having a helix-api package.
>>
>> thanks,
>> Kishore G
>>
>>
>> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
>>
>>> The biggest problem is that we pretty much have to backport bug fixes
>>> forever if we completely break compatibility. Here's what I think we could
>>> do:
>>>
>>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>>> stuff)
>>> 2) Keep the interfaces for the most common currently used APIs
>>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
>>> REST), but use our new implementations underneath (HelixConnection,
>>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>>> This means there will be breakages for people who used the implementation
>>> classes directly, but the changes would be minor.
>>> 3) Provide adapters between common new and old APIs
>>> (HelixConnectionAdapter is an example)
>>> 4) Maintain the model package as-is
>>>
>>> This way, there's a clear transition path from one branch to another and
>>> we can eventually end support for the old branch.
>>> ________________________________
>>>> Date: Sat, 22 Feb 2014 07:33:10 -0800
>>>> Subject: Helix V2 0.7.0
>>>> From: g.kishore@gmail.com
>>>> To: dev@helix.apache.org; user@helix.apache.org
>>>>
>>>> Hi,
>>>>
>>>> We have made lot of good changes in 0.7.0 and improved the api's.
>>>> However, I think it is not easy/intuitive for a new user. One of the
>>>> problems with 0.7.0 code is that we tried to maintain complete backward
>>>> compatibility with respect to both logical api's and physical layout on
>>>> zookeeper.
>>>>
>>>> Trying to maintain the logical api backward compatibility has
>>>> definitely caused some pain and did not allow us to do the right thing
>>>> in 0.7.0 and it has made our code base huge. The reasoning here is -
>>>> when we built Helix ((almost 3 years back), we did not anticipate Helix
>>>> being used in other systems. So our main focus was minimal code and to
>>>> make sure it works for the use case we had. We did not gather much
>>>> feedback from users.
>>>>
>>>> However, we are seeing the usage grow and while everyone agrees that
>>>> the high level concepts are good, it is apparent that api's are making
>>>> people shy away from Helix. I would even say some of the terminologies
>>>> are confusing until you spend quite some time with Helix.
>>>>
>>>> I want to see what others think about this.
>>>>
>>>> We have two options going forward
>>>>
>>>> Option1: Continue to maintain backward compatibility and improving the
>>> api's
>>>> Option2: Break the api compatibility and call it Helix V2. We redesign
>>>> our api's and make it more intuitive and easier/flexible to use.
>>>>
>>>> I think the core functionality and design is great and don't see much
>>>> change needed in the architecture (Do let us know if you think we need
>>>> any change). What is lacking is documentation and a simple set of api's
>>>> that are intuitive.
>>>>
>>>> While Option 1 is great for existing users, I prefer Option2. We will
>>>> redesign the 0.7.0 api's without maintaining backward compatibility.
>>>> Lot of work has already been done in 0.7.0, so we are not that far.
>>>> This also gives chance to the community to contribute and provide
>>>> suggestions/feedback/ideas.
>>>>
>>>> For existing users, we will continue to maintain 0.6.2 and continue to
>>>> make critical bug fixes. But no new features will be added.
>>>>
>>>> Thoughts ?
>>>>
>>>> thanks,
>>>> Kishore G
>>> 		 	   		  

RE: Helix V2 0.7.0

[Kanak Biscuitwala <ka...@hotmail.com>]

+1 for helix-api as a package that only contains interfaces.

----------------------------------------
> Date: Sat, 22 Feb 2014 10:44:19 -0800
> Subject: Re: Helix V2 0.7.0
> From: g.kishore@gmail.com
> To: dev@helix.apache.org
> CC: user@helix.apache.org
>
> Good suggestion, I think Vinayak had suggestions on similar lines but I
> think he suggestion writing high level api's on top of existing api's, as
> opposed to changing the changing the underlying implementation. What that
> means is we leave the existing api's as is but add high level api. What
> this means is we can keep the core as is but add helix-api module that
> provides high level interfaces. We can re-use existing implementation under
> the hood. I like the idea of creating a separate module helix-api that is
> decoupled from implementation. This will enforce us to have no dependency
> on zookeeper in api and allows one to plug other forms of storage. For
> example many usecases dont need the level of consistency we get from
> Zookeeper.
>
> Looks like we might have to make decisions on a case by case basis.
>
> Can we make a decision of having a helix-api package.
>
> thanks,
> Kishore G
>
>
> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
>
>> The biggest problem is that we pretty much have to backport bug fixes
>> forever if we completely break compatibility. Here's what I think we could
>> do:
>>
>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>> stuff)
>> 2) Keep the interfaces for the most common currently used APIs
>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
>> REST), but use our new implementations underneath (HelixConnection,
>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>> This means there will be breakages for people who used the implementation
>> classes directly, but the changes would be minor.
>> 3) Provide adapters between common new and old APIs
>> (HelixConnectionAdapter is an example)
>> 4) Maintain the model package as-is
>>
>> This way, there's a clear transition path from one branch to another and
>> we can eventually end support for the old branch.
>> ________________________________
>>> Date: Sat, 22 Feb 2014 07:33:10 -0800
>>> Subject: Helix V2 0.7.0
>>> From: g.kishore@gmail.com
>>> To: dev@helix.apache.org; user@helix.apache.org
>>>
>>> Hi,
>>>
>>> We have made lot of good changes in 0.7.0 and improved the api's.
>>> However, I think it is not easy/intuitive for a new user. One of the
>>> problems with 0.7.0 code is that we tried to maintain complete backward
>>> compatibility with respect to both logical api's and physical layout on
>>> zookeeper.
>>>
>>> Trying to maintain the logical api backward compatibility has
>>> definitely caused some pain and did not allow us to do the right thing
>>> in 0.7.0 and it has made our code base huge. The reasoning here is -
>>> when we built Helix ((almost 3 years back), we did not anticipate Helix
>>> being used in other systems. So our main focus was minimal code and to
>>> make sure it works for the use case we had. We did not gather much
>>> feedback from users.
>>>
>>> However, we are seeing the usage grow and while everyone agrees that
>>> the high level concepts are good, it is apparent that api's are making
>>> people shy away from Helix. I would even say some of the terminologies
>>> are confusing until you spend quite some time with Helix.
>>>
>>> I want to see what others think about this.
>>>
>>> We have two options going forward
>>>
>>> Option1: Continue to maintain backward compatibility and improving the
>> api's
>>> Option2: Break the api compatibility and call it Helix V2. We redesign
>>> our api's and make it more intuitive and easier/flexible to use.
>>>
>>> I think the core functionality and design is great and don't see much
>>> change needed in the architecture (Do let us know if you think we need
>>> any change). What is lacking is documentation and a simple set of api's
>>> that are intuitive.
>>>
>>> While Option 1 is great for existing users, I prefer Option2. We will
>>> redesign the 0.7.0 api's without maintaining backward compatibility.
>>> Lot of work has already been done in 0.7.0, so we are not that far.
>>> This also gives chance to the community to contribute and provide
>>> suggestions/feedback/ideas.
>>>
>>> For existing users, we will continue to maintain 0.6.2 and continue to
>>> make critical bug fixes. But no new features will be added.
>>>
>>> Thoughts ?
>>>
>>> thanks,
>>> Kishore G
>> 		 	   		  

RE: Helix V2 0.7.0

[Kanak Biscuitwala <ka...@hotmail.com>]

+1 for helix-api as a package that only contains interfaces.

----------------------------------------
> Date: Sat, 22 Feb 2014 10:44:19 -0800
> Subject: Re: Helix V2 0.7.0
> From: g.kishore@gmail.com
> To: dev@helix.apache.org
> CC: user@helix.apache.org
>
> Good suggestion, I think Vinayak had suggestions on similar lines but I
> think he suggestion writing high level api's on top of existing api's, as
> opposed to changing the changing the underlying implementation. What that
> means is we leave the existing api's as is but add high level api. What
> this means is we can keep the core as is but add helix-api module that
> provides high level interfaces. We can re-use existing implementation under
> the hood. I like the idea of creating a separate module helix-api that is
> decoupled from implementation. This will enforce us to have no dependency
> on zookeeper in api and allows one to plug other forms of storage. For
> example many usecases dont need the level of consistency we get from
> Zookeeper.
>
> Looks like we might have to make decisions on a case by case basis.
>
> Can we make a decision of having a helix-api package.
>
> thanks,
> Kishore G
>
>
> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
>
>> The biggest problem is that we pretty much have to backport bug fixes
>> forever if we completely break compatibility. Here's what I think we could
>> do:
>>
>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>> stuff)
>> 2) Keep the interfaces for the most common currently used APIs
>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
>> REST), but use our new implementations underneath (HelixConnection,
>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>> This means there will be breakages for people who used the implementation
>> classes directly, but the changes would be minor.
>> 3) Provide adapters between common new and old APIs
>> (HelixConnectionAdapter is an example)
>> 4) Maintain the model package as-is
>>
>> This way, there's a clear transition path from one branch to another and
>> we can eventually end support for the old branch.
>> ________________________________
>>> Date: Sat, 22 Feb 2014 07:33:10 -0800
>>> Subject: Helix V2 0.7.0
>>> From: g.kishore@gmail.com
>>> To: dev@helix.apache.org; user@helix.apache.org
>>>
>>> Hi,
>>>
>>> We have made lot of good changes in 0.7.0 and improved the api's.
>>> However, I think it is not easy/intuitive for a new user. One of the
>>> problems with 0.7.0 code is that we tried to maintain complete backward
>>> compatibility with respect to both logical api's and physical layout on
>>> zookeeper.
>>>
>>> Trying to maintain the logical api backward compatibility has
>>> definitely caused some pain and did not allow us to do the right thing
>>> in 0.7.0 and it has made our code base huge. The reasoning here is -
>>> when we built Helix ((almost 3 years back), we did not anticipate Helix
>>> being used in other systems. So our main focus was minimal code and to
>>> make sure it works for the use case we had. We did not gather much
>>> feedback from users.
>>>
>>> However, we are seeing the usage grow and while everyone agrees that
>>> the high level concepts are good, it is apparent that api's are making
>>> people shy away from Helix. I would even say some of the terminologies
>>> are confusing until you spend quite some time with Helix.
>>>
>>> I want to see what others think about this.
>>>
>>> We have two options going forward
>>>
>>> Option1: Continue to maintain backward compatibility and improving the
>> api's
>>> Option2: Break the api compatibility and call it Helix V2. We redesign
>>> our api's and make it more intuitive and easier/flexible to use.
>>>
>>> I think the core functionality and design is great and don't see much
>>> change needed in the architecture (Do let us know if you think we need
>>> any change). What is lacking is documentation and a simple set of api's
>>> that are intuitive.
>>>
>>> While Option 1 is great for existing users, I prefer Option2. We will
>>> redesign the 0.7.0 api's without maintaining backward compatibility.
>>> Lot of work has already been done in 0.7.0, so we are not that far.
>>> This also gives chance to the community to contribute and provide
>>> suggestions/feedback/ideas.
>>>
>>> For existing users, we will continue to maintain 0.6.2 and continue to
>>> make critical bug fixes. But no new features will be added.
>>>
>>> Thoughts ?
>>>
>>> thanks,
>>> Kishore G
>> 		 	   		  

Re: Helix V2 0.7.0

[Matthieu Morel <mm...@apache.org>]

Hi,

I'm also in favor of improving the API even though that involves breaking backward compatibility. 
it looks like the 0.7.0 version captures concepts and usage much better.

As Json pointed out, most compatibility issues should be detected by swapping libs and fixing compilation errors.

But that also means a few issues might remain. In particular, if there are semantic changes, it would be good to have 
them pointed out somewhere (small wiki page or something like that, with a mapping old concepts/API -> new concepts/API).

Thanks!

Matthieu

On Feb 23, 2014, at 02:20 , Zhen Zhang <ne...@gmail.com> wrote:

> +1 for helix-api package that contains public interface only and decouples
> from implementation. The api package should includes Accessor, HelixAdmin,
> HelixManager, HelixConnection, MessagingService, etc. We can start by
> listing all the necessary interfaces. I am not sure if we have a clear
> separation between API and SPI. For example, we are using Accessor and
> MessagingService as both API and SPI. Nevertheless, we need to stick to the
> interfaces inside Helix also. Current Helix code base has many places where
> we use concrete implementations instead of interfaces.
> 
> For backward compatibility, I agree with Kishore that as long as we are
> compatible with physical model, most of the compatibility issues should be
> solved at compile time. We might have to keep the old interfaces for
> current uses, but need to clean them up. For example, we can't get rid of
> HelixManager totally and enforce people to switch to the new
> HelixConnectiont/HelixParticipant, but we should remove/change confusing
> api's. Users should be able to easily fix the compatibility issues at
> compile time.
> 
> Thanks,
> Jason
> 
> 
> 
> 
> On Sat, Feb 22, 2014 at 11:48 AM, <bo...@schulman.com> wrote:
> 
>> Here's how I'd evaluate what to do, consider these 4 sub-problems:
>> 
>> 1. Ease of learning
>> - are the APIs accessible for a first-timer?  Like it or not, that's an
>> important factor in getting wider adoption.  Necessary but not sufficient.
>> - a separate helix-api package is a good way to keep it as simple as
>> possible, because then you only have to understand that to build a system
>> using Helix
>> - also, a new package gives us an opportunity to change terminology if
>> that will help new users
>> 
>> 2. Functionality
>> - can Helix do what I want?
>> - as long as the APIs, say in a helix-api package don't hide things a
>> system developer needs, this isn't a tradeoff
>> - careful terminology might help here
>> 
>> 3. Maintainability/Extensibility
>> - for a Helix developer, is it possible to add functionality?  That only
>> matters if there are gaps in (2), it's hard to know if that will keep
>> coming up as more use-cases emerge
>> - for a Helix committer, is the cost of maintenance too high, particularly
>> to maintain backward compatibility
>> 
>> 4. Backward compatibility
>> - we don't want to leave early adopters out in the cold
>> 
>> Given that framework for evaluation, the helix-api package looks like the
>> way to go.  I'm not quite sure how the old branch would be retired, other
>> than polling users and making a judgment call.  There are more people using
>> Helix than we know about, as I continue to run into people using it that we
>> didn't know about.
>> 
>> The key is to get the APIs right, easier said than done.  That's my $0.02.
>> 
>> Thanks,
>> Bob
>> 
>> 
>> Good suggestion, I think Vinayak had suggestions on similar lines but I
>>> think he suggestion writing high level api's on top of existing api's, as
>>> opposed to changing the  changing the underlying implementation. What that
>>> means is we leave the existing api's as is but add high level api. What
>>> this means is we can keep the core as is but add helix-api module that
>>> provides high level interfaces. We can re-use existing implementation
>>> under
>>> the hood. I like the idea of creating a separate module helix-api that is
>>> decoupled from implementation. This will enforce us to have no dependency
>>> on zookeeper in api and allows one to plug other forms of storage. For
>>> example many usecases dont need the level of consistency we get from
>>> Zookeeper.
>>> 
>>> Looks like we might have to make decisions on a case by case basis.
>>> 
>>> Can we make a decision of having a helix-api package.
>>> 
>>> thanks,
>>> Kishore G
>>> 
>>> 
>>> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <kanak.b@hotmail.com
>>>> wrote:
>>> 
>>> The biggest problem is that we pretty much have to backport bug fixes
>>>> forever if we completely break compatibility. Here's what I think we
>>>> could
>>>> do:
>>>> 
>>>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>>>> stuff)
>>>> 2) Keep the interfaces for the most common currently used APIs
>>>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command
>>>> line,
>>>> REST), but use our new implementations underneath (HelixConnection,
>>>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>>>> This means there will be breakages for people who used the implementation
>>>> classes directly, but the changes would be minor.
>>>> 3) Provide adapters between common new and old APIs
>>>> (HelixConnectionAdapter is an example)
>>>> 4) Maintain the model package as-is
>>>> 
>>>> This way, there's a clear transition path from one branch to another and
>>>> we can eventually end support for the old branch.
>>>> ________________________________
>>>>> Date: Sat, 22 Feb 2014 07:33:10 -0800
>>>>> Subject: Helix V2 0.7.0
>>>>> From: g.kishore@gmail.com
>>>>> To: dev@helix.apache.org; user@helix.apache.org
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> We have made lot of good changes in 0.7.0 and improved the api's.
>>>>> However, I think it is not easy/intuitive for a new user. One of the
>>>>> problems with 0.7.0 code is that we tried to maintain complete backward
>>>>> compatibility with respect to both logical api's and physical layout on
>>>>> zookeeper.
>>>>> 
>>>>> Trying to maintain the logical api backward compatibility has
>>>>> definitely caused some pain and did not allow us to do the right thing
>>>>> in 0.7.0 and it has made our code base huge. The reasoning here is -
>>>>> when we built Helix ((almost 3 years back), we did not anticipate Helix
>>>>> being used in other systems. So our main focus was minimal code and to
>>>>> make sure it works for the use case we had. We did not gather much
>>>>> feedback from users.
>>>>> 
>>>>> However, we are seeing the usage grow and while everyone agrees that
>>>>> the high level concepts are good, it is apparent that api's are making
>>>>> people shy away from Helix. I would even say some of the terminologies
>>>>> are confusing until you spend quite some time with Helix.
>>>>> 
>>>>> I want to see what others think about this.
>>>>> 
>>>>> We have two options going forward
>>>>> 
>>>>> Option1: Continue to maintain backward compatibility and improving the
>>>> api's
>>>>> Option2: Break the api compatibility and call it Helix V2. We redesign
>>>>> our api's and make it more intuitive and easier/flexible to use.
>>>>> 
>>>>> I think the core functionality and design is great and don't see much
>>>>> change needed in the architecture (Do let us know if you think we need
>>>>> any change). What is lacking is documentation and a simple set of api's
>>>>> that are intuitive.
>>>>> 
>>>>> While Option 1 is great for existing users, I prefer Option2. We will
>>>>> redesign the 0.7.0 api's without maintaining backward compatibility.
>>>>> Lot of work has already been done in 0.7.0, so we are not that far.
>>>>> This also gives chance to the community to contribute and provide
>>>>> suggestions/feedback/ideas.
>>>>> 
>>>>> For existing users, we will continue to maintain 0.6.2 and continue to
>>>>> make critical bug fixes. But no new features will be added.
>>>>> 
>>>>> Thoughts ?
>>>>> 
>>>>> thanks,
>>>>> Kishore G
>>>> 
>>>> 
>>> 
>> 
>> 
>> 

Re: Helix V2 0.7.0

[Matthieu Morel <mm...@apache.org>]

Hi,

I'm also in favor of improving the API even though that involves breaking backward compatibility. 
it looks like the 0.7.0 version captures concepts and usage much better.

As Json pointed out, most compatibility issues should be detected by swapping libs and fixing compilation errors.

But that also means a few issues might remain. In particular, if there are semantic changes, it would be good to have 
them pointed out somewhere (small wiki page or something like that, with a mapping old concepts/API -> new concepts/API).

Thanks!

Matthieu

On Feb 23, 2014, at 02:20 , Zhen Zhang <ne...@gmail.com> wrote:

> +1 for helix-api package that contains public interface only and decouples
> from implementation. The api package should includes Accessor, HelixAdmin,
> HelixManager, HelixConnection, MessagingService, etc. We can start by
> listing all the necessary interfaces. I am not sure if we have a clear
> separation between API and SPI. For example, we are using Accessor and
> MessagingService as both API and SPI. Nevertheless, we need to stick to the
> interfaces inside Helix also. Current Helix code base has many places where
> we use concrete implementations instead of interfaces.
> 
> For backward compatibility, I agree with Kishore that as long as we are
> compatible with physical model, most of the compatibility issues should be
> solved at compile time. We might have to keep the old interfaces for
> current uses, but need to clean them up. For example, we can't get rid of
> HelixManager totally and enforce people to switch to the new
> HelixConnectiont/HelixParticipant, but we should remove/change confusing
> api's. Users should be able to easily fix the compatibility issues at
> compile time.
> 
> Thanks,
> Jason
> 
> 
> 
> 
> On Sat, Feb 22, 2014 at 11:48 AM, <bo...@schulman.com> wrote:
> 
>> Here's how I'd evaluate what to do, consider these 4 sub-problems:
>> 
>> 1. Ease of learning
>> - are the APIs accessible for a first-timer?  Like it or not, that's an
>> important factor in getting wider adoption.  Necessary but not sufficient.
>> - a separate helix-api package is a good way to keep it as simple as
>> possible, because then you only have to understand that to build a system
>> using Helix
>> - also, a new package gives us an opportunity to change terminology if
>> that will help new users
>> 
>> 2. Functionality
>> - can Helix do what I want?
>> - as long as the APIs, say in a helix-api package don't hide things a
>> system developer needs, this isn't a tradeoff
>> - careful terminology might help here
>> 
>> 3. Maintainability/Extensibility
>> - for a Helix developer, is it possible to add functionality?  That only
>> matters if there are gaps in (2), it's hard to know if that will keep
>> coming up as more use-cases emerge
>> - for a Helix committer, is the cost of maintenance too high, particularly
>> to maintain backward compatibility
>> 
>> 4. Backward compatibility
>> - we don't want to leave early adopters out in the cold
>> 
>> Given that framework for evaluation, the helix-api package looks like the
>> way to go.  I'm not quite sure how the old branch would be retired, other
>> than polling users and making a judgment call.  There are more people using
>> Helix than we know about, as I continue to run into people using it that we
>> didn't know about.
>> 
>> The key is to get the APIs right, easier said than done.  That's my $0.02.
>> 
>> Thanks,
>> Bob
>> 
>> 
>> Good suggestion, I think Vinayak had suggestions on similar lines but I
>>> think he suggestion writing high level api's on top of existing api's, as
>>> opposed to changing the  changing the underlying implementation. What that
>>> means is we leave the existing api's as is but add high level api. What
>>> this means is we can keep the core as is but add helix-api module that
>>> provides high level interfaces. We can re-use existing implementation
>>> under
>>> the hood. I like the idea of creating a separate module helix-api that is
>>> decoupled from implementation. This will enforce us to have no dependency
>>> on zookeeper in api and allows one to plug other forms of storage. For
>>> example many usecases dont need the level of consistency we get from
>>> Zookeeper.
>>> 
>>> Looks like we might have to make decisions on a case by case basis.
>>> 
>>> Can we make a decision of having a helix-api package.
>>> 
>>> thanks,
>>> Kishore G
>>> 
>>> 
>>> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <kanak.b@hotmail.com
>>>> wrote:
>>> 
>>> The biggest problem is that we pretty much have to backport bug fixes
>>>> forever if we completely break compatibility. Here's what I think we
>>>> could
>>>> do:
>>>> 
>>>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>>>> stuff)
>>>> 2) Keep the interfaces for the most common currently used APIs
>>>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command
>>>> line,
>>>> REST), but use our new implementations underneath (HelixConnection,
>>>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>>>> This means there will be breakages for people who used the implementation
>>>> classes directly, but the changes would be minor.
>>>> 3) Provide adapters between common new and old APIs
>>>> (HelixConnectionAdapter is an example)
>>>> 4) Maintain the model package as-is
>>>> 
>>>> This way, there's a clear transition path from one branch to another and
>>>> we can eventually end support for the old branch.
>>>> ________________________________
>>>>> Date: Sat, 22 Feb 2014 07:33:10 -0800
>>>>> Subject: Helix V2 0.7.0
>>>>> From: g.kishore@gmail.com
>>>>> To: dev@helix.apache.org; user@helix.apache.org
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> We have made lot of good changes in 0.7.0 and improved the api's.
>>>>> However, I think it is not easy/intuitive for a new user. One of the
>>>>> problems with 0.7.0 code is that we tried to maintain complete backward
>>>>> compatibility with respect to both logical api's and physical layout on
>>>>> zookeeper.
>>>>> 
>>>>> Trying to maintain the logical api backward compatibility has
>>>>> definitely caused some pain and did not allow us to do the right thing
>>>>> in 0.7.0 and it has made our code base huge. The reasoning here is -
>>>>> when we built Helix ((almost 3 years back), we did not anticipate Helix
>>>>> being used in other systems. So our main focus was minimal code and to
>>>>> make sure it works for the use case we had. We did not gather much
>>>>> feedback from users.
>>>>> 
>>>>> However, we are seeing the usage grow and while everyone agrees that
>>>>> the high level concepts are good, it is apparent that api's are making
>>>>> people shy away from Helix. I would even say some of the terminologies
>>>>> are confusing until you spend quite some time with Helix.
>>>>> 
>>>>> I want to see what others think about this.
>>>>> 
>>>>> We have two options going forward
>>>>> 
>>>>> Option1: Continue to maintain backward compatibility and improving the
>>>> api's
>>>>> Option2: Break the api compatibility and call it Helix V2. We redesign
>>>>> our api's and make it more intuitive and easier/flexible to use.
>>>>> 
>>>>> I think the core functionality and design is great and don't see much
>>>>> change needed in the architecture (Do let us know if you think we need
>>>>> any change). What is lacking is documentation and a simple set of api's
>>>>> that are intuitive.
>>>>> 
>>>>> While Option 1 is great for existing users, I prefer Option2. We will
>>>>> redesign the 0.7.0 api's without maintaining backward compatibility.
>>>>> Lot of work has already been done in 0.7.0, so we are not that far.
>>>>> This also gives chance to the community to contribute and provide
>>>>> suggestions/feedback/ideas.
>>>>> 
>>>>> For existing users, we will continue to maintain 0.6.2 and continue to
>>>>> make critical bug fixes. But no new features will be added.
>>>>> 
>>>>> Thoughts ?
>>>>> 
>>>>> thanks,
>>>>> Kishore G
>>>> 
>>>> 
>>> 
>> 
>> 
>> 

Re: Helix V2 0.7.0

[Zhen Zhang <ne...@gmail.com>]

+1 for helix-api package that contains public interface only and decouples
from implementation. The api package should includes Accessor, HelixAdmin,
HelixManager, HelixConnection, MessagingService, etc. We can start by
listing all the necessary interfaces. I am not sure if we have a clear
separation between API and SPI. For example, we are using Accessor and
MessagingService as both API and SPI. Nevertheless, we need to stick to the
interfaces inside Helix also. Current Helix code base has many places where
we use concrete implementations instead of interfaces.

For backward compatibility, I agree with Kishore that as long as we are
compatible with physical model, most of the compatibility issues should be
solved at compile time. We might have to keep the old interfaces for
current uses, but need to clean them up. For example, we can't get rid of
HelixManager totally and enforce people to switch to the new
HelixConnectiont/HelixParticipant, but we should remove/change confusing
api's. Users should be able to easily fix the compatibility issues at
compile time.

Thanks,
Jason




On Sat, Feb 22, 2014 at 11:48 AM, <bo...@schulman.com> wrote:

> Here's how I'd evaluate what to do, consider these 4 sub-problems:
>
> 1. Ease of learning
> - are the APIs accessible for a first-timer?  Like it or not, that's an
> important factor in getting wider adoption.  Necessary but not sufficient.
> - a separate helix-api package is a good way to keep it as simple as
> possible, because then you only have to understand that to build a system
> using Helix
> - also, a new package gives us an opportunity to change terminology if
> that will help new users
>
> 2. Functionality
> - can Helix do what I want?
> - as long as the APIs, say in a helix-api package don't hide things a
> system developer needs, this isn't a tradeoff
> - careful terminology might help here
>
> 3. Maintainability/Extensibility
> - for a Helix developer, is it possible to add functionality?  That only
> matters if there are gaps in (2), it's hard to know if that will keep
> coming up as more use-cases emerge
> - for a Helix committer, is the cost of maintenance too high, particularly
> to maintain backward compatibility
>
> 4. Backward compatibility
> - we don't want to leave early adopters out in the cold
>
> Given that framework for evaluation, the helix-api package looks like the
> way to go.  I'm not quite sure how the old branch would be retired, other
> than polling users and making a judgment call.  There are more people using
> Helix than we know about, as I continue to run into people using it that we
> didn't know about.
>
> The key is to get the APIs right, easier said than done.  That's my $0.02.
>
> Thanks,
> Bob
>
>
>  Good suggestion, I think Vinayak had suggestions on similar lines but I
>> think he suggestion writing high level api's on top of existing api's, as
>> opposed to changing the  changing the underlying implementation. What that
>> means is we leave the existing api's as is but add high level api. What
>> this means is we can keep the core as is but add helix-api module that
>> provides high level interfaces. We can re-use existing implementation
>> under
>> the hood. I like the idea of creating a separate module helix-api that is
>> decoupled from implementation. This will enforce us to have no dependency
>> on zookeeper in api and allows one to plug other forms of storage. For
>> example many usecases dont need the level of consistency we get from
>> Zookeeper.
>>
>> Looks like we might have to make decisions on a case by case basis.
>>
>> Can we make a decision of having a helix-api package.
>>
>> thanks,
>> Kishore G
>>
>>
>> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <kanak.b@hotmail.com
>> >wrote:
>>
>>  The biggest problem is that we pretty much have to backport bug fixes
>>> forever if we completely break compatibility. Here's what I think we
>>> could
>>> do:
>>>
>>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>>> stuff)
>>> 2) Keep the interfaces for the most common currently used APIs
>>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command
>>> line,
>>> REST), but use our new implementations underneath (HelixConnection,
>>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>>> This means there will be breakages for people who used the implementation
>>> classes directly, but the changes would be minor.
>>> 3) Provide adapters between common new and old APIs
>>> (HelixConnectionAdapter is an example)
>>> 4) Maintain the model package as-is
>>>
>>> This way, there's a clear transition path from one branch to another and
>>> we can eventually end support for the old branch.
>>> ________________________________
>>> > Date: Sat, 22 Feb 2014 07:33:10 -0800
>>> > Subject: Helix V2 0.7.0
>>> > From: g.kishore@gmail.com
>>> > To: dev@helix.apache.org; user@helix.apache.org
>>> >
>>> > Hi,
>>> >
>>> > We have made lot of good changes in 0.7.0 and improved the api's.
>>> > However, I think it is not easy/intuitive for a new user. One of the
>>> > problems with 0.7.0 code is that we tried to maintain complete backward
>>> > compatibility with respect to both logical api's and physical layout on
>>> > zookeeper.
>>> >
>>> > Trying to maintain the logical api backward compatibility has
>>> > definitely caused some pain and did not allow us to do the right thing
>>> > in 0.7.0 and it has made our code base huge. The reasoning here is -
>>> > when we built Helix ((almost 3 years back), we did not anticipate Helix
>>> > being used in other systems. So our main focus was minimal code and to
>>> > make sure it works for the use case we had. We did not gather much
>>> > feedback from users.
>>> >
>>> > However, we are seeing the usage grow and while everyone agrees that
>>> > the high level concepts are good, it is apparent that api's are making
>>> > people shy away from Helix. I would even say some of the terminologies
>>> > are confusing until you spend quite some time with Helix.
>>> >
>>> > I want to see what others think about this.
>>> >
>>> > We have two options going forward
>>> >
>>> > Option1: Continue to maintain backward compatibility and improving the
>>> api's
>>> > Option2: Break the api compatibility and call it Helix V2. We redesign
>>> > our api's and make it more intuitive and easier/flexible to use.
>>> >
>>> > I think the core functionality and design is great and don't see much
>>> > change needed in the architecture (Do let us know if you think we need
>>> > any change). What is lacking is documentation and a simple set of api's
>>> > that are intuitive.
>>> >
>>> > While Option 1 is great for existing users, I prefer Option2. We will
>>> > redesign the 0.7.0 api's without maintaining backward compatibility.
>>> > Lot of work has already been done in 0.7.0, so we are not that far.
>>> > This also gives chance to the community to contribute and provide
>>> > suggestions/feedback/ideas.
>>> >
>>> > For existing users, we will continue to maintain 0.6.2 and continue to
>>> > make critical bug fixes. But no new features will be added.
>>> >
>>> > Thoughts ?
>>> >
>>> > thanks,
>>> > Kishore G
>>>
>>>
>>
>
>
>

Re: Helix V2 0.7.0

[Zhen Zhang <ne...@gmail.com>]

+1 for helix-api package that contains public interface only and decouples
from implementation. The api package should includes Accessor, HelixAdmin,
HelixManager, HelixConnection, MessagingService, etc. We can start by
listing all the necessary interfaces. I am not sure if we have a clear
separation between API and SPI. For example, we are using Accessor and
MessagingService as both API and SPI. Nevertheless, we need to stick to the
interfaces inside Helix also. Current Helix code base has many places where
we use concrete implementations instead of interfaces.

For backward compatibility, I agree with Kishore that as long as we are
compatible with physical model, most of the compatibility issues should be
solved at compile time. We might have to keep the old interfaces for
current uses, but need to clean them up. For example, we can't get rid of
HelixManager totally and enforce people to switch to the new
HelixConnectiont/HelixParticipant, but we should remove/change confusing
api's. Users should be able to easily fix the compatibility issues at
compile time.

Thanks,
Jason




On Sat, Feb 22, 2014 at 11:48 AM, <bo...@schulman.com> wrote:

> Here's how I'd evaluate what to do, consider these 4 sub-problems:
>
> 1. Ease of learning
> - are the APIs accessible for a first-timer?  Like it or not, that's an
> important factor in getting wider adoption.  Necessary but not sufficient.
> - a separate helix-api package is a good way to keep it as simple as
> possible, because then you only have to understand that to build a system
> using Helix
> - also, a new package gives us an opportunity to change terminology if
> that will help new users
>
> 2. Functionality
> - can Helix do what I want?
> - as long as the APIs, say in a helix-api package don't hide things a
> system developer needs, this isn't a tradeoff
> - careful terminology might help here
>
> 3. Maintainability/Extensibility
> - for a Helix developer, is it possible to add functionality?  That only
> matters if there are gaps in (2), it's hard to know if that will keep
> coming up as more use-cases emerge
> - for a Helix committer, is the cost of maintenance too high, particularly
> to maintain backward compatibility
>
> 4. Backward compatibility
> - we don't want to leave early adopters out in the cold
>
> Given that framework for evaluation, the helix-api package looks like the
> way to go.  I'm not quite sure how the old branch would be retired, other
> than polling users and making a judgment call.  There are more people using
> Helix than we know about, as I continue to run into people using it that we
> didn't know about.
>
> The key is to get the APIs right, easier said than done.  That's my $0.02.
>
> Thanks,
> Bob
>
>
>  Good suggestion, I think Vinayak had suggestions on similar lines but I
>> think he suggestion writing high level api's on top of existing api's, as
>> opposed to changing the  changing the underlying implementation. What that
>> means is we leave the existing api's as is but add high level api. What
>> this means is we can keep the core as is but add helix-api module that
>> provides high level interfaces. We can re-use existing implementation
>> under
>> the hood. I like the idea of creating a separate module helix-api that is
>> decoupled from implementation. This will enforce us to have no dependency
>> on zookeeper in api and allows one to plug other forms of storage. For
>> example many usecases dont need the level of consistency we get from
>> Zookeeper.
>>
>> Looks like we might have to make decisions on a case by case basis.
>>
>> Can we make a decision of having a helix-api package.
>>
>> thanks,
>> Kishore G
>>
>>
>> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <kanak.b@hotmail.com
>> >wrote:
>>
>>  The biggest problem is that we pretty much have to backport bug fixes
>>> forever if we completely break compatibility. Here's what I think we
>>> could
>>> do:
>>>
>>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>>> stuff)
>>> 2) Keep the interfaces for the most common currently used APIs
>>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command
>>> line,
>>> REST), but use our new implementations underneath (HelixConnection,
>>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>>> This means there will be breakages for people who used the implementation
>>> classes directly, but the changes would be minor.
>>> 3) Provide adapters between common new and old APIs
>>> (HelixConnectionAdapter is an example)
>>> 4) Maintain the model package as-is
>>>
>>> This way, there's a clear transition path from one branch to another and
>>> we can eventually end support for the old branch.
>>> ________________________________
>>> > Date: Sat, 22 Feb 2014 07:33:10 -0800
>>> > Subject: Helix V2 0.7.0
>>> > From: g.kishore@gmail.com
>>> > To: dev@helix.apache.org; user@helix.apache.org
>>> >
>>> > Hi,
>>> >
>>> > We have made lot of good changes in 0.7.0 and improved the api's.
>>> > However, I think it is not easy/intuitive for a new user. One of the
>>> > problems with 0.7.0 code is that we tried to maintain complete backward
>>> > compatibility with respect to both logical api's and physical layout on
>>> > zookeeper.
>>> >
>>> > Trying to maintain the logical api backward compatibility has
>>> > definitely caused some pain and did not allow us to do the right thing
>>> > in 0.7.0 and it has made our code base huge. The reasoning here is -
>>> > when we built Helix ((almost 3 years back), we did not anticipate Helix
>>> > being used in other systems. So our main focus was minimal code and to
>>> > make sure it works for the use case we had. We did not gather much
>>> > feedback from users.
>>> >
>>> > However, we are seeing the usage grow and while everyone agrees that
>>> > the high level concepts are good, it is apparent that api's are making
>>> > people shy away from Helix. I would even say some of the terminologies
>>> > are confusing until you spend quite some time with Helix.
>>> >
>>> > I want to see what others think about this.
>>> >
>>> > We have two options going forward
>>> >
>>> > Option1: Continue to maintain backward compatibility and improving the
>>> api's
>>> > Option2: Break the api compatibility and call it Helix V2. We redesign
>>> > our api's and make it more intuitive and easier/flexible to use.
>>> >
>>> > I think the core functionality and design is great and don't see much
>>> > change needed in the architecture (Do let us know if you think we need
>>> > any change). What is lacking is documentation and a simple set of api's
>>> > that are intuitive.
>>> >
>>> > While Option 1 is great for existing users, I prefer Option2. We will
>>> > redesign the 0.7.0 api's without maintaining backward compatibility.
>>> > Lot of work has already been done in 0.7.0, so we are not that far.
>>> > This also gives chance to the community to contribute and provide
>>> > suggestions/feedback/ideas.
>>> >
>>> > For existing users, we will continue to maintain 0.6.2 and continue to
>>> > make critical bug fixes. But no new features will be added.
>>> >
>>> > Thoughts ?
>>> >
>>> > thanks,
>>> > Kishore G
>>>
>>>
>>
>
>
>

Re: Helix V2 0.7.0

[bo...@schulman.com]

Here's how I'd evaluate what to do, consider these 4 sub-problems:

1. Ease of learning
- are the APIs accessible for a first-timer?  Like it or not, that's  
an important factor in getting wider adoption.  Necessary but not  
sufficient.
- a separate helix-api package is a good way to keep it as simple as  
possible, because then you only have to understand that to build a  
system using Helix
- also, a new package gives us an opportunity to change terminology if  
that will help new users

2. Functionality
- can Helix do what I want?
- as long as the APIs, say in a helix-api package don't hide things a  
system developer needs, this isn't a tradeoff
- careful terminology might help here

3. Maintainability/Extensibility
- for a Helix developer, is it possible to add functionality?  That  
only matters if there are gaps in (2), it's hard to know if that will  
keep coming up as more use-cases emerge
- for a Helix committer, is the cost of maintenance too high,  
particularly to maintain backward compatibility

4. Backward compatibility
- we don't want to leave early adopters out in the cold

Given that framework for evaluation, the helix-api package looks like  
the way to go.  I'm not quite sure how the old branch would be  
retired, other than polling users and making a judgment call.  There  
are more people using Helix than we know about, as I continue to run  
into people using it that we didn't know about.

The key is to get the APIs right, easier said than done.  That's my $0.02.

Thanks,
Bob

> Good suggestion, I think Vinayak had suggestions on similar lines but I
> think he suggestion writing high level api's on top of existing api's, as
> opposed to changing the  changing the underlying implementation. What that
> means is we leave the existing api's as is but add high level api. What
> this means is we can keep the core as is but add helix-api module that
> provides high level interfaces. We can re-use existing implementation under
> the hood. I like the idea of creating a separate module helix-api that is
> decoupled from implementation. This will enforce us to have no dependency
> on zookeeper in api and allows one to plug other forms of storage. For
> example many usecases dont need the level of consistency we get from
> Zookeeper.
>
> Looks like we might have to make decisions on a case by case basis.
>
> Can we make a decision of having a helix-api package.
>
> thanks,
> Kishore G
>
>
> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala  
> <ka...@hotmail.com>wrote:
>
>> The biggest problem is that we pretty much have to backport bug fixes
>> forever if we completely break compatibility. Here's what I think we could
>> do:
>>
>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>> stuff)
>> 2) Keep the interfaces for the most common currently used APIs
>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
>> REST), but use our new implementations underneath (HelixConnection,
>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>> This means there will be breakages for people who used the implementation
>> classes directly, but the changes would be minor.
>> 3) Provide adapters between common new and old APIs
>> (HelixConnectionAdapter is an example)
>> 4) Maintain the model package as-is
>>
>> This way, there's a clear transition path from one branch to another and
>> we can eventually end support for the old branch.
>> ________________________________
>> > Date: Sat, 22 Feb 2014 07:33:10 -0800
>> > Subject: Helix V2 0.7.0
>> > From: g.kishore@gmail.com
>> > To: dev@helix.apache.org; user@helix.apache.org
>> >
>> > Hi,
>> >
>> > We have made lot of good changes in 0.7.0 and improved the api's.
>> > However, I think it is not easy/intuitive for a new user. One of the
>> > problems with 0.7.0 code is that we tried to maintain complete backward
>> > compatibility with respect to both logical api's and physical layout on
>> > zookeeper.
>> >
>> > Trying to maintain the logical api backward compatibility has
>> > definitely caused some pain and did not allow us to do the right thing
>> > in 0.7.0 and it has made our code base huge. The reasoning here is -
>> > when we built Helix ((almost 3 years back), we did not anticipate Helix
>> > being used in other systems. So our main focus was minimal code and to
>> > make sure it works for the use case we had. We did not gather much
>> > feedback from users.
>> >
>> > However, we are seeing the usage grow and while everyone agrees that
>> > the high level concepts are good, it is apparent that api's are making
>> > people shy away from Helix. I would even say some of the terminologies
>> > are confusing until you spend quite some time with Helix.
>> >
>> > I want to see what others think about this.
>> >
>> > We have two options going forward
>> >
>> > Option1: Continue to maintain backward compatibility and improving the
>> api's
>> > Option2: Break the api compatibility and call it Helix V2. We redesign
>> > our api's and make it more intuitive and easier/flexible to use.
>> >
>> > I think the core functionality and design is great and don't see much
>> > change needed in the architecture (Do let us know if you think we need
>> > any change). What is lacking is documentation and a simple set of api's
>> > that are intuitive.
>> >
>> > While Option 1 is great for existing users, I prefer Option2. We will
>> > redesign the 0.7.0 api's without maintaining backward compatibility.
>> > Lot of work has already been done in 0.7.0, so we are not that far.
>> > This also gives chance to the community to contribute and provide
>> > suggestions/feedback/ideas.
>> >
>> > For existing users, we will continue to maintain 0.6.2 and continue to
>> > make critical bug fixes. But no new features will be added.
>> >
>> > Thoughts ?
>> >
>> > thanks,
>> > Kishore G
>>
>

Re: Helix V2 0.7.0

[bo...@schulman.com]

Here's how I'd evaluate what to do, consider these 4 sub-problems:

1. Ease of learning
- are the APIs accessible for a first-timer?  Like it or not, that's  
an important factor in getting wider adoption.  Necessary but not  
sufficient.
- a separate helix-api package is a good way to keep it as simple as  
possible, because then you only have to understand that to build a  
system using Helix
- also, a new package gives us an opportunity to change terminology if  
that will help new users

2. Functionality
- can Helix do what I want?
- as long as the APIs, say in a helix-api package don't hide things a  
system developer needs, this isn't a tradeoff
- careful terminology might help here

3. Maintainability/Extensibility
- for a Helix developer, is it possible to add functionality?  That  
only matters if there are gaps in (2), it's hard to know if that will  
keep coming up as more use-cases emerge
- for a Helix committer, is the cost of maintenance too high,  
particularly to maintain backward compatibility

4. Backward compatibility
- we don't want to leave early adopters out in the cold

Given that framework for evaluation, the helix-api package looks like  
the way to go.  I'm not quite sure how the old branch would be  
retired, other than polling users and making a judgment call.  There  
are more people using Helix than we know about, as I continue to run  
into people using it that we didn't know about.

The key is to get the APIs right, easier said than done.  That's my $0.02.

Thanks,
Bob

> Good suggestion, I think Vinayak had suggestions on similar lines but I
> think he suggestion writing high level api's on top of existing api's, as
> opposed to changing the  changing the underlying implementation. What that
> means is we leave the existing api's as is but add high level api. What
> this means is we can keep the core as is but add helix-api module that
> provides high level interfaces. We can re-use existing implementation under
> the hood. I like the idea of creating a separate module helix-api that is
> decoupled from implementation. This will enforce us to have no dependency
> on zookeeper in api and allows one to plug other forms of storage. For
> example many usecases dont need the level of consistency we get from
> Zookeeper.
>
> Looks like we might have to make decisions on a case by case basis.
>
> Can we make a decision of having a helix-api package.
>
> thanks,
> Kishore G
>
>
> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala  
> <ka...@hotmail.com>wrote:
>
>> The biggest problem is that we pretty much have to backport bug fixes
>> forever if we completely break compatibility. Here's what I think we could
>> do:
>>
>> 1) Remove APIs that we're confident no one uses or can use (the alerting
>> stuff)
>> 2) Keep the interfaces for the most common currently used APIs
>> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
>> REST), but use our new implementations underneath (HelixConnection,
>> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
>> This means there will be breakages for people who used the implementation
>> classes directly, but the changes would be minor.
>> 3) Provide adapters between common new and old APIs
>> (HelixConnectionAdapter is an example)
>> 4) Maintain the model package as-is
>>
>> This way, there's a clear transition path from one branch to another and
>> we can eventually end support for the old branch.
>> ________________________________
>> > Date: Sat, 22 Feb 2014 07:33:10 -0800
>> > Subject: Helix V2 0.7.0
>> > From: g.kishore@gmail.com
>> > To: dev@helix.apache.org; user@helix.apache.org
>> >
>> > Hi,
>> >
>> > We have made lot of good changes in 0.7.0 and improved the api's.
>> > However, I think it is not easy/intuitive for a new user. One of the
>> > problems with 0.7.0 code is that we tried to maintain complete backward
>> > compatibility with respect to both logical api's and physical layout on
>> > zookeeper.
>> >
>> > Trying to maintain the logical api backward compatibility has
>> > definitely caused some pain and did not allow us to do the right thing
>> > in 0.7.0 and it has made our code base huge. The reasoning here is -
>> > when we built Helix ((almost 3 years back), we did not anticipate Helix
>> > being used in other systems. So our main focus was minimal code and to
>> > make sure it works for the use case we had. We did not gather much
>> > feedback from users.
>> >
>> > However, we are seeing the usage grow and while everyone agrees that
>> > the high level concepts are good, it is apparent that api's are making
>> > people shy away from Helix. I would even say some of the terminologies
>> > are confusing until you spend quite some time with Helix.
>> >
>> > I want to see what others think about this.
>> >
>> > We have two options going forward
>> >
>> > Option1: Continue to maintain backward compatibility and improving the
>> api's
>> > Option2: Break the api compatibility and call it Helix V2. We redesign
>> > our api's and make it more intuitive and easier/flexible to use.
>> >
>> > I think the core functionality and design is great and don't see much
>> > change needed in the architecture (Do let us know if you think we need
>> > any change). What is lacking is documentation and a simple set of api's
>> > that are intuitive.
>> >
>> > While Option 1 is great for existing users, I prefer Option2. We will
>> > redesign the 0.7.0 api's without maintaining backward compatibility.
>> > Lot of work has already been done in 0.7.0, so we are not that far.
>> > This also gives chance to the community to contribute and provide
>> > suggestions/feedback/ideas.
>> >
>> > For existing users, we will continue to maintain 0.6.2 and continue to
>> > make critical bug fixes. But no new features will be added.
>> >
>> > Thoughts ?
>> >
>> > thanks,
>> > Kishore G
>>
>

Re: Helix V2 0.7.0

[kishore g <g....@gmail.com>]

Good suggestion, I think Vinayak had suggestions on similar lines but I
think he suggestion writing high level api's on top of existing api's, as
opposed to changing the  changing the underlying implementation. What that
means is we leave the existing api's as is but add high level api. What
this means is we can keep the core as is but add helix-api module that
provides high level interfaces. We can re-use existing implementation under
the hood. I like the idea of creating a separate module helix-api that is
decoupled from implementation. This will enforce us to have no dependency
on zookeeper in api and allows one to plug other forms of storage. For
example many usecases dont need the level of consistency we get from
Zookeeper.

Looks like we might have to make decisions on a case by case basis.

Can we make a decision of having a helix-api package.

thanks,
Kishore G


On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:

> The biggest problem is that we pretty much have to backport bug fixes
> forever if we completely break compatibility. Here's what I think we could
> do:
>
> 1) Remove APIs that we're confident no one uses or can use (the alerting
> stuff)
> 2) Keep the interfaces for the most common currently used APIs
> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
> REST), but use our new implementations underneath (HelixConnection,
> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
> This means there will be breakages for people who used the implementation
> classes directly, but the changes would be minor.
> 3) Provide adapters between common new and old APIs
> (HelixConnectionAdapter is an example)
> 4) Maintain the model package as-is
>
> This way, there's a clear transition path from one branch to another and
> we can eventually end support for the old branch.
> ________________________________
> > Date: Sat, 22 Feb 2014 07:33:10 -0800
> > Subject: Helix V2 0.7.0
> > From: g.kishore@gmail.com
> > To: dev@helix.apache.org; user@helix.apache.org
> >
> > Hi,
> >
> > We have made lot of good changes in 0.7.0 and improved the api's.
> > However, I think it is not easy/intuitive for a new user. One of the
> > problems with 0.7.0 code is that we tried to maintain complete backward
> > compatibility with respect to both logical api's and physical layout on
> > zookeeper.
> >
> > Trying to maintain the logical api backward compatibility has
> > definitely caused some pain and did not allow us to do the right thing
> > in 0.7.0 and it has made our code base huge. The reasoning here is -
> > when we built Helix ((almost 3 years back), we did not anticipate Helix
> > being used in other systems. So our main focus was minimal code and to
> > make sure it works for the use case we had. We did not gather much
> > feedback from users.
> >
> > However, we are seeing the usage grow and while everyone agrees that
> > the high level concepts are good, it is apparent that api's are making
> > people shy away from Helix. I would even say some of the terminologies
> > are confusing until you spend quite some time with Helix.
> >
> > I want to see what others think about this.
> >
> > We have two options going forward
> >
> > Option1: Continue to maintain backward compatibility and improving the
> api's
> > Option2: Break the api compatibility and call it Helix V2. We redesign
> > our api's and make it more intuitive and easier/flexible to use.
> >
> > I think the core functionality and design is great and don't see much
> > change needed in the architecture (Do let us know if you think we need
> > any change). What is lacking is documentation and a simple set of api's
> > that are intuitive.
> >
> > While Option 1 is great for existing users, I prefer Option2. We will
> > redesign the 0.7.0 api's without maintaining backward compatibility.
> > Lot of work has already been done in 0.7.0, so we are not that far.
> > This also gives chance to the community to contribute and provide
> > suggestions/feedback/ideas.
> >
> > For existing users, we will continue to maintain 0.6.2 and continue to
> > make critical bug fixes. But no new features will be added.
> >
> > Thoughts ?
> >
> > thanks,
> > Kishore G
>

Re: Helix V2 0.7.0

[kishore g <g....@gmail.com>]

Good suggestion, I think Vinayak had suggestions on similar lines but I
think he suggestion writing high level api's on top of existing api's, as
opposed to changing the  changing the underlying implementation. What that
means is we leave the existing api's as is but add high level api. What
this means is we can keep the core as is but add helix-api module that
provides high level interfaces. We can re-use existing implementation under
the hood. I like the idea of creating a separate module helix-api that is
decoupled from implementation. This will enforce us to have no dependency
on zookeeper in api and allows one to plug other forms of storage. For
example many usecases dont need the level of consistency we get from
Zookeeper.

Looks like we might have to make decisions on a case by case basis.

Can we make a decision of having a helix-api package.

thanks,
Kishore G


On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:

> The biggest problem is that we pretty much have to backport bug fixes
> forever if we completely break compatibility. Here's what I think we could
> do:
>
> 1) Remove APIs that we're confident no one uses or can use (the alerting
> stuff)
> 2) Keep the interfaces for the most common currently used APIs
> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
> REST), but use our new implementations underneath (HelixConnection,
> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
> This means there will be breakages for people who used the implementation
> classes directly, but the changes would be minor.
> 3) Provide adapters between common new and old APIs
> (HelixConnectionAdapter is an example)
> 4) Maintain the model package as-is
>
> This way, there's a clear transition path from one branch to another and
> we can eventually end support for the old branch.
> ________________________________
> > Date: Sat, 22 Feb 2014 07:33:10 -0800
> > Subject: Helix V2 0.7.0
> > From: g.kishore@gmail.com
> > To: dev@helix.apache.org; user@helix.apache.org
> >
> > Hi,
> >
> > We have made lot of good changes in 0.7.0 and improved the api's.
> > However, I think it is not easy/intuitive for a new user. One of the
> > problems with 0.7.0 code is that we tried to maintain complete backward
> > compatibility with respect to both logical api's and physical layout on
> > zookeeper.
> >
> > Trying to maintain the logical api backward compatibility has
> > definitely caused some pain and did not allow us to do the right thing
> > in 0.7.0 and it has made our code base huge. The reasoning here is -
> > when we built Helix ((almost 3 years back), we did not anticipate Helix
> > being used in other systems. So our main focus was minimal code and to
> > make sure it works for the use case we had. We did not gather much
> > feedback from users.
> >
> > However, we are seeing the usage grow and while everyone agrees that
> > the high level concepts are good, it is apparent that api's are making
> > people shy away from Helix. I would even say some of the terminologies
> > are confusing until you spend quite some time with Helix.
> >
> > I want to see what others think about this.
> >
> > We have two options going forward
> >
> > Option1: Continue to maintain backward compatibility and improving the
> api's
> > Option2: Break the api compatibility and call it Helix V2. We redesign
> > our api's and make it more intuitive and easier/flexible to use.
> >
> > I think the core functionality and design is great and don't see much
> > change needed in the architecture (Do let us know if you think we need
> > any change). What is lacking is documentation and a simple set of api's
> > that are intuitive.
> >
> > While Option 1 is great for existing users, I prefer Option2. We will
> > redesign the 0.7.0 api's without maintaining backward compatibility.
> > Lot of work has already been done in 0.7.0, so we are not that far.
> > This also gives chance to the community to contribute and provide
> > suggestions/feedback/ideas.
> >
> > For existing users, we will continue to maintain 0.6.2 and continue to
> > make critical bug fixes. But no new features will be added.
> >
> > Thoughts ?
> >
> > thanks,
> > Kishore G
>

Re: Helix V2 0.7.0

[kishore g <g....@gmail.com>]

Very nicely put, I like the separation between API and SPI. Lets start
first by laying out all the entities in Helix.

I see pros and cons in maintaining compatibility. But if we know that
current api's are broken I prefer getting rid of them instead of carrying
it over and always having to maintain them.

Also as long we are compatible with physical layout on zookeeper, most of
the compatibility issues should be resolved at compile time.

Will start another thread to discuss the data model and high level api's.

thanks,
kishore G


On Sat, Feb 22, 2014 at 10:58 AM, Sandeep Nayak <os...@gmail.com> wrote:

> Hey guys,
>
> I prefer option 2, its clear and does not have package specific rules
> i.e API breakage is acceptable in certain areas whereas others we need
> to keep it as is. Its going to be too difficult for new contributors
> to be on-boarded if this ruleset needs to be followed.
>
> My suggestion is to consider two sets
>
> (a) API: Which clients/users of Helix will stick to and use. This API
> needs to be clear i.e. users should know what is API boundary and what
> is leaking into implementation. If Helix contract to its users is
> clear, users bypass it (ideally we don't allow them to) at their own
> risk.
>
> (b) SPI: This API is what Helix contributors will adhere to for
> subsystems in Helix. SPI will be the contract for someone who is
> changing a core subsystem like for example Helix uses zookeeper, lets
> say tomorrow it needs to get replaced then the contract from Helix to
> the zookeeper layer if abstracted through an SPI would allow other
> adapters to be added to replace zookeeper and Helix code does not need
> to change when it talks to the new adapters.
>
> As regards the steps listed, here are my thoughts
>
> 1) Remove APIs that we're confident no one uses or can use (the alerting
> stuff)
> [Sandeep] What about the APIs which we are not confident about? Why
> keep carrying forward APIs which possibly or in some cases definitely
> need to change because they were not clear/clean but cannot be changed
> with the fear of breaking backward compatibility? Besides controlling
> and tracking such APIs will be a management overhead.
>
> 2) Keep the interfaces for the most common currently used APIs
> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command
> line, REST), but use our new implementations underneath
> (HelixConnection, HelixController, HelixParticipant, HelixSpectator,
> HelixAdministrator). This means there will be breakages for people who
> used the implementation classes directly, but the changes would be
> minor.
> [Sandeep] The problem I see here is that some of the existing APIs
> need to be changed, point in case HelixAdmin needs to throw correct
> exceptions rather than HelixException which is a runtime exception. If
> we want to fix it we will need to break API compatibility.
>
> 3) Provide adapters between common new and old APIs
> (HelixConnectionAdapter is an example)
> [Sandeep] Regardless of what approach is taken, this is a good idea
> wherever possible.
>
> 4) Maintain the model package as-is
>
> Sandeep
>
> On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <ka...@hotmail.com>
> wrote:
> > The biggest problem is that we pretty much have to backport bug fixes
> forever if we completely break compatibility. Here's what I think we could
> do:
> >
> > 1) Remove APIs that we're confident no one uses or can use (the alerting
> stuff)
> > 2) Keep the interfaces for the most common currently used APIs
> (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line,
> REST), but use our new implementations underneath (HelixConnection,
> HelixController, HelixParticipant, HelixSpectator, HelixAdministrator).
> This means there will be breakages for people who used the implementation
> classes directly, but the changes would be minor.
> > 3) Provide adapters between common new and old APIs
> (HelixConnectionAdapter is an example)
> > 4) Maintain the model package as-is
> >
> > This way, there's a clear transition path from one branch to another and
> we can eventually end support for the old branch.
> > ________________________________
> >> Date: Sat, 22 Feb 2014 07:33:10 -0800
> >> Subject: Helix V2 0.7.0
> >> From: g.kishore@gmail.com
> >> To: dev@helix.apache.org; user@helix.apache.org
> >>
> >> Hi,
> >>
> >> We have made lot of good changes in 0.7.0 and improved the api's.
> >> However, I think it is not easy/intuitive for a new user. One of the
> >> problems with 0.7.0 code is that we tried to maintain complete backward
> >> compatibility with respect to both logical api's and physical layout on
> >> zookeeper.
> >>
> >> Trying to maintain the logical api backward compatibility has
> >> definitely caused some pain and did not allow us to do the right thing
> >> in 0.7.0 and it has made our code base huge. The reasoning here is -
> >> when we built Helix ((almost 3 years back), we did not anticipate Helix
> >> being used in other systems. So our main focus was minimal code and to
> >> make sure it works for the use case we had. We did not gather much
> >> feedback from users.
> >>
> >> However, we are seeing the usage grow and while everyone agrees that
> >> the high level concepts are good, it is apparent that api's are making
> >> people shy away from Helix. I would even say some of the terminologies
> >> are confusing until you spend quite some time with Helix.
> >>
> >> I want to see what others think about this.
> >>
> >> We have two options going forward
> >>
> >> Option1: Continue to maintain backward compatibility and improving the
> api's
> >> Option2: Break the api compatibility and call it Helix V2. We redesign
> >> our api's and make it more intuitive and easier/flexible to use.
> >>
> >> I think the core functionality and design is great and don't see much
> >> change needed in the architecture (Do let us know if you think we need
> >> any change). What is lacking is documentation and a simple set of api's
> >> that are intuitive.
> >>
> >> While Option 1 is great for existing users, I prefer Option2. We will
> >> redesign the 0.7.0 api's without maintaining backward compatibility.
> >> Lot of work has already been done in 0.7.0, so we are not that far.
> >> This also gives chance to the community to contribute and provide
> >> suggestions/feedback/ideas.
> >>
> >> For existing users, we will continue to maintain 0.6.2 and continue to
> >> make critical bug fixes. But no new features will be added.
> >>
> >> Thoughts ?
> >>
> >> thanks,
> >> Kishore G
>

Re: Helix V2 0.7.0

[Sandeep Nayak <os...@gmail.com>]

Hey guys,

I prefer option 2, its clear and does not have package specific rules
i.e API breakage is acceptable in certain areas whereas others we need
to keep it as is. Its going to be too difficult for new contributors
to be on-boarded if this ruleset needs to be followed.

My suggestion is to consider two sets

(a) API: Which clients/users of Helix will stick to and use. This API
needs to be clear i.e. users should know what is API boundary and what
is leaking into implementation. If Helix contract to its users is
clear, users bypass it (ideally we don't allow them to) at their own
risk.

(b) SPI: This API is what Helix contributors will adhere to for
subsystems in Helix. SPI will be the contract for someone who is
changing a core subsystem like for example Helix uses zookeeper, lets
say tomorrow it needs to get replaced then the contract from Helix to
the zookeeper layer if abstracted through an SPI would allow other
adapters to be added to replace zookeeper and Helix code does not need
to change when it talks to the new adapters.

As regards the steps listed, here are my thoughts

1) Remove APIs that we're confident no one uses or can use (the alerting stuff)
[Sandeep] What about the APIs which we are not confident about? Why
keep carrying forward APIs which possibly or in some cases definitely
need to change because they were not clear/clean but cannot be changed
with the fear of breaking backward compatibility? Besides controlling
and tracking such APIs will be a management overhead.

2) Keep the interfaces for the most common currently used APIs
(HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command
line, REST), but use our new implementations underneath
(HelixConnection, HelixController, HelixParticipant, HelixSpectator,
HelixAdministrator). This means there will be breakages for people who
used the implementation classes directly, but the changes would be
minor.
[Sandeep] The problem I see here is that some of the existing APIs
need to be changed, point in case HelixAdmin needs to throw correct
exceptions rather than HelixException which is a runtime exception. If
we want to fix it we will need to break API compatibility.

3) Provide adapters between common new and old APIs
(HelixConnectionAdapter is an example)
[Sandeep] Regardless of what approach is taken, this is a good idea
wherever possible.

4) Maintain the model package as-is

Sandeep

On Sat, Feb 22, 2014 at 10:09 AM, Kanak Biscuitwala <ka...@hotmail.com> wrote:
> The biggest problem is that we pretty much have to backport bug fixes forever if we completely break compatibility. Here's what I think we could do:
>
> 1) Remove APIs that we're confident no one uses or can use (the alerting stuff)
> 2) Keep the interfaces for the most common currently used APIs (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line, REST), but use our new implementations underneath (HelixConnection, HelixController, HelixParticipant, HelixSpectator, HelixAdministrator). This means there will be breakages for people who used the implementation classes directly, but the changes would be minor.
> 3) Provide adapters between common new and old APIs (HelixConnectionAdapter is an example)
> 4) Maintain the model package as-is
>
> This way, there's a clear transition path from one branch to another and we can eventually end support for the old branch.
> ________________________________
>> Date: Sat, 22 Feb 2014 07:33:10 -0800
>> Subject: Helix V2 0.7.0
>> From: g.kishore@gmail.com
>> To: dev@helix.apache.org; user@helix.apache.org
>>
>> Hi,
>>
>> We have made lot of good changes in 0.7.0 and improved the api's.
>> However, I think it is not easy/intuitive for a new user. One of the
>> problems with 0.7.0 code is that we tried to maintain complete backward
>> compatibility with respect to both logical api's and physical layout on
>> zookeeper.
>>
>> Trying to maintain the logical api backward compatibility has
>> definitely caused some pain and did not allow us to do the right thing
>> in 0.7.0 and it has made our code base huge. The reasoning here is -
>> when we built Helix ((almost 3 years back), we did not anticipate Helix
>> being used in other systems. So our main focus was minimal code and to
>> make sure it works for the use case we had. We did not gather much
>> feedback from users.
>>
>> However, we are seeing the usage grow and while everyone agrees that
>> the high level concepts are good, it is apparent that api's are making
>> people shy away from Helix. I would even say some of the terminologies
>> are confusing until you spend quite some time with Helix.
>>
>> I want to see what others think about this.
>>
>> We have two options going forward
>>
>> Option1: Continue to maintain backward compatibility and improving the api's
>> Option2: Break the api compatibility and call it Helix V2. We redesign
>> our api's and make it more intuitive and easier/flexible to use.
>>
>> I think the core functionality and design is great and don't see much
>> change needed in the architecture (Do let us know if you think we need
>> any change). What is lacking is documentation and a simple set of api's
>> that are intuitive.
>>
>> While Option 1 is great for existing users, I prefer Option2. We will
>> redesign the 0.7.0 api's without maintaining backward compatibility.
>> Lot of work has already been done in 0.7.0, so we are not that far.
>> This also gives chance to the community to contribute and provide
>> suggestions/feedback/ideas.
>>
>> For existing users, we will continue to maintain 0.6.2 and continue to
>> make critical bug fixes. But no new features will be added.
>>
>> Thoughts ?
>>
>> thanks,
>> Kishore G

RE: Helix V2 0.7.0

[Kanak Biscuitwala <ka...@hotmail.com>]

The biggest problem is that we pretty much have to backport bug fixes forever if we completely break compatibility. Here's what I think we could do:

1) Remove APIs that we're confident no one uses or can use (the alerting stuff)
2) Keep the interfaces for the most common currently used APIs (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line, REST), but use our new implementations underneath (HelixConnection, HelixController, HelixParticipant, HelixSpectator, HelixAdministrator). This means there will be breakages for people who used the implementation classes directly, but the changes would be minor.
3) Provide adapters between common new and old APIs (HelixConnectionAdapter is an example)
4) Maintain the model package as-is

This way, there's a clear transition path from one branch to another and we can eventually end support for the old branch.
________________________________
> Date: Sat, 22 Feb 2014 07:33:10 -0800 
> Subject: Helix V2 0.7.0 
> From: g.kishore@gmail.com 
> To: dev@helix.apache.org; user@helix.apache.org 
> 
> Hi, 
> 
> We have made lot of good changes in 0.7.0 and improved the api's. 
> However, I think it is not easy/intuitive for a new user. One of the 
> problems with 0.7.0 code is that we tried to maintain complete backward 
> compatibility with respect to both logical api's and physical layout on 
> zookeeper. 
> 
> Trying to maintain the logical api backward compatibility has 
> definitely caused some pain and did not allow us to do the right thing 
> in 0.7.0 and it has made our code base huge. The reasoning here is - 
> when we built Helix ((almost 3 years back), we did not anticipate Helix 
> being used in other systems. So our main focus was minimal code and to 
> make sure it works for the use case we had. We did not gather much 
> feedback from users. 
> 
> However, we are seeing the usage grow and while everyone agrees that 
> the high level concepts are good, it is apparent that api's are making 
> people shy away from Helix. I would even say some of the terminologies 
> are confusing until you spend quite some time with Helix. 
> 
> I want to see what others think about this. 
> 
> We have two options going forward 
> 
> Option1: Continue to maintain backward compatibility and improving the api's 
> Option2: Break the api compatibility and call it Helix V2. We redesign 
> our api's and make it more intuitive and easier/flexible to use. 
> 
> I think the core functionality and design is great and don't see much 
> change needed in the architecture (Do let us know if you think we need 
> any change). What is lacking is documentation and a simple set of api's 
> that are intuitive. 
> 
> While Option 1 is great for existing users, I prefer Option2. We will 
> redesign the 0.7.0 api's without maintaining backward compatibility. 
> Lot of work has already been done in 0.7.0, so we are not that far. 
> This also gives chance to the community to contribute and provide 
> suggestions/feedback/ideas. 
> 
> For existing users, we will continue to maintain 0.6.2 and continue to 
> make critical bug fixes. But no new features will be added. 
> 
> Thoughts ? 
> 
> thanks, 
> Kishore G 		 	   		  

RE: Helix V2 0.7.0

[Kanak Biscuitwala <ka...@hotmail.com>]

The biggest problem is that we pretty much have to backport bug fixes forever if we completely break compatibility. Here's what I think we could do:

1) Remove APIs that we're confident no one uses or can use (the alerting stuff)
2) Keep the interfaces for the most common currently used APIs (HelixManagerFactory, HelixManager, HelixAdmin, ClusterSetup, command line, REST), but use our new implementations underneath (HelixConnection, HelixController, HelixParticipant, HelixSpectator, HelixAdministrator). This means there will be breakages for people who used the implementation classes directly, but the changes would be minor.
3) Provide adapters between common new and old APIs (HelixConnectionAdapter is an example)
4) Maintain the model package as-is

This way, there's a clear transition path from one branch to another and we can eventually end support for the old branch.
________________________________
> Date: Sat, 22 Feb 2014 07:33:10 -0800 
> Subject: Helix V2 0.7.0 
> From: g.kishore@gmail.com 
> To: dev@helix.apache.org; user@helix.apache.org 
> 
> Hi, 
> 
> We have made lot of good changes in 0.7.0 and improved the api's. 
> However, I think it is not easy/intuitive for a new user. One of the 
> problems with 0.7.0 code is that we tried to maintain complete backward 
> compatibility with respect to both logical api's and physical layout on 
> zookeeper. 
> 
> Trying to maintain the logical api backward compatibility has 
> definitely caused some pain and did not allow us to do the right thing 
> in 0.7.0 and it has made our code base huge. The reasoning here is - 
> when we built Helix ((almost 3 years back), we did not anticipate Helix 
> being used in other systems. So our main focus was minimal code and to 
> make sure it works for the use case we had. We did not gather much 
> feedback from users. 
> 
> However, we are seeing the usage grow and while everyone agrees that 
> the high level concepts are good, it is apparent that api's are making 
> people shy away from Helix. I would even say some of the terminologies 
> are confusing until you spend quite some time with Helix. 
> 
> I want to see what others think about this. 
> 
> We have two options going forward 
> 
> Option1: Continue to maintain backward compatibility and improving the api's 
> Option2: Break the api compatibility and call it Helix V2. We redesign 
> our api's and make it more intuitive and easier/flexible to use. 
> 
> I think the core functionality and design is great and don't see much 
> change needed in the architecture (Do let us know if you think we need 
> any change). What is lacking is documentation and a simple set of api's 
> that are intuitive. 
> 
> While Option 1 is great for existing users, I prefer Option2. We will 
> redesign the 0.7.0 api's without maintaining backward compatibility. 
> Lot of work has already been done in 0.7.0, so we are not that far. 
> This also gives chance to the community to contribute and provide 
> suggestions/feedback/ideas. 
> 
> For existing users, we will continue to maintain 0.6.2 and continue to 
> make critical bug fixes. But no new features will be added. 
> 
> Thoughts ? 
> 
> thanks, 
> Kishore G