You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@helix.apache.org by kishore g <g....@gmail.com> on 2014/01/14 20:27:22 UTC
helix provisioning
Hi,
Here is a prototype of supporting the concept of provisioning in Helix.
This was the work done by one of the interns at LinkedIn. You can find more
info here
http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
The implementation done by intern was outside of Helix. Kanak and I took a
stab at it to treat provisioning as a first class citizen of Helix.
The work is in helix-provisioning branch
https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
Most of the code is here,
https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
Here is a YARN specific implementation
https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
Its still a WIP and main goal was to see if we can abstract Resource
Managers like YARN, Mesos, EC2 from the application.
I am working on providing the steps to try out launching a Helix cluster on
YARN.
Feedback/Thoughts? Any interest in contributing to make this production
ready ?
thanks,
Kishore G
RE: helix provisioning
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Looks good to me. It might also be nice to have support for key-value service-level configs. Also, is it easy to externally change the upper and lower bounds for min/max containers?
Date: Sun, 26 Jan 2014 14:17:51 -0800
Subject: Re: helix provisioning
From: g.kishore@gmail.com
To: user@helix.apache.org
CC: dev@helix.apache.org
Here is a sample app spec. does it make sense ?
appName: test
configs:
k1: v1
services:
- name: myservice
participantClass: org.apache.helix.myApp.SimpleWebserver
minContainers:3
maxContainers:3
configs:
- participantId: myservice_0
port: 9500
- participantId: myservice_1
port: 9501
- participantId: myservice_2
port: 9502
resources:
- name: distributedLock
numPartitions: 6
numReplicas: 2
rebalanceMode: FULL_AUTO
stateModel: OnlineOffline
tag: myservice
configs:
k1: v1
On Sat, Jan 18, 2014 at 11:22 AM, kishore g <g....@gmail.com> wrote:
Thats a good suggestion.
We also need to think how the controller pipeline works.
That means the targetprovider will now be across multiple services. Currently we have ability to plugin rebalancer and provisioners per resource. But sometimes this is too fine and might need logic that spans across resources and services.
I started another thread on composite rebalancers, I think we might need a similar concept in targetprovider, provisioners.
On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <ka...@hotmail.com> wrote:
Maybe it should be at both scopes? Sort of like putting bounds cross-app and cross-service within an app. This allows full control over an organization's physical resources.
> Date: Sat, 18 Jan 2014 09:28:32 -0800
> Subject: Re: helix provisioning
> From: g.kishore@gmail.com
> To: dev@helix.apache.org; user@helix.apache.org
>
> Thanks Swaroop for adding interfaces to specify the application spec.
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>
> From what I understand
> AppName maps to cluster name
> Service maps to instances, there can be multiple service components in an
> application. For example, in distributed data system, there may be a
> storage layer and router layer.
>
> The min/max containers seems to be at application level but it probably
> should be at a Service scope ?
>
> what do you think ?
>
>
>
>
>
>
>
>
>
>
> On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com> wrote:
>
> > Hi,
> >
> > Here is a prototype of supporting the concept of provisioning in Helix.
> > This was the work done by one of the interns at LinkedIn. You can find more
> > info here
> >
> > http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
> >
> > The implementation done by intern was outside of Helix. Kanak and I took a
> > stab at it to treat provisioning as a first class citizen of Helix.
> >
> > The work is in helix-provisioning branch
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
> >
> > Most of the code is here,
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
> >
> > Here is a YARN specific implementation
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
> >
> > Its still a WIP and main goal was to see if we can abstract Resource
> > Managers like YARN, Mesos, EC2 from the application.
> >
> > I am working on providing the steps to try out launching a Helix cluster
> > on YARN.
> >
> > Feedback/Thoughts? Any interest in contributing to make this production
> > ready ?
> >
> > thanks,
> > Kishore G
> >
> >
> >
> >
> >
> >
> >
> >
RE: helix provisioning
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Looks good to me. It might also be nice to have support for key-value service-level configs. Also, is it easy to externally change the upper and lower bounds for min/max containers?
Date: Sun, 26 Jan 2014 14:17:51 -0800
Subject: Re: helix provisioning
From: g.kishore@gmail.com
To: user@helix.apache.org
CC: dev@helix.apache.org
Here is a sample app spec. does it make sense ?
appName: test
configs:
k1: v1
services:
- name: myservice
participantClass: org.apache.helix.myApp.SimpleWebserver
minContainers:3
maxContainers:3
configs:
- participantId: myservice_0
port: 9500
- participantId: myservice_1
port: 9501
- participantId: myservice_2
port: 9502
resources:
- name: distributedLock
numPartitions: 6
numReplicas: 2
rebalanceMode: FULL_AUTO
stateModel: OnlineOffline
tag: myservice
configs:
k1: v1
On Sat, Jan 18, 2014 at 11:22 AM, kishore g <g....@gmail.com> wrote:
Thats a good suggestion.
We also need to think how the controller pipeline works.
That means the targetprovider will now be across multiple services. Currently we have ability to plugin rebalancer and provisioners per resource. But sometimes this is too fine and might need logic that spans across resources and services.
I started another thread on composite rebalancers, I think we might need a similar concept in targetprovider, provisioners.
On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <ka...@hotmail.com> wrote:
Maybe it should be at both scopes? Sort of like putting bounds cross-app and cross-service within an app. This allows full control over an organization's physical resources.
> Date: Sat, 18 Jan 2014 09:28:32 -0800
> Subject: Re: helix provisioning
> From: g.kishore@gmail.com
> To: dev@helix.apache.org; user@helix.apache.org
>
> Thanks Swaroop for adding interfaces to specify the application spec.
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>
> From what I understand
> AppName maps to cluster name
> Service maps to instances, there can be multiple service components in an
> application. For example, in distributed data system, there may be a
> storage layer and router layer.
>
> The min/max containers seems to be at application level but it probably
> should be at a Service scope ?
>
> what do you think ?
>
>
>
>
>
>
>
>
>
>
> On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com> wrote:
>
> > Hi,
> >
> > Here is a prototype of supporting the concept of provisioning in Helix.
> > This was the work done by one of the interns at LinkedIn. You can find more
> > info here
> >
> > http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
> >
> > The implementation done by intern was outside of Helix. Kanak and I took a
> > stab at it to treat provisioning as a first class citizen of Helix.
> >
> > The work is in helix-provisioning branch
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
> >
> > Most of the code is here,
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
> >
> > Here is a YARN specific implementation
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
> >
> > Its still a WIP and main goal was to see if we can abstract Resource
> > Managers like YARN, Mesos, EC2 from the application.
> >
> > I am working on providing the steps to try out launching a Helix cluster
> > on YARN.
> >
> > Feedback/Thoughts? Any interest in contributing to make this production
> > ready ?
> >
> > thanks,
> > Kishore G
> >
> >
> >
> >
> >
> >
> >
> >
Re: helix provisioning
Posted by kishore g <g....@gmail.com>.
Here is a sample app spec. does it make sense ?
appName: test
configs:
k1: v1
services:
- name: myservice
participantClass: org.apache.helix.myApp.SimpleWebserver
minContainers:3
maxContainers:3
configs:
- participantId: myservice_0
port: 9500
- participantId: myservice_1
port: 9501
- participantId: myservice_2
port: 9502
resources:
- name: distributedLock
numPartitions: 6
numReplicas: 2
rebalanceMode: FULL_AUTO
stateModel: OnlineOffline
tag: myservice
configs:
k1: v1
On Sat, Jan 18, 2014 at 11:22 AM, kishore g <g....@gmail.com> wrote:
> Thats a good suggestion.
>
> We also need to think how the controller pipeline works.
>
> That means the targetprovider will now be across multiple services.
> Currently we have ability to plugin rebalancer and provisioners per
> resource. But sometimes this is too fine and might need logic that spans
> across resources and services.
>
> I started another thread on composite rebalancers, I think we might need a
> similar concept in targetprovider, provisioners.
>
>
>
>
> On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
>
>> Maybe it should be at both scopes? Sort of like putting bounds cross-app
>> and cross-service within an app. This allows full control over an
>> organization's physical resources.
>>
>> > Date: Sat, 18 Jan 2014 09:28:32 -0800
>> > Subject: Re: helix provisioning
>> > From: g.kishore@gmail.com
>> > To: dev@helix.apache.org; user@helix.apache.org
>>
>> >
>> > Thanks Swaroop for adding interfaces to specify the application spec.
>> >
>> >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>> >
>> > From what I understand
>> > AppName maps to cluster name
>> > Service maps to instances, there can be multiple service components in
>> an
>> > application. For example, in distributed data system, there may be a
>> > storage layer and router layer.
>> >
>> > The min/max containers seems to be at application level but it probably
>> > should be at a Service scope ?
>> >
>> > what do you think ?
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > Here is a prototype of supporting the concept of provisioning in
>> Helix.
>> > > This was the work done by one of the interns at LinkedIn. You can
>> find more
>> > > info here
>> > >
>> > >
>> http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
>> > >
>> > > The implementation done by intern was outside of Helix. Kanak and I
>> took a
>> > > stab at it to treat provisioning as a first class citizen of Helix.
>> > >
>> > > The work is in helix-provisioning branch
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
>> > >
>> > > Most of the code is here,
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
>> > >
>> > > Here is a YARN specific implementation
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
>> > >
>> > > Its still a WIP and main goal was to see if we can abstract Resource
>> > > Managers like YARN, Mesos, EC2 from the application.
>> > >
>> > > I am working on providing the steps to try out launching a Helix
>> cluster
>> > > on YARN.
>> > >
>> > > Feedback/Thoughts? Any interest in contributing to make this
>> production
>> > > ready ?
>> > >
>> > > thanks,
>> > > Kishore G
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>>
>
>
Re: helix provisioning
Posted by kishore g <g....@gmail.com>.
Here is a sample app spec. does it make sense ?
appName: test
configs:
k1: v1
services:
- name: myservice
participantClass: org.apache.helix.myApp.SimpleWebserver
minContainers:3
maxContainers:3
configs:
- participantId: myservice_0
port: 9500
- participantId: myservice_1
port: 9501
- participantId: myservice_2
port: 9502
resources:
- name: distributedLock
numPartitions: 6
numReplicas: 2
rebalanceMode: FULL_AUTO
stateModel: OnlineOffline
tag: myservice
configs:
k1: v1
On Sat, Jan 18, 2014 at 11:22 AM, kishore g <g....@gmail.com> wrote:
> Thats a good suggestion.
>
> We also need to think how the controller pipeline works.
>
> That means the targetprovider will now be across multiple services.
> Currently we have ability to plugin rebalancer and provisioners per
> resource. But sometimes this is too fine and might need logic that spans
> across resources and services.
>
> I started another thread on composite rebalancers, I think we might need a
> similar concept in targetprovider, provisioners.
>
>
>
>
> On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
>
>> Maybe it should be at both scopes? Sort of like putting bounds cross-app
>> and cross-service within an app. This allows full control over an
>> organization's physical resources.
>>
>> > Date: Sat, 18 Jan 2014 09:28:32 -0800
>> > Subject: Re: helix provisioning
>> > From: g.kishore@gmail.com
>> > To: dev@helix.apache.org; user@helix.apache.org
>>
>> >
>> > Thanks Swaroop for adding interfaces to specify the application spec.
>> >
>> >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>> >
>> > From what I understand
>> > AppName maps to cluster name
>> > Service maps to instances, there can be multiple service components in
>> an
>> > application. For example, in distributed data system, there may be a
>> > storage layer and router layer.
>> >
>> > The min/max containers seems to be at application level but it probably
>> > should be at a Service scope ?
>> >
>> > what do you think ?
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > Here is a prototype of supporting the concept of provisioning in
>> Helix.
>> > > This was the work done by one of the interns at LinkedIn. You can
>> find more
>> > > info here
>> > >
>> > >
>> http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
>> > >
>> > > The implementation done by intern was outside of Helix. Kanak and I
>> took a
>> > > stab at it to treat provisioning as a first class citizen of Helix.
>> > >
>> > > The work is in helix-provisioning branch
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
>> > >
>> > > Most of the code is here,
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
>> > >
>> > > Here is a YARN specific implementation
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
>> > >
>> > > Its still a WIP and main goal was to see if we can abstract Resource
>> > > Managers like YARN, Mesos, EC2 from the application.
>> > >
>> > > I am working on providing the steps to try out launching a Helix
>> cluster
>> > > on YARN.
>> > >
>> > > Feedback/Thoughts? Any interest in contributing to make this
>> production
>> > > ready ?
>> > >
>> > > thanks,
>> > > Kishore G
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>>
>
>
Re: helix provisioning
Posted by kishore g <g....@gmail.com>.
Thats a good suggestion.
We also need to think how the controller pipeline works.
That means the targetprovider will now be across multiple services.
Currently we have ability to plugin rebalancer and provisioners per
resource. But sometimes this is too fine and might need logic that spans
across resources and services.
I started another thread on composite rebalancers, I think we might need a
similar concept in targetprovider, provisioners.
On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
> Maybe it should be at both scopes? Sort of like putting bounds cross-app
> and cross-service within an app. This allows full control over an
> organization's physical resources.
>
> > Date: Sat, 18 Jan 2014 09:28:32 -0800
> > Subject: Re: helix provisioning
> > From: g.kishore@gmail.com
> > To: dev@helix.apache.org; user@helix.apache.org
>
> >
> > Thanks Swaroop for adding interfaces to specify the application spec.
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
> >
> > From what I understand
> > AppName maps to cluster name
> > Service maps to instances, there can be multiple service components in an
> > application. For example, in distributed data system, there may be a
> > storage layer and router layer.
> >
> > The min/max containers seems to be at application level but it probably
> > should be at a Service scope ?
> >
> > what do you think ?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Here is a prototype of supporting the concept of provisioning in Helix.
> > > This was the work done by one of the interns at LinkedIn. You can find
> more
> > > info here
> > >
> > >
> http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
> > >
> > > The implementation done by intern was outside of Helix. Kanak and I
> took a
> > > stab at it to treat provisioning as a first class citizen of Helix.
> > >
> > > The work is in helix-provisioning branch
> > >
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
> > >
> > > Most of the code is here,
> > >
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
> > >
> > > Here is a YARN specific implementation
> > >
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
> > >
> > > Its still a WIP and main goal was to see if we can abstract Resource
> > > Managers like YARN, Mesos, EC2 from the application.
> > >
> > > I am working on providing the steps to try out launching a Helix
> cluster
> > > on YARN.
> > >
> > > Feedback/Thoughts? Any interest in contributing to make this production
> > > ready ?
> > >
> > > thanks,
> > > Kishore G
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
>
Re: helix provisioning
Posted by kishore g <g....@gmail.com>.
Thats a good suggestion.
We also need to think how the controller pipeline works.
That means the targetprovider will now be across multiple services.
Currently we have ability to plugin rebalancer and provisioners per
resource. But sometimes this is too fine and might need logic that spans
across resources and services.
I started another thread on composite rebalancers, I think we might need a
similar concept in targetprovider, provisioners.
On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
> Maybe it should be at both scopes? Sort of like putting bounds cross-app
> and cross-service within an app. This allows full control over an
> organization's physical resources.
>
> > Date: Sat, 18 Jan 2014 09:28:32 -0800
> > Subject: Re: helix provisioning
> > From: g.kishore@gmail.com
> > To: dev@helix.apache.org; user@helix.apache.org
>
> >
> > Thanks Swaroop for adding interfaces to specify the application spec.
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
> >
> > From what I understand
> > AppName maps to cluster name
> > Service maps to instances, there can be multiple service components in an
> > application. For example, in distributed data system, there may be a
> > storage layer and router layer.
> >
> > The min/max containers seems to be at application level but it probably
> > should be at a Service scope ?
> >
> > what do you think ?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Here is a prototype of supporting the concept of provisioning in Helix.
> > > This was the work done by one of the interns at LinkedIn. You can find
> more
> > > info here
> > >
> > >
> http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
> > >
> > > The implementation done by intern was outside of Helix. Kanak and I
> took a
> > > stab at it to treat provisioning as a first class citizen of Helix.
> > >
> > > The work is in helix-provisioning branch
> > >
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
> > >
> > > Most of the code is here,
> > >
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
> > >
> > > Here is a YARN specific implementation
> > >
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
> > >
> > > Its still a WIP and main goal was to see if we can abstract Resource
> > > Managers like YARN, Mesos, EC2 from the application.
> > >
> > > I am working on providing the steps to try out launching a Helix
> cluster
> > > on YARN.
> > >
> > > Feedback/Thoughts? Any interest in contributing to make this production
> > > ready ?
> > >
> > > thanks,
> > > Kishore G
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
>
RE: helix provisioning
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Maybe it should be at both scopes? Sort of like putting bounds cross-app and cross-service within an app. This allows full control over an organization's physical resources.
> Date: Sat, 18 Jan 2014 09:28:32 -0800
> Subject: Re: helix provisioning
> From: g.kishore@gmail.com
> To: dev@helix.apache.org; user@helix.apache.org
>
> Thanks Swaroop for adding interfaces to specify the application spec.
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>
> From what I understand
> AppName maps to cluster name
> Service maps to instances, there can be multiple service components in an
> application. For example, in distributed data system, there may be a
> storage layer and router layer.
>
> The min/max containers seems to be at application level but it probably
> should be at a Service scope ?
>
> what do you think ?
>
>
>
>
>
>
>
>
>
>
> On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com> wrote:
>
> > Hi,
> >
> > Here is a prototype of supporting the concept of provisioning in Helix.
> > This was the work done by one of the interns at LinkedIn. You can find more
> > info here
> >
> > http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
> >
> > The implementation done by intern was outside of Helix. Kanak and I took a
> > stab at it to treat provisioning as a first class citizen of Helix.
> >
> > The work is in helix-provisioning branch
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
> >
> > Most of the code is here,
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
> >
> > Here is a YARN specific implementation
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
> >
> > Its still a WIP and main goal was to see if we can abstract Resource
> > Managers like YARN, Mesos, EC2 from the application.
> >
> > I am working on providing the steps to try out launching a Helix cluster
> > on YARN.
> >
> > Feedback/Thoughts? Any interest in contributing to make this production
> > ready ?
> >
> > thanks,
> > Kishore G
> >
> >
> >
> >
> >
> >
> >
> >
RE: helix provisioning
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Maybe it should be at both scopes? Sort of like putting bounds cross-app and cross-service within an app. This allows full control over an organization's physical resources.
> Date: Sat, 18 Jan 2014 09:28:32 -0800
> Subject: Re: helix provisioning
> From: g.kishore@gmail.com
> To: dev@helix.apache.org; user@helix.apache.org
>
> Thanks Swaroop for adding interfaces to specify the application spec.
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>
> From what I understand
> AppName maps to cluster name
> Service maps to instances, there can be multiple service components in an
> application. For example, in distributed data system, there may be a
> storage layer and router layer.
>
> The min/max containers seems to be at application level but it probably
> should be at a Service scope ?
>
> what do you think ?
>
>
>
>
>
>
>
>
>
>
> On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com> wrote:
>
> > Hi,
> >
> > Here is a prototype of supporting the concept of provisioning in Helix.
> > This was the work done by one of the interns at LinkedIn. You can find more
> > info here
> >
> > http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
> >
> > The implementation done by intern was outside of Helix. Kanak and I took a
> > stab at it to treat provisioning as a first class citizen of Helix.
> >
> > The work is in helix-provisioning branch
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
> >
> > Most of the code is here,
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
> >
> > Here is a YARN specific implementation
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
> >
> > Its still a WIP and main goal was to see if we can abstract Resource
> > Managers like YARN, Mesos, EC2 from the application.
> >
> > I am working on providing the steps to try out launching a Helix cluster
> > on YARN.
> >
> > Feedback/Thoughts? Any interest in contributing to make this production
> > ready ?
> >
> > thanks,
> > Kishore G
> >
> >
> >
> >
> >
> >
> >
> >
Re: helix provisioning
Posted by kishore g <g....@gmail.com>.
Thanks Swaroop for adding interfaces to specify the application spec.
https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>From what I understand
AppName maps to cluster name
Service maps to instances, there can be multiple service components in an
application. For example, in distributed data system, there may be a
storage layer and router layer.
The min/max containers seems to be at application level but it probably
should be at a Service scope ?
what do you think ?
On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com> wrote:
> Hi,
>
> Here is a prototype of supporting the concept of provisioning in Helix.
> This was the work done by one of the interns at LinkedIn. You can find more
> info here
>
> http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
>
> The implementation done by intern was outside of Helix. Kanak and I took a
> stab at it to treat provisioning as a first class citizen of Helix.
>
> The work is in helix-provisioning branch
>
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
>
> Most of the code is here,
>
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
>
> Here is a YARN specific implementation
>
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
>
> Its still a WIP and main goal was to see if we can abstract Resource
> Managers like YARN, Mesos, EC2 from the application.
>
> I am working on providing the steps to try out launching a Helix cluster
> on YARN.
>
> Feedback/Thoughts? Any interest in contributing to make this production
> ready ?
>
> thanks,
> Kishore G
>
>
>
>
>
>
>
>
Re: helix provisioning
Posted by kishore g <g....@gmail.com>.
Thanks Swaroop for adding interfaces to specify the application spec.
https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>From what I understand
AppName maps to cluster name
Service maps to instances, there can be multiple service components in an
application. For example, in distributed data system, there may be a
storage layer and router layer.
The min/max containers seems to be at application level but it probably
should be at a Service scope ?
what do you think ?
On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g....@gmail.com> wrote:
> Hi,
>
> Here is a prototype of supporting the concept of provisioning in Helix.
> This was the work done by one of the interns at LinkedIn. You can find more
> info here
>
> http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
>
> The implementation done by intern was outside of Helix. Kanak and I took a
> stab at it to treat provisioning as a first class citizen of Helix.
>
> The work is in helix-provisioning branch
>
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
>
> Most of the code is here,
>
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
>
> Here is a YARN specific implementation
>
>
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
>
> Its still a WIP and main goal was to see if we can abstract Resource
> Managers like YARN, Mesos, EC2 from the application.
>
> I am working on providing the steps to try out launching a Helix cluster
> on YARN.
>
> Feedback/Thoughts? Any interest in contributing to make this production
> ready ?
>
> thanks,
> Kishore G
>
>
>
>
>
>
>
>