You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by kishore g <g....@gmail.com> on 2013/02/14 18:14:26 UTC
Re: Specifying partition placement using AUTO
Hi Ming,
There is an example
https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=blob;f=helix-core/src/main/java/org/apache/helix/examples/IdealStateExample.java;hb=437eb42e
.
admin.addResource(clusterName, resourceName, numPartitions,
stateModelName, "AUTO");
// rebalance resource "TestDB" using 3 replicas
admin.rebalance(clusterName, resourceName, numReplica);
This will automatically do the partition placement among the nodes in the
cluster. We use a mix of consistent hashing and crush
http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf to do the placement.
But i think you need a way to come up with your own logic of assigning the
replica set for a partition. You can find an example for that in
BaseStageTest. See setupIdealState
https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=blob;f=helix-core/src/test/java/org/apache/helix/controller/stages/BaseStageTest.java;h=ed54e6facdc17b40bd8eeb56c4cfaa5ad39ea91b;hb=437eb42e
Here is the code snippet.http://pastebin.com/e2LXtq46.
Hope this helps
Thanks,
Kishore G
On Thu, Feb 14, 2013 at 8:03 AM, Ming Fang <mi...@mac.com> wrote:
> Can someone point me to an example of how to specify partition placement
> when using AUTO mode?
> The link here http://helix.incubator.apache.org/Features.html says it's
> possible but I was not able to find any example or unit test that does this.
> Thanks
> --ming
Re: Potential bug in manual partition placement
Posted by Ming Fang <mi...@mac.com>.
Thanks for the detailed explanation.
I've learned a lot as a result.
Sent from my iPad
On Feb 22, 2013, at 1:12 AM, kishore g <g....@gmail.com> wrote:
> Hi Ming,
>
> It is easier to understand if you look at the transition order in the first email you sent.
> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_0
> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_1
> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_1
> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_0
>
> If you see Helix at this point Helix has not sent any transition from OFFLINE to SLAVE to localhost_12001, this is because you have set the constraint that max number of nodes that can be in SLAVE state is 2 ( replicas=2) in the state model definition.
>
> For MyResource_1 Since localhost_12002 and localhost_12000 are already slave, localhost_12001 can never become a slave since that would violate the constraint of slave <2. Since it cannot become slave, it cannot become Master.
>
> For MyResource_0, you can see that it first made localhost_12000 master and hence it could send message to localhost_12001 to become Slave.
>
> localhost_12000 transitioning from SLAVE to MASTER for MyResource_0
> localhost_12001 transitioning from OFFLINE to SLAVE for MyResource_0
>
> Helix-50 fixes the random selection of nodes to sort messages based on the preference list.
>
> Thanks,
> Kishore G
>
>
> On Thu, Feb 21, 2013 at 4:42 PM, Zhen Zhang <ne...@gmail.com> wrote:
>> Hi Ming, thanks the feedback. With REPLICAS set to 2, it's a random behavior that Helix controller will pick up any two of the hosts in the preference list and do the transitions. In your case it happens that it will work fine. We have updated the jira accordingly and will fix it soon.
>> https://issues.apache.org/jira/browse/HELIX-50
>>
>> Thanks,
>> Zhen
>>
>> On Thu, Feb 21, 2013 at 4:34 PM, Ming Fang <mi...@mac.com> wrote:
>>> Thanks for pointing that out.
>>> It does work was expected after I set REPLICAS to 3.
>>>
>>> But the strange thing is even with REPLICAS set to 2 and placement configured as below, everything works.
>>> "MyResource_0" : [ "localhost_12000", "localhost_12001", "localhost_12002" ],
>>> "MyResource_1" : [ "localhost_12000", "localhost_12001", "localhost_12002" ]
>>>
>>> On Feb 20, 2013, at 2:02 AM, kishore g <g....@gmail.com> wrote:
>>>
>>>> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json has replicas set to 2 but the preference list for each partition is of size 3. If you set the number of REPLICAS to 3, it should work.
>>>>
>>>> We do some validation of the idealstate but we dont validate that number of replicas is same as the preference list size for all partitions. Created JIRA https://issues.apache.org/jira/browse/HELIX-50
>>>>
>>>>
>>>> Thanks,
>>>> Kishore G
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Feb 19, 2013 at 7:08 PM, Ming Fang <mi...@mac.com> wrote:
>>>>> I've "repurpose" the Quickstart example in an attempt to implement manual placement of partitions.
>>>>> I'm using JSON file and the relevant section is below
>>>>>
>>>>> "MyResource_0" : [ "localhost_12000", "localhost_12001", "localhost_12002" ],
>>>>> "MyResource_1" : [ "localhost_12001", "localhost_12000", "localhost_12002" ]
>>>>>
>>>>> The goal is to make _12000 the MASTER for MyResource_0 and _12001 the MASTER of MyResource_1.
>>>>> The last instance, _12002 will serve as the last resort backup for both partitions in the event the other two died.
>>>>> This is a small example of what I was hoping to implement as part of a larger system.
>>>>>
>>>>> You may run the example here
>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/java/org/apache/helix/examples/ManualPlacementTest.java
>>>>>
>>>>> using the JSON file here
>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json
>>>>>
>>>>> The problem is when I run this, the output looks like this
>>>>>
>>>>> STARTING Zookeeper at localhost:2199
>>>>> Creating cluster: HELIX_QUICKSTART
>>>>> Adding 3 participants to the cluster
>>>>> Added participant: localhost_12000
>>>>> Added participant: localhost_12001
>>>>> Added participant: localhost_12002
>>>>> Starting Participants
>>>>> Started Participant: localhost_12000
>>>>> Started Participant: localhost_12001
>>>>> Started Participant: localhost_12002
>>>>> Starting Helix Controller
>>>>> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_0
>>>>> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_1
>>>>> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_1
>>>>> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_0
>>>>> localhost_12000 transitioning from SLAVE to MASTER for MyResource_0
>>>>> localhost_12001 transitioning from OFFLINE to SLAVE for MyResource_0
>>>>> CLUSTER STATE: After starting 3 nodes
>>>>> localhost_12000 localhost_12001 localhost_12002
>>>>> MyResource_0 M S S
>>>>> MyResource_1 S - S
>>>>> ###################################################################
>>>>>
>>>>> Notice there is no MASTER for MyResource_1.
>>>>> I've been trying to debug this for a day now with no success.
>>>>>
>>>>> Did I stumble onto an actual bug?
>>>>
>>>
>>
>
Re: Potential bug in manual partition placement
Posted by kishore g <g....@gmail.com>.
Hi Ming,
It is easier to understand if you look at the transition order in the first
email you sent.
localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_0
localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_1
localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_1
localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_0
If you see Helix at this point Helix has not sent any transition from
OFFLINE to SLAVE to localhost_12001, this is because you have set the
constraint that max number of nodes that can be in SLAVE state is 2 (
replicas=2) in the state model definition.
For MyResource_1 Since localhost_12002 and localhost_12000 are already
slave, localhost_12001 can never become a slave since that would violate
the constraint of slave <2. Since it cannot become slave, it cannot become
Master.
For MyResource_0, you can see that it first made localhost_12000 master and
hence it could send message to localhost_12001 to become Slave.
localhost_12000 transitioning from SLAVE to MASTER for MyResource_0
localhost_12001 transitioning from OFFLINE to SLAVE for MyResource_0
Helix-50 fixes the random selection of nodes to sort messages based on the
preference list.
Thanks,
Kishore G
On Thu, Feb 21, 2013 at 4:42 PM, Zhen Zhang <ne...@gmail.com> wrote:
> Hi Ming, thanks the feedback. With REPLICAS set to 2, it's a random
> behavior that Helix controller will pick up any two of the hosts in the
> preference list and do the transitions. In your case it happens that it
> will work fine. We have updated the jira accordingly and will fix it soon.
> https://issues.apache.org/jira/browse/HELIX-50
>
> Thanks,
> Zhen
>
> On Thu, Feb 21, 2013 at 4:34 PM, Ming Fang <mi...@mac.com> wrote:
>
>> Thanks for pointing that out.
>> It does work was expected after I set REPLICAS to 3.
>>
>> But the strange thing is even with REPLICAS set to 2 and placement
>> configured as below, everything works.
>> "MyResource_0" : [ "localhost_12000", "localhost_12001",
>> "localhost_12002" ],
>> "MyResource_1" : [ "localhost_12000", "localhost_12001",
>> "localhost_12002" ]
>>
>> On Feb 20, 2013, at 2:02 AM, kishore g <g....@gmail.com> wrote:
>>
>>
>> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json has
>> replicas set to 2 but the preference list for each partition is of size 3.
>> If you set the number of REPLICAS to 3, it should work.
>>
>> We do some validation of the idealstate but we dont validate that number
>> of replicas is same as the preference list size for all partitions. Created
>> JIRA https://issues.apache.org/jira/browse/HELIX-50
>>
>>
>> Thanks,
>> Kishore G
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Feb 19, 2013 at 7:08 PM, Ming Fang <mi...@mac.com> wrote:
>>
>>> I've "repurpose" the Quickstart example in an attempt to implement
>>> manual placement of partitions.
>>> I'm using JSON file and the relevant section is below
>>>
>>> "MyResource_0" : [ "localhost_12000", "localhost_12001",
>>> "localhost_12002" ],
>>> "MyResource_1" : [ "localhost_12001", "localhost_12000",
>>> "localhost_12002" ]
>>>
>>> The goal is to make _12000 the MASTER for MyResource_0 and _12001 the
>>> MASTER of MyResource_1.
>>> The last instance, _12002 will serve as the last resort backup for both
>>> partitions in the event the other two died.
>>> This is a small example of what I was hoping to implement as part of a
>>> larger system.
>>>
>>> You may run the example here
>>>
>>> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/java/org/apache/helix/examples/ManualPlacementTest.java
>>>
>>> using the JSON file here
>>>
>>> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json
>>>
>>> The problem is when I run this, the output looks like this
>>>
>>> STARTING Zookeeper at localhost:2199
>>> Creating cluster: HELIX_QUICKSTART
>>> Adding 3 participants to the cluster
>>> Added participant: localhost_12000
>>> Added participant: localhost_12001
>>> Added participant: localhost_12002
>>> Starting Participants
>>> Started Participant: localhost_12000
>>> Started Participant: localhost_12001
>>> Started Participant: localhost_12002
>>> Starting Helix Controller
>>> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_0
>>> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_1
>>> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_1
>>> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_0
>>> localhost_12000 transitioning from SLAVE to MASTER for MyResource_0
>>> localhost_12001 transitioning from OFFLINE to SLAVE for MyResource_0
>>> CLUSTER STATE: After starting 3 nodes
>>> localhost_12000 localhost_12001 localhost_12002
>>> MyResource_0 M S S
>>> MyResource_1 S - S
>>> ###################################################################
>>>
>>> Notice there is no MASTER for MyResource_1.
>>> I've been trying to debug this for a day now with no success.
>>>
>>> Did I stumble onto an actual bug?
>>
>>
>>
>>
>
Re: Potential bug in manual partition placement
Posted by Zhen Zhang <ne...@gmail.com>.
Hi Ming, thanks the feedback. With REPLICAS set to 2, it's a random
behavior that Helix controller will pick up any two of the hosts in the
preference list and do the transitions. In your case it happens that it
will work fine. We have updated the jira accordingly and will fix it soon.
https://issues.apache.org/jira/browse/HELIX-50
Thanks,
Zhen
On Thu, Feb 21, 2013 at 4:34 PM, Ming Fang <mi...@mac.com> wrote:
> Thanks for pointing that out.
> It does work was expected after I set REPLICAS to 3.
>
> But the strange thing is even with REPLICAS set to 2 and placement
> configured as below, everything works.
> "MyResource_0" : [ "localhost_12000", "localhost_12001",
> "localhost_12002" ],
> "MyResource_1" : [ "localhost_12000", "localhost_12001",
> "localhost_12002" ]
>
> On Feb 20, 2013, at 2:02 AM, kishore g <g....@gmail.com> wrote:
>
>
> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json has
> replicas set to 2 but the preference list for each partition is of size 3.
> If you set the number of REPLICAS to 3, it should work.
>
> We do some validation of the idealstate but we dont validate that number
> of replicas is same as the preference list size for all partitions. Created
> JIRA https://issues.apache.org/jira/browse/HELIX-50
>
>
> Thanks,
> Kishore G
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Feb 19, 2013 at 7:08 PM, Ming Fang <mi...@mac.com> wrote:
>
>> I've "repurpose" the Quickstart example in an attempt to implement manual
>> placement of partitions.
>> I'm using JSON file and the relevant section is below
>>
>> "MyResource_0" : [ "localhost_12000", "localhost_12001",
>> "localhost_12002" ],
>> "MyResource_1" : [ "localhost_12001", "localhost_12000",
>> "localhost_12002" ]
>>
>> The goal is to make _12000 the MASTER for MyResource_0 and _12001 the
>> MASTER of MyResource_1.
>> The last instance, _12002 will serve as the last resort backup for both
>> partitions in the event the other two died.
>> This is a small example of what I was hoping to implement as part of a
>> larger system.
>>
>> You may run the example here
>>
>> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/java/org/apache/helix/examples/ManualPlacementTest.java
>>
>> using the JSON file here
>>
>> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json
>>
>> The problem is when I run this, the output looks like this
>>
>> STARTING Zookeeper at localhost:2199
>> Creating cluster: HELIX_QUICKSTART
>> Adding 3 participants to the cluster
>> Added participant: localhost_12000
>> Added participant: localhost_12001
>> Added participant: localhost_12002
>> Starting Participants
>> Started Participant: localhost_12000
>> Started Participant: localhost_12001
>> Started Participant: localhost_12002
>> Starting Helix Controller
>> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_0
>> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_1
>> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_1
>> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_0
>> localhost_12000 transitioning from SLAVE to MASTER for MyResource_0
>> localhost_12001 transitioning from OFFLINE to SLAVE for MyResource_0
>> CLUSTER STATE: After starting 3 nodes
>> localhost_12000 localhost_12001 localhost_12002
>> MyResource_0 M S S
>> MyResource_1 S - S
>> ###################################################################
>>
>> Notice there is no MASTER for MyResource_1.
>> I've been trying to debug this for a day now with no success.
>>
>> Did I stumble onto an actual bug?
>
>
>
>
Re: Potential bug in manual partition placement
Posted by Ming Fang <mi...@mac.com>.
Thanks for pointing that out.
It does work was expected after I set REPLICAS to 3.
But the strange thing is even with REPLICAS set to 2 and placement configured as below, everything works.
"MyResource_0" : [ "localhost_12000", "localhost_12001", "localhost_12002" ],
"MyResource_1" : [ "localhost_12000", "localhost_12001", "localhost_12002" ]
On Feb 20, 2013, at 2:02 AM, kishore g <g....@gmail.com> wrote:
> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json has replicas set to 2 but the preference list for each partition is of size 3. If you set the number of REPLICAS to 3, it should work.
>
> We do some validation of the idealstate but we dont validate that number of replicas is same as the preference list size for all partitions. Created JIRA https://issues.apache.org/jira/browse/HELIX-50
>
>
> Thanks,
> Kishore G
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Feb 19, 2013 at 7:08 PM, Ming Fang <mi...@mac.com> wrote:
> I've "repurpose" the Quickstart example in an attempt to implement manual placement of partitions.
> I'm using JSON file and the relevant section is below
>
> "MyResource_0" : [ "localhost_12000", "localhost_12001", "localhost_12002" ],
> "MyResource_1" : [ "localhost_12001", "localhost_12000", "localhost_12002" ]
>
> The goal is to make _12000 the MASTER for MyResource_0 and _12001 the MASTER of MyResource_1.
> The last instance, _12002 will serve as the last resort backup for both partitions in the event the other two died.
> This is a small example of what I was hoping to implement as part of a larger system.
>
> You may run the example here
> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/java/org/apache/helix/examples/ManualPlacementTest.java
>
> using the JSON file here
> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json
>
> The problem is when I run this, the output looks like this
>
> STARTING Zookeeper at localhost:2199
> Creating cluster: HELIX_QUICKSTART
> Adding 3 participants to the cluster
> Added participant: localhost_12000
> Added participant: localhost_12001
> Added participant: localhost_12002
> Starting Participants
> Started Participant: localhost_12000
> Started Participant: localhost_12001
> Started Participant: localhost_12002
> Starting Helix Controller
> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_0
> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_1
> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_1
> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_0
> localhost_12000 transitioning from SLAVE to MASTER for MyResource_0
> localhost_12001 transitioning from OFFLINE to SLAVE for MyResource_0
> CLUSTER STATE: After starting 3 nodes
> localhost_12000 localhost_12001 localhost_12002
> MyResource_0 M S S
> MyResource_1 S - S
> ###################################################################
>
> Notice there is no MASTER for MyResource_1.
> I've been trying to debug this for a day now with no success.
>
> Did I stumble onto an actual bug?
>
Re: Potential bug in manual partition placement
Posted by kishore g <g....@gmail.com>.
https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json
has
replicas set to 2 but the preference list for each partition is of size 3.
If you set the number of REPLICAS to 3, it should work.
We do some validation of the idealstate but we dont validate that number of
replicas is same as the preference list size for all partitions. Created
JIRA https://issues.apache.org/jira/browse/HELIX-50
Thanks,
Kishore G
On Tue, Feb 19, 2013 at 7:08 PM, Ming Fang <mi...@mac.com> wrote:
> I've "repurpose" the Quickstart example in an attempt to implement manual
> placement of partitions.
> I'm using JSON file and the relevant section is below
>
> "MyResource_0" : [ "localhost_12000", "localhost_12001",
> "localhost_12002" ],
> "MyResource_1" : [ "localhost_12001", "localhost_12000",
> "localhost_12002" ]
>
> The goal is to make _12000 the MASTER for MyResource_0 and _12001 the
> MASTER of MyResource_1.
> The last instance, _12002 will serve as the last resort backup for both
> partitions in the event the other two died.
> This is a small example of what I was hoping to implement as part of a
> larger system.
>
> You may run the example here
>
> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/java/org/apache/helix/examples/ManualPlacementTest.java
>
> using the JSON file here
>
> https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json
>
> The problem is when I run this, the output looks like this
>
> STARTING Zookeeper at localhost:2199
> Creating cluster: HELIX_QUICKSTART
> Adding 3 participants to the cluster
> Added participant: localhost_12000
> Added participant: localhost_12001
> Added participant: localhost_12002
> Starting Participants
> Started Participant: localhost_12000
> Started Participant: localhost_12001
> Started Participant: localhost_12002
> Starting Helix Controller
> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_0
> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_1
> localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_1
> localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_0
> localhost_12000 transitioning from SLAVE to MASTER for MyResource_0
> localhost_12001 transitioning from OFFLINE to SLAVE for MyResource_0
> CLUSTER STATE: After starting 3 nodes
> localhost_12000 localhost_12001 localhost_12002
> MyResource_0 M S S
> MyResource_1 S - S
> ###################################################################
>
> Notice there is no MASTER for MyResource_1.
> I've been trying to debug this for a day now with no success.
>
> Did I stumble onto an actual bug?
Potential bug in manual partition placement
Posted by Ming Fang <mi...@mac.com>.
I've "repurpose" the Quickstart example in an attempt to implement manual placement of partitions.
I'm using JSON file and the relevant section is below
"MyResource_0" : [ "localhost_12000", "localhost_12001", "localhost_12002" ],
"MyResource_1" : [ "localhost_12001", "localhost_12000", "localhost_12002" ]
The goal is to make _12000 the MASTER for MyResource_0 and _12001 the MASTER of MyResource_1.
The last instance, _12002 will serve as the last resort backup for both partitions in the event the other two died.
This is a small example of what I was hoping to implement as part of a larger system.
You may run the example here
https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/java/org/apache/helix/examples/ManualPlacementTest.java
using the JSON file here
https://github.com/mingfang/apache-helix/blob/master/helix-core/src/main/resources/manual.json
The problem is when I run this, the output looks like this
STARTING Zookeeper at localhost:2199
Creating cluster: HELIX_QUICKSTART
Adding 3 participants to the cluster
Added participant: localhost_12000
Added participant: localhost_12001
Added participant: localhost_12002
Starting Participants
Started Participant: localhost_12000
Started Participant: localhost_12001
Started Participant: localhost_12002
Starting Helix Controller
localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_0
localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_1
localhost_12000 transitioning from OFFLINE to SLAVE for MyResource_1
localhost_12002 transitioning from OFFLINE to SLAVE for MyResource_0
localhost_12000 transitioning from SLAVE to MASTER for MyResource_0
localhost_12001 transitioning from OFFLINE to SLAVE for MyResource_0
CLUSTER STATE: After starting 3 nodes
localhost_12000 localhost_12001 localhost_12002
MyResource_0 M S S
MyResource_1 S - S
###################################################################
Notice there is no MASTER for MyResource_1.
I've been trying to debug this for a day now with no success.
Did I stumble onto an actual bug?
Re: Specifying partition placement using AUTO
Posted by kishore g <g....@gmail.com>.
Glad it worked for you and thanks for the suggestions. MasterSlave,
LeaderStandby, OnlineOffline does come by default. If you create the
cluster using the command line tool, then those state models are created by
default. But if you use helix admin api, it probably does not. There should
probably be an api to add default state
models.
You are right, you can use JSON file option to upload for any mode. Will
update the docs accordingly.
Thanks,
Kishore G
On Mon, Feb 18, 2013 at 8:58 PM, Ming Fang <mi...@mac.com> wrote:
> Thanks Kishore's help I was able to get this working.
>
> I just wanted to point out a couple of things in hopes of helping others.
>
> 1-I had to add the MasterSlave model def like this
> admin.addStateModelDef(CLUSTER_NAME, "MasterSlave", new
> StateModelDefinition(
> new
> StateModelConfigGenerator().generateConfigForMasterSlave()));
>
> At first I thought MasterSlave was a special model that was already
> registered by default. Maybe it should be.
>
> 2-I was able to use AUTO mode with a JSON file. The documentation
> suggests that this was only possible with CUSTOMIZED mode, but it appears
> to work for AUTO too.
> I prefer using a JSON file to create the ideal state.
>
> Finally thanks to the Helix team for giving us an excellent product.
>
> On Feb 14, 2013, at 12:14 PM, kishore g <g....@gmail.com> wrote:
>
> Hi Ming,
>
> There is an example
> https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=blob;f=helix-core/src/main/java/org/apache/helix/examples/IdealStateExample.java;hb=437eb42e
> .
>
> admin.addResource(clusterName, resourceName, numPartitions, stateModelName, "AUTO");
> // rebalance resource "TestDB" using 3 replicas
> admin.rebalance(clusterName, resourceName, numReplica);
>
> This will automatically do the partition placement among the nodes in the
> cluster. We use a mix of consistent hashing and crush
> http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf to do the placement.
>
> But i think you need a way to come up with your own logic of assigning the
> replica set for a partition. You can find an example for that in
> BaseStageTest. See setupIdealState
> https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=blob;f=helix-core/src/test/java/org/apache/helix/controller/stages/BaseStageTest.java;h=ed54e6facdc17b40bd8eeb56c4cfaa5ad39ea91b;hb=437eb42e
>
> Here is the code snippet.http://pastebin.com/e2LXtq46.
>
> Hope this helps
>
> Thanks,
> Kishore G
>
>
>
>
>
>
>
> On Thu, Feb 14, 2013 at 8:03 AM, Ming Fang <mi...@mac.com> wrote:
>
>> Can someone point me to an example of how to specify partition placement
>> when using AUTO mode?
>> The link here http://helix.incubator.apache.org/Features.html says it's
>> possible but I was not able to find any example or unit test that does this.
>> Thanks
>> --ming
>
>
>
>
Re: Specifying partition placement using AUTO
Posted by Ming Fang <mi...@mac.com>.
Thanks Kishore's help I was able to get this working.
I just wanted to point out a couple of things in hopes of helping others.
1-I had to add the MasterSlave model def like this
admin.addStateModelDef(CLUSTER_NAME, "MasterSlave", new StateModelDefinition(
new StateModelConfigGenerator().generateConfigForMasterSlave()));
At first I thought MasterSlave was a special model that was already registered by default. Maybe it should be.
2-I was able to use AUTO mode with a JSON file. The documentation suggests that this was only possible with CUSTOMIZED mode, but it appears to work for AUTO too.
I prefer using a JSON file to create the ideal state.
Finally thanks to the Helix team for giving us an excellent product.
On Feb 14, 2013, at 12:14 PM, kishore g <g....@gmail.com> wrote:
> Hi Ming,
>
> There is an example https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=blob;f=helix-core/src/main/java/org/apache/helix/examples/IdealStateExample.java;hb=437eb42e.
> admin.addResource(clusterName, resourceName, numPartitions, stateModelName, "AUTO");
>
> // rebalance resource "TestDB" using 3 replicas
> admin.rebalance(clusterName, resourceName, numReplica);
>
> This will automatically do the partition placement among the nodes in the cluster. We use a mix of consistent hashing and crush http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf to do the placement.
>
> But i think you need a way to come up with your own logic of assigning the replica set for a partition. You can find an example for that in BaseStageTest. See setupIdealState https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=blob;f=helix-core/src/test/java/org/apache/helix/controller/stages/BaseStageTest.java;h=ed54e6facdc17b40bd8eeb56c4cfaa5ad39ea91b;hb=437eb42e
>
> Here is the code snippet.http://pastebin.com/e2LXtq46.
>
> Hope this helps
>
> Thanks,
> Kishore G
>
>
>
>
>
>
>
> On Thu, Feb 14, 2013 at 8:03 AM, Ming Fang <mi...@mac.com> wrote:
> Can someone point me to an example of how to specify partition placement when using AUTO mode?
> The link here http://helix.incubator.apache.org/Features.html says it's possible but I was not able to find any example or unit test that does this.
> Thanks
> --ming
>
Re: Specifying partition placement using AUTO
Posted by Ming Fang <mi...@mac.com>.
Thanks Kishore.
Sent from my iPad
On Feb 14, 2013, at 12:14 PM, kishore g <g....@gmail.com> wrote:
> Hi Ming,
>
> There is an example
> https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=blob;f=helix-core/src/main/java/org/apache/helix/examples/IdealStateExample.java;hb=437eb42e
> .
> admin.addResource(clusterName, resourceName, numPartitions,
> stateModelName, "AUTO");
> // rebalance resource "TestDB" using 3 replicas
> admin.rebalance(clusterName, resourceName, numReplica);
>
> This will automatically do the partition placement among the nodes in the
> cluster. We use a mix of consistent hashing and crush
> http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf to do the placement.
>
> But i think you need a way to come up with your own logic of assigning the
> replica set for a partition. You can find an example for that in
> BaseStageTest. See setupIdealState
> https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=blob;f=helix-core/src/test/java/org/apache/helix/controller/stages/BaseStageTest.java;h=ed54e6facdc17b40bd8eeb56c4cfaa5ad39ea91b;hb=437eb42e
>
> Here is the code snippet.http://pastebin.com/e2LXtq46.
>
> Hope this helps
>
> Thanks,
> Kishore G
>
>
>
>
>
>
>
> On Thu, Feb 14, 2013 at 8:03 AM, Ming Fang <mi...@mac.com> wrote:
>
>> Can someone point me to an example of how to specify partition placement
>> when using AUTO mode?
>> The link here http://helix.incubator.apache.org/Features.html says it's
>> possible but I was not able to find any example or unit test that does this.
>> Thanks
>> --ming