You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@helix.apache.org by Neutron sharc <ne...@gmail.com> on 2016/05/02 21:38:21 UTC
Re: want to get dead replicas in USER_DEFINED rebalance callback
Thanks Kishore for your reply!
What I see is that, a resource's ideal state becomes empty after
external view converges with the assignment.
When I create a resource I compute an initial IS, and attach a
USER_DEFINED rebalancer. After Helix stabilizes a resource, its
"IDEALSTATES" mapFields is wiped off in zookeeper. When the next
round of rebalancing starts, computeResourceMapping() will always get
an empty idealState.
On Fri, Apr 29, 2016 at 1:16 PM, kishore g <g....@gmail.com> wrote:
> Hi,
>
> Current state will not show dead replicas. You need to use previous
> idealstate to derive that info. The logic will be something like this
>
> computeResource(.....) {
> List<Instances> instances =
> previousIdealState.getInstancesForPartition(P0)
> foreach instance
> if(!liveInstances.contain(instance)){
> //NEED TO ASSIGN ANOTHER INSTANCE TO FOR THIS PARTITION
> }
> }
>
> This allows your logic to be idempotent and not depend on incremental
> changes.
>
> thanks,
> Kishore G
>
> On Thu, Apr 28, 2016 at 4:27 PM, Neutron sharc <ne...@gmail.com>
> wrote:
>
>> Hi team,
>>
>> in USER_DEFINED rebalance mode, the callback computeResourceMapping()
>> accepts a “currentState”. Does this variable include replicas on a
>> dead participant ?
>>
>> For example, my resource has a partition P1 master replica on
>> participant node1, a slave replica on participant node2. When node1
>> dies, in callback computeResourceMapping() I retrieve P1’s replicas:
>>
>> Map<ParticipantId, State> replicas =
>> currentState.getCurrentStateMap(resourceId, partitionId);
>>
>>
>> Here the “replicas” includes only node2, there is no entry for node1.
>>
>> However, I want to know all replicas including dead ones, so that I
>> can know that a master replica is gone and I should failover to an
>> existing slave, instead of starting a new master.
>>
>>
>> Appreciate any comments!
>>
>>
>> -Neutron
>>
Re: want to get dead replicas in USER_DEFINED rebalance callback
Posted by Neutron sharc <ne...@gmail.com>.
This is the simpleFields of my IS:
"simpleFields" : {
"IDEAL_STATE_MODE" : "AUTO",
"MAX_PARTITIONS_PER_INSTANCE" : "1",
"NUM_PARTITIONS" : "3",
"REBALANCER_CLASS_NAME" : "com.hcd.hcdadmin.M1Rebalancer",
"REBALANCE_MODE" : "USER_DEFINED",
"REPLICAS" : "3",
"STATE_MODEL_DEF_REF" : "M1StateModel",
"STATE_MODEL_FACTORY_NAME" : "DEFAULT"
}
On Mon, May 2, 2016 at 12:49 PM, kishore g <g....@gmail.com> wrote:
> Can you paste the initial IS that you compute. I am mainly interested in
> the simple fields.
>
> On Mon, May 2, 2016 at 12:38 PM, Neutron sharc <ne...@gmail.com>
> wrote:
>
>> Thanks Kishore for your reply!
>>
>> What I see is that, a resource's ideal state becomes empty after
>> external view converges with the assignment.
>>
>> When I create a resource I compute an initial IS, and attach a
>> USER_DEFINED rebalancer. After Helix stabilizes a resource, its
>> "IDEALSTATES" mapFields is wiped off in zookeeper. When the next
>> round of rebalancing starts, computeResourceMapping() will always get
>> an empty idealState.
>>
>>
>>
>>
>> On Fri, Apr 29, 2016 at 1:16 PM, kishore g <g....@gmail.com> wrote:
>> > Hi,
>> >
>> > Current state will not show dead replicas. You need to use previous
>> > idealstate to derive that info. The logic will be something like this
>> >
>> > computeResource(.....) {
>> > List<Instances> instances =
>> > previousIdealState.getInstancesForPartition(P0)
>> > foreach instance
>> > if(!liveInstances.contain(instance)){
>> > //NEED TO ASSIGN ANOTHER INSTANCE TO FOR THIS PARTITION
>> > }
>> > }
>> >
>> > This allows your logic to be idempotent and not depend on incremental
>> > changes.
>> >
>> > thanks,
>> > Kishore G
>> >
>> > On Thu, Apr 28, 2016 at 4:27 PM, Neutron sharc <ne...@gmail.com>
>> > wrote:
>> >
>> >> Hi team,
>> >>
>> >> in USER_DEFINED rebalance mode, the callback computeResourceMapping()
>> >> accepts a “currentState”. Does this variable include replicas on a
>> >> dead participant ?
>> >>
>> >> For example, my resource has a partition P1 master replica on
>> >> participant node1, a slave replica on participant node2. When node1
>> >> dies, in callback computeResourceMapping() I retrieve P1’s replicas:
>> >>
>> >> Map<ParticipantId, State> replicas =
>> >> currentState.getCurrentStateMap(resourceId, partitionId);
>> >>
>> >>
>> >> Here the “replicas” includes only node2, there is no entry for node1.
>> >>
>> >> However, I want to know all replicas including dead ones, so that I
>> >> can know that a master replica is gone and I should failover to an
>> >> existing slave, instead of starting a new master.
>> >>
>> >>
>> >> Appreciate any comments!
>> >>
>> >>
>> >> -Neutron
>> >>
>>
Re: want to get dead replicas in USER_DEFINED rebalance callback
Posted by kishore g <g....@gmail.com>.
Can you paste the initial IS that you compute. I am mainly interested in
the simple fields.
On Mon, May 2, 2016 at 12:38 PM, Neutron sharc <ne...@gmail.com>
wrote:
> Thanks Kishore for your reply!
>
> What I see is that, a resource's ideal state becomes empty after
> external view converges with the assignment.
>
> When I create a resource I compute an initial IS, and attach a
> USER_DEFINED rebalancer. After Helix stabilizes a resource, its
> "IDEALSTATES" mapFields is wiped off in zookeeper. When the next
> round of rebalancing starts, computeResourceMapping() will always get
> an empty idealState.
>
>
>
>
> On Fri, Apr 29, 2016 at 1:16 PM, kishore g <g....@gmail.com> wrote:
> > Hi,
> >
> > Current state will not show dead replicas. You need to use previous
> > idealstate to derive that info. The logic will be something like this
> >
> > computeResource(.....) {
> > List<Instances> instances =
> > previousIdealState.getInstancesForPartition(P0)
> > foreach instance
> > if(!liveInstances.contain(instance)){
> > //NEED TO ASSIGN ANOTHER INSTANCE TO FOR THIS PARTITION
> > }
> > }
> >
> > This allows your logic to be idempotent and not depend on incremental
> > changes.
> >
> > thanks,
> > Kishore G
> >
> > On Thu, Apr 28, 2016 at 4:27 PM, Neutron sharc <ne...@gmail.com>
> > wrote:
> >
> >> Hi team,
> >>
> >> in USER_DEFINED rebalance mode, the callback computeResourceMapping()
> >> accepts a “currentState”. Does this variable include replicas on a
> >> dead participant ?
> >>
> >> For example, my resource has a partition P1 master replica on
> >> participant node1, a slave replica on participant node2. When node1
> >> dies, in callback computeResourceMapping() I retrieve P1’s replicas:
> >>
> >> Map<ParticipantId, State> replicas =
> >> currentState.getCurrentStateMap(resourceId, partitionId);
> >>
> >>
> >> Here the “replicas” includes only node2, there is no entry for node1.
> >>
> >> However, I want to know all replicas including dead ones, so that I
> >> can know that a master replica is gone and I should failover to an
> >> existing slave, instead of starting a new master.
> >>
> >>
> >> Appreciate any comments!
> >>
> >>
> >> -Neutron
> >>
>