You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@helix.apache.org by ShaoFeng Shi <sh...@apache.org> on 2016/02/04 02:51:06 UTC
Mapping from a resource partition to a logical data partition
Hello,
I'm trying to use Helix (0.7.1) to manage our resource partitions on a
cluster. My scenario is, every 5 minutes a partition will be added; What I
do now is, get the ideal state, +1 for the partition number and then update
it with ZKHelixAdmin. With a rebalance action, the new partition will be
created and assigned to an instance. What the instance can get from Helix
is the resource name and partition id. To map the parition to my logical
data, I maintained a mapping table in our datastore, which looks like:
{
"resource_0": 201601010000_ 201601010005,
"resource_1": 201601010005_ 201601010010,
...
"resource_11": 201601010055_ 201601010100
}
Here it has 12 partitions; Now I want to discard some old partitions, say
the first 6 partitions; It seems in Helix the partitions must start from
0, so with an update on the IdealState, set # of partitions to 6, the new
partitions on the cluster would become to:
resource_0,resource_1, ..., resource_5,
To make sure the partitions wouldn't be wrongly mapped, I need update my
mapping table before the rebalance. While that may not ensure the atomic
between the two updates.
So my question is, what's the suggested way to do the resource partition
mapping? does Helix allow user to specify additional information on a
partition (if have this, I don't need maintain the mapping outside)? Can we
have some simple APIs like addPartition(String parititionName),
dropParitition(String partitionName), just like that for resource? The
numeric paritition id can be an internal ID and not exposed to user.
I guess many entry users will have such questions. Just raise this for a
broad discussion; Any comment and suggestion is welcomed. Thanks for your
time!
--
Best regards,
Shaofeng Shi
Re: Mapping from a resource partition to a logical data partition
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Kishore, I updated my code, will run it over the night (my side) to see
how to runs; Thanks again for your help!
2016-02-05 21:09 GMT+08:00 kishore g <g....@gmail.com>:
> oh, looks like you are not removing the entries from Idealstate.
>
> You can do this.
>
> Idealstate is = helixadmin.getResourceIdealState(clutername, resourcename).
>
> //update the idealstate
> List<String> partitionsToDrop = ....
> for(String partition:partitionsToDrop){
> is.getrecord().getListFields().remove(partition)
> is.getrecord().getMapFields().remove(partition)
> }
>
> is.setNumPartitions(is.getRecord().getMapFields().size());
>
> can you try this.
>
> thanks
> Kishore G
>
>
>
> On Fri, Feb 5, 2016 at 4:52 AM, ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> I see there is a JIRA HELIX-132
>> <https://issues.apache.org/jira/browse/HELIX-132> which has fixed that
>> issue in 0.7.1; but from my case, after drop the resource and add back, the
>> external view still has the old partitions (got from helix-admin.sh),
>> interesting...
>>
>> 2016-02-05 10:49 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>
>>> I didn't; But after add the transitions between Dropped and Offline, it
>>> got the same behavior.
>>> I also changed my code to drop the resource and create again when its
>>> partition number reaches a limit; But the external view wasn't deleted,
>>> that makes Helix thought all the partitions have already been propery
>>> assigned; Following is a state after the resoruce be recreated, you can see
>>> although its "NUM_PARTITIONS" is 8, the partitions in external view is 20
>>> (which is the up limit):
>>>
>>> IdealState for Resource_Stream:
>>> {
>>> "id" : "Resource_Stream",
>>> "mapFields" : {
>>> "Resource_Stream_0" : {
>>> },
>>> "Resource_Stream_1" : {
>>> },
>>> "Resource_Stream_10" : {
>>> },
>>> "Resource_Stream_11" : {
>>> },
>>> "Resource_Stream_12" : {
>>> },
>>> "Resource_Stream_13" : {
>>> },
>>> "Resource_Stream_14" : {
>>> },
>>> "Resource_Stream_15" : {
>>> },
>>> "Resource_Stream_16" : {
>>> },
>>> "Resource_Stream_17" : {
>>> },
>>> "Resource_Stream_18" : {
>>> },
>>> "Resource_Stream_19" : {
>>> },
>>> "Resource_Stream_2" : {
>>> },
>>> "Resource_Stream_3" : {
>>> },
>>> "Resource_Stream_4" : {
>>> },
>>> "Resource_Stream_5" : {
>>> },
>>> "Resource_Stream_6" : {
>>> },
>>> "Resource_Stream_7" : {
>>> },
>>> "Resource_Stream_8" : {
>>> },
>>> "Resource_Stream_9" : {
>>> }
>>> },
>>> "listFields" : {
>>> "Resource_Stream_0" : [ ],
>>> "Resource_Stream_1" : [ ],
>>> "Resource_Stream_10" : [ ],
>>> "Resource_Stream_11" : [ ],
>>> "Resource_Stream_12" : [ ],
>>> "Resource_Stream_13" : [ ],
>>> "Resource_Stream_14" : [ ],
>>> "Resource_Stream_15" : [ ],
>>> "Resource_Stream_16" : [ ],
>>> "Resource_Stream_17" : [ ],
>>> "Resource_Stream_18" : [ ],
>>> "Resource_Stream_19" : [ ],
>>> "Resource_Stream_2" : [ ],
>>> "Resource_Stream_3" : [ ],
>>> "Resource_Stream_4" : [ ],
>>> "Resource_Stream_5" : [ ],
>>> "Resource_Stream_6" : [ ],
>>> "Resource_Stream_7" : [ ],
>>> "Resource_Stream_8" : [ ],
>>> "Resource_Stream_9" : [ ]
>>> },
>>> "simpleFields" : {
>>> "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
>>> "INSTANCE_GROUP_TAG" : "Tag_StreamBuilder",
>>> "NUM_PARTITIONS" : "8",
>>> "REBALANCE_MODE" : "FULL_AUTO",
>>> "REPLICAS" : "2",
>>> "STATE_MODEL_DEF_REF" : "LeaderStandby",
>>> "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
>>> }
>>> }
>>>
>>> ExternalView for Resource_Stream:
>>> {
>>> "id" : "Resource_Stream",
>>> "mapFields" : {
>>> "Resource_Stream_0" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_1" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_10" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_11" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_12" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_13" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_14" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_15" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_16" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_17" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_18" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_19" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_2" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_3" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_4" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_5" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_6" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_7" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> },
>>> "Resource_Stream_8" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>>> },
>>> "Resource_Stream_9" : {
>>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>>> }
>>> },
>>> "listFields" : {
>>> },
>>> "simpleFields" : {
>>> "BUCKET_SIZE" : "0"
>>> }
>>> }
>>>
>>> 2016-02-04 23:08 GMT+08:00 kishore g <g....@gmail.com>:
>>>
>>>> Ha ave you defined offline to dropped state transition in your state
>>>> transition handler.
>>>> On Feb 3, 2016 11:31 PM, "ShaoFeng Shi" <sh...@apache.org> wrote:
>>>>
>>>>> I found an issue: the numberOfPartitions can only be increased, and
>>>>> couldn't be decreased; Even if I set it to 0, and then do a rebalance, the
>>>>> old partitions still exist in idea view and external view; So, how can
>>>>> descrease or drop a partition?
>>>>>
>>>>>
>>>>>
>>>>> 2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>>>>
>>>>>> Nice, that you Kishore!
>>>>>>
>>>>>> 2016-02-04 10:35 GMT+08:00 kishore g <g....@gmail.com>:
>>>>>>
>>>>>>> This is needed for another project I am working on. There is no
>>>>>>> reason for Helix to depend on this convention. I will fix this.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <shaofengshi@apache.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I'm trying to use Helix (0.7.1) to manage our resource partitions
>>>>>>>> on a
>>>>>>>> cluster. My scenario is, every 5 minutes a partition will be added;
>>>>>>>> What I
>>>>>>>> do now is, get the ideal state, +1 for the partition number and
>>>>>>>> then update
>>>>>>>> it with ZKHelixAdmin. With a rebalance action, the new partition
>>>>>>>> will be
>>>>>>>> created and assigned to an instance. What the instance can get from
>>>>>>>> Helix
>>>>>>>> is the resource name and partition id. To map the parition to my
>>>>>>>> logical
>>>>>>>> data, I maintained a mapping table in our datastore, which looks
>>>>>>>> like:
>>>>>>>>
>>>>>>>> {
>>>>>>>> "resource_0": 201601010000_ 201601010005,
>>>>>>>> "resource_1": 201601010005_ 201601010010,
>>>>>>>> ...
>>>>>>>> "resource_11": 201601010055_ 201601010100
>>>>>>>> }
>>>>>>>>
>>>>>>>> Here it has 12 partitions; Now I want to discard some old
>>>>>>>> partitions, say
>>>>>>>> the first 6 partitions; It seems in Helix the partitions must
>>>>>>>> start from
>>>>>>>> 0, so with an update on the IdealState, set # of partitions to 6,
>>>>>>>> the new
>>>>>>>> partitions on the cluster would become to:
>>>>>>>>
>>>>>>>> resource_0,resource_1, ..., resource_5,
>>>>>>>>
>>>>>>>> To make sure the partitions wouldn't be wrongly mapped, I need
>>>>>>>> update my
>>>>>>>> mapping table before the rebalance. While that may not ensure the
>>>>>>>> atomic
>>>>>>>> between the two updates.
>>>>>>>>
>>>>>>>> So my question is, what's the suggested way to do the resource
>>>>>>>> partition
>>>>>>>> mapping? does Helix allow user to specify additional information on
>>>>>>>> a
>>>>>>>> partition (if have this, I don't need maintain the mapping
>>>>>>>> outside)? Can we
>>>>>>>> have some simple APIs like addPartition(String parititionName),
>>>>>>>> dropParitition(String partitionName), just like that for resource?
>>>>>>>> The
>>>>>>>> numeric paritition id can be an internal ID and not exposed to user.
>>>>>>>>
>>>>>>>> I guess many entry users will have such questions. Just raise this
>>>>>>>> for a
>>>>>>>> broad discussion; Any comment and suggestion is welcomed. Thanks
>>>>>>>> for your
>>>>>>>> time!
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Shaofeng Shi
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>>
>>>>>> Shaofeng Shi
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>>
>
--
Best regards,
Shaofeng Shi
Re: Mapping from a resource partition to a logical data partition
Posted by kishore g <g....@gmail.com>.
oh, looks like you are not removing the entries from Idealstate.
You can do this.
Idealstate is = helixadmin.getResourceIdealState(clutername, resourcename).
//update the idealstate
List<String> partitionsToDrop = ....
for(String partition:partitionsToDrop){
is.getrecord().getListFields().remove(partition)
is.getrecord().getMapFields().remove(partition)
}
is.setNumPartitions(is.getRecord().getMapFields().size());
can you try this.
thanks
Kishore G
On Fri, Feb 5, 2016 at 4:52 AM, ShaoFeng Shi <sh...@apache.org> wrote:
> I see there is a JIRA HELIX-132
> <https://issues.apache.org/jira/browse/HELIX-132> which has fixed that
> issue in 0.7.1; but from my case, after drop the resource and add back, the
> external view still has the old partitions (got from helix-admin.sh),
> interesting...
>
> 2016-02-05 10:49 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
>> I didn't; But after add the transitions between Dropped and Offline, it
>> got the same behavior.
>> I also changed my code to drop the resource and create again when its
>> partition number reaches a limit; But the external view wasn't deleted,
>> that makes Helix thought all the partitions have already been propery
>> assigned; Following is a state after the resoruce be recreated, you can see
>> although its "NUM_PARTITIONS" is 8, the partitions in external view is 20
>> (which is the up limit):
>>
>> IdealState for Resource_Stream:
>> {
>> "id" : "Resource_Stream",
>> "mapFields" : {
>> "Resource_Stream_0" : {
>> },
>> "Resource_Stream_1" : {
>> },
>> "Resource_Stream_10" : {
>> },
>> "Resource_Stream_11" : {
>> },
>> "Resource_Stream_12" : {
>> },
>> "Resource_Stream_13" : {
>> },
>> "Resource_Stream_14" : {
>> },
>> "Resource_Stream_15" : {
>> },
>> "Resource_Stream_16" : {
>> },
>> "Resource_Stream_17" : {
>> },
>> "Resource_Stream_18" : {
>> },
>> "Resource_Stream_19" : {
>> },
>> "Resource_Stream_2" : {
>> },
>> "Resource_Stream_3" : {
>> },
>> "Resource_Stream_4" : {
>> },
>> "Resource_Stream_5" : {
>> },
>> "Resource_Stream_6" : {
>> },
>> "Resource_Stream_7" : {
>> },
>> "Resource_Stream_8" : {
>> },
>> "Resource_Stream_9" : {
>> }
>> },
>> "listFields" : {
>> "Resource_Stream_0" : [ ],
>> "Resource_Stream_1" : [ ],
>> "Resource_Stream_10" : [ ],
>> "Resource_Stream_11" : [ ],
>> "Resource_Stream_12" : [ ],
>> "Resource_Stream_13" : [ ],
>> "Resource_Stream_14" : [ ],
>> "Resource_Stream_15" : [ ],
>> "Resource_Stream_16" : [ ],
>> "Resource_Stream_17" : [ ],
>> "Resource_Stream_18" : [ ],
>> "Resource_Stream_19" : [ ],
>> "Resource_Stream_2" : [ ],
>> "Resource_Stream_3" : [ ],
>> "Resource_Stream_4" : [ ],
>> "Resource_Stream_5" : [ ],
>> "Resource_Stream_6" : [ ],
>> "Resource_Stream_7" : [ ],
>> "Resource_Stream_8" : [ ],
>> "Resource_Stream_9" : [ ]
>> },
>> "simpleFields" : {
>> "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
>> "INSTANCE_GROUP_TAG" : "Tag_StreamBuilder",
>> "NUM_PARTITIONS" : "8",
>> "REBALANCE_MODE" : "FULL_AUTO",
>> "REPLICAS" : "2",
>> "STATE_MODEL_DEF_REF" : "LeaderStandby",
>> "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
>> }
>> }
>>
>> ExternalView for Resource_Stream:
>> {
>> "id" : "Resource_Stream",
>> "mapFields" : {
>> "Resource_Stream_0" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_1" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_10" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_11" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_12" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_13" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_14" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_15" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_16" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_17" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_18" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_19" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_2" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_3" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_4" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_5" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_6" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_7" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> },
>> "Resource_Stream_8" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
>> },
>> "Resource_Stream_9" : {
>> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
>> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
>> }
>> },
>> "listFields" : {
>> },
>> "simpleFields" : {
>> "BUCKET_SIZE" : "0"
>> }
>> }
>>
>> 2016-02-04 23:08 GMT+08:00 kishore g <g....@gmail.com>:
>>
>>> Ha ave you defined offline to dropped state transition in your state
>>> transition handler.
>>> On Feb 3, 2016 11:31 PM, "ShaoFeng Shi" <sh...@apache.org> wrote:
>>>
>>>> I found an issue: the numberOfPartitions can only be increased, and
>>>> couldn't be decreased; Even if I set it to 0, and then do a rebalance, the
>>>> old partitions still exist in idea view and external view; So, how can
>>>> descrease or drop a partition?
>>>>
>>>>
>>>>
>>>> 2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>>>
>>>>> Nice, that you Kishore!
>>>>>
>>>>> 2016-02-04 10:35 GMT+08:00 kishore g <g....@gmail.com>:
>>>>>
>>>>>> This is needed for another project I am working on. There is no
>>>>>> reason for Helix to depend on this convention. I will fix this.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <sh...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm trying to use Helix (0.7.1) to manage our resource partitions on
>>>>>>> a
>>>>>>> cluster. My scenario is, every 5 minutes a partition will be added;
>>>>>>> What I
>>>>>>> do now is, get the ideal state, +1 for the partition number and then
>>>>>>> update
>>>>>>> it with ZKHelixAdmin. With a rebalance action, the new partition
>>>>>>> will be
>>>>>>> created and assigned to an instance. What the instance can get from
>>>>>>> Helix
>>>>>>> is the resource name and partition id. To map the parition to my
>>>>>>> logical
>>>>>>> data, I maintained a mapping table in our datastore, which looks
>>>>>>> like:
>>>>>>>
>>>>>>> {
>>>>>>> "resource_0": 201601010000_ 201601010005,
>>>>>>> "resource_1": 201601010005_ 201601010010,
>>>>>>> ...
>>>>>>> "resource_11": 201601010055_ 201601010100
>>>>>>> }
>>>>>>>
>>>>>>> Here it has 12 partitions; Now I want to discard some old
>>>>>>> partitions, say
>>>>>>> the first 6 partitions; It seems in Helix the partitions must start
>>>>>>> from
>>>>>>> 0, so with an update on the IdealState, set # of partitions to 6,
>>>>>>> the new
>>>>>>> partitions on the cluster would become to:
>>>>>>>
>>>>>>> resource_0,resource_1, ..., resource_5,
>>>>>>>
>>>>>>> To make sure the partitions wouldn't be wrongly mapped, I need
>>>>>>> update my
>>>>>>> mapping table before the rebalance. While that may not ensure the
>>>>>>> atomic
>>>>>>> between the two updates.
>>>>>>>
>>>>>>> So my question is, what's the suggested way to do the resource
>>>>>>> partition
>>>>>>> mapping? does Helix allow user to specify additional information on a
>>>>>>> partition (if have this, I don't need maintain the mapping outside)?
>>>>>>> Can we
>>>>>>> have some simple APIs like addPartition(String parititionName),
>>>>>>> dropParitition(String partitionName), just like that for resource?
>>>>>>> The
>>>>>>> numeric paritition id can be an internal ID and not exposed to user.
>>>>>>>
>>>>>>> I guess many entry users will have such questions. Just raise this
>>>>>>> for a
>>>>>>> broad discussion; Any comment and suggestion is welcomed. Thanks for
>>>>>>> your
>>>>>>> time!
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Shaofeng Shi
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi
>>>>
>>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>
Re: Mapping from a resource partition to a logical data partition
Posted by ShaoFeng Shi <sh...@apache.org>.
I see there is a JIRA HELIX-132
<https://issues.apache.org/jira/browse/HELIX-132> which has fixed that
issue in 0.7.1; but from my case, after drop the resource and add back, the
external view still has the old partitions (got from helix-admin.sh),
interesting...
2016-02-05 10:49 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> I didn't; But after add the transitions between Dropped and Offline, it
> got the same behavior.
> I also changed my code to drop the resource and create again when its
> partition number reaches a limit; But the external view wasn't deleted,
> that makes Helix thought all the partitions have already been propery
> assigned; Following is a state after the resoruce be recreated, you can see
> although its "NUM_PARTITIONS" is 8, the partitions in external view is 20
> (which is the up limit):
>
> IdealState for Resource_Stream:
> {
> "id" : "Resource_Stream",
> "mapFields" : {
> "Resource_Stream_0" : {
> },
> "Resource_Stream_1" : {
> },
> "Resource_Stream_10" : {
> },
> "Resource_Stream_11" : {
> },
> "Resource_Stream_12" : {
> },
> "Resource_Stream_13" : {
> },
> "Resource_Stream_14" : {
> },
> "Resource_Stream_15" : {
> },
> "Resource_Stream_16" : {
> },
> "Resource_Stream_17" : {
> },
> "Resource_Stream_18" : {
> },
> "Resource_Stream_19" : {
> },
> "Resource_Stream_2" : {
> },
> "Resource_Stream_3" : {
> },
> "Resource_Stream_4" : {
> },
> "Resource_Stream_5" : {
> },
> "Resource_Stream_6" : {
> },
> "Resource_Stream_7" : {
> },
> "Resource_Stream_8" : {
> },
> "Resource_Stream_9" : {
> }
> },
> "listFields" : {
> "Resource_Stream_0" : [ ],
> "Resource_Stream_1" : [ ],
> "Resource_Stream_10" : [ ],
> "Resource_Stream_11" : [ ],
> "Resource_Stream_12" : [ ],
> "Resource_Stream_13" : [ ],
> "Resource_Stream_14" : [ ],
> "Resource_Stream_15" : [ ],
> "Resource_Stream_16" : [ ],
> "Resource_Stream_17" : [ ],
> "Resource_Stream_18" : [ ],
> "Resource_Stream_19" : [ ],
> "Resource_Stream_2" : [ ],
> "Resource_Stream_3" : [ ],
> "Resource_Stream_4" : [ ],
> "Resource_Stream_5" : [ ],
> "Resource_Stream_6" : [ ],
> "Resource_Stream_7" : [ ],
> "Resource_Stream_8" : [ ],
> "Resource_Stream_9" : [ ]
> },
> "simpleFields" : {
> "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
> "INSTANCE_GROUP_TAG" : "Tag_StreamBuilder",
> "NUM_PARTITIONS" : "8",
> "REBALANCE_MODE" : "FULL_AUTO",
> "REPLICAS" : "2",
> "STATE_MODEL_DEF_REF" : "LeaderStandby",
> "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
> }
> }
>
> ExternalView for Resource_Stream:
> {
> "id" : "Resource_Stream",
> "mapFields" : {
> "Resource_Stream_0" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_1" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_10" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_11" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_12" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_13" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_14" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_15" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_16" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_17" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_18" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_19" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_2" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_3" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_4" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_5" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_6" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_7" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> },
> "Resource_Stream_8" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
> },
> "Resource_Stream_9" : {
> "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
> "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
> }
> },
> "listFields" : {
> },
> "simpleFields" : {
> "BUCKET_SIZE" : "0"
> }
> }
>
> 2016-02-04 23:08 GMT+08:00 kishore g <g....@gmail.com>:
>
>> Ha ave you defined offline to dropped state transition in your state
>> transition handler.
>> On Feb 3, 2016 11:31 PM, "ShaoFeng Shi" <sh...@apache.org> wrote:
>>
>>> I found an issue: the numberOfPartitions can only be increased, and
>>> couldn't be decreased; Even if I set it to 0, and then do a rebalance, the
>>> old partitions still exist in idea view and external view; So, how can
>>> descrease or drop a partition?
>>>
>>>
>>>
>>> 2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>>
>>>> Nice, that you Kishore!
>>>>
>>>> 2016-02-04 10:35 GMT+08:00 kishore g <g....@gmail.com>:
>>>>
>>>>> This is needed for another project I am working on. There is no reason
>>>>> for Helix to depend on this convention. I will fix this.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <sh...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to use Helix (0.7.1) to manage our resource partitions on a
>>>>>> cluster. My scenario is, every 5 minutes a partition will be added;
>>>>>> What I
>>>>>> do now is, get the ideal state, +1 for the partition number and then
>>>>>> update
>>>>>> it with ZKHelixAdmin. With a rebalance action, the new partition will
>>>>>> be
>>>>>> created and assigned to an instance. What the instance can get from
>>>>>> Helix
>>>>>> is the resource name and partition id. To map the parition to my
>>>>>> logical
>>>>>> data, I maintained a mapping table in our datastore, which looks like:
>>>>>>
>>>>>> {
>>>>>> "resource_0": 201601010000_ 201601010005,
>>>>>> "resource_1": 201601010005_ 201601010010,
>>>>>> ...
>>>>>> "resource_11": 201601010055_ 201601010100
>>>>>> }
>>>>>>
>>>>>> Here it has 12 partitions; Now I want to discard some old partitions,
>>>>>> say
>>>>>> the first 6 partitions; It seems in Helix the partitions must start
>>>>>> from
>>>>>> 0, so with an update on the IdealState, set # of partitions to 6, the
>>>>>> new
>>>>>> partitions on the cluster would become to:
>>>>>>
>>>>>> resource_0,resource_1, ..., resource_5,
>>>>>>
>>>>>> To make sure the partitions wouldn't be wrongly mapped, I need update
>>>>>> my
>>>>>> mapping table before the rebalance. While that may not ensure the
>>>>>> atomic
>>>>>> between the two updates.
>>>>>>
>>>>>> So my question is, what's the suggested way to do the resource
>>>>>> partition
>>>>>> mapping? does Helix allow user to specify additional information on a
>>>>>> partition (if have this, I don't need maintain the mapping outside)?
>>>>>> Can we
>>>>>> have some simple APIs like addPartition(String parititionName),
>>>>>> dropParitition(String partitionName), just like that for resource? The
>>>>>> numeric paritition id can be an internal ID and not exposed to user.
>>>>>>
>>>>>> I guess many entry users will have such questions. Just raise this
>>>>>> for a
>>>>>> broad discussion; Any comment and suggestion is welcomed. Thanks for
>>>>>> your
>>>>>> time!
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>>
>>>>>> Shaofeng Shi
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi
>>>
>>>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>
--
Best regards,
Shaofeng Shi
Re: Mapping from a resource partition to a logical data partition
Posted by ShaoFeng Shi <sh...@apache.org>.
I didn't; But after add the transitions between Dropped and Offline, it got
the same behavior.
I also changed my code to drop the resource and create again when its
partition number reaches a limit; But the external view wasn't deleted,
that makes Helix thought all the partitions have already been propery
assigned; Following is a state after the resoruce be recreated, you can see
although its "NUM_PARTITIONS" is 8, the partitions in external view is 20
(which is the up limit):
IdealState for Resource_Stream:
{
"id" : "Resource_Stream",
"mapFields" : {
"Resource_Stream_0" : {
},
"Resource_Stream_1" : {
},
"Resource_Stream_10" : {
},
"Resource_Stream_11" : {
},
"Resource_Stream_12" : {
},
"Resource_Stream_13" : {
},
"Resource_Stream_14" : {
},
"Resource_Stream_15" : {
},
"Resource_Stream_16" : {
},
"Resource_Stream_17" : {
},
"Resource_Stream_18" : {
},
"Resource_Stream_19" : {
},
"Resource_Stream_2" : {
},
"Resource_Stream_3" : {
},
"Resource_Stream_4" : {
},
"Resource_Stream_5" : {
},
"Resource_Stream_6" : {
},
"Resource_Stream_7" : {
},
"Resource_Stream_8" : {
},
"Resource_Stream_9" : {
}
},
"listFields" : {
"Resource_Stream_0" : [ ],
"Resource_Stream_1" : [ ],
"Resource_Stream_10" : [ ],
"Resource_Stream_11" : [ ],
"Resource_Stream_12" : [ ],
"Resource_Stream_13" : [ ],
"Resource_Stream_14" : [ ],
"Resource_Stream_15" : [ ],
"Resource_Stream_16" : [ ],
"Resource_Stream_17" : [ ],
"Resource_Stream_18" : [ ],
"Resource_Stream_19" : [ ],
"Resource_Stream_2" : [ ],
"Resource_Stream_3" : [ ],
"Resource_Stream_4" : [ ],
"Resource_Stream_5" : [ ],
"Resource_Stream_6" : [ ],
"Resource_Stream_7" : [ ],
"Resource_Stream_8" : [ ],
"Resource_Stream_9" : [ ]
},
"simpleFields" : {
"IDEAL_STATE_MODE" : "AUTO_REBALANCE",
"INSTANCE_GROUP_TAG" : "Tag_StreamBuilder",
"NUM_PARTITIONS" : "8",
"REBALANCE_MODE" : "FULL_AUTO",
"REPLICAS" : "2",
"STATE_MODEL_DEF_REF" : "LeaderStandby",
"STATE_MODEL_FACTORY_NAME" : "DEFAULT"
}
}
ExternalView for Resource_Stream:
{
"id" : "Resource_Stream",
"mapFields" : {
"Resource_Stream_0" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_1" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_10" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_11" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_12" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_13" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_14" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_15" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_16" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_17" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_18" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_19" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_2" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_3" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_4" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_5" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_6" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_7" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
},
"Resource_Stream_8" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
},
"Resource_Stream_9" : {
"kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
"kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
}
},
"listFields" : {
},
"simpleFields" : {
"BUCKET_SIZE" : "0"
}
}
2016-02-04 23:08 GMT+08:00 kishore g <g....@gmail.com>:
> Ha ave you defined offline to dropped state transition in your state
> transition handler.
> On Feb 3, 2016 11:31 PM, "ShaoFeng Shi" <sh...@apache.org> wrote:
>
>> I found an issue: the numberOfPartitions can only be increased, and
>> couldn't be decreased; Even if I set it to 0, and then do a rebalance, the
>> old partitions still exist in idea view and external view; So, how can
>> descrease or drop a partition?
>>
>>
>>
>> 2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>
>>> Nice, that you Kishore!
>>>
>>> 2016-02-04 10:35 GMT+08:00 kishore g <g....@gmail.com>:
>>>
>>>> This is needed for another project I am working on. There is no reason
>>>> for Helix to depend on this convention. I will fix this.
>>>>
>>>>
>>>>
>>>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <sh...@apache.org>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to use Helix (0.7.1) to manage our resource partitions on a
>>>>> cluster. My scenario is, every 5 minutes a partition will be added;
>>>>> What I
>>>>> do now is, get the ideal state, +1 for the partition number and then
>>>>> update
>>>>> it with ZKHelixAdmin. With a rebalance action, the new partition will
>>>>> be
>>>>> created and assigned to an instance. What the instance can get from
>>>>> Helix
>>>>> is the resource name and partition id. To map the parition to my
>>>>> logical
>>>>> data, I maintained a mapping table in our datastore, which looks like:
>>>>>
>>>>> {
>>>>> "resource_0": 201601010000_ 201601010005,
>>>>> "resource_1": 201601010005_ 201601010010,
>>>>> ...
>>>>> "resource_11": 201601010055_ 201601010100
>>>>> }
>>>>>
>>>>> Here it has 12 partitions; Now I want to discard some old partitions,
>>>>> say
>>>>> the first 6 partitions; It seems in Helix the partitions must start
>>>>> from
>>>>> 0, so with an update on the IdealState, set # of partitions to 6, the
>>>>> new
>>>>> partitions on the cluster would become to:
>>>>>
>>>>> resource_0,resource_1, ..., resource_5,
>>>>>
>>>>> To make sure the partitions wouldn't be wrongly mapped, I need update
>>>>> my
>>>>> mapping table before the rebalance. While that may not ensure the
>>>>> atomic
>>>>> between the two updates.
>>>>>
>>>>> So my question is, what's the suggested way to do the resource
>>>>> partition
>>>>> mapping? does Helix allow user to specify additional information on a
>>>>> partition (if have this, I don't need maintain the mapping outside)?
>>>>> Can we
>>>>> have some simple APIs like addPartition(String parititionName),
>>>>> dropParitition(String partitionName), just like that for resource? The
>>>>> numeric paritition id can be an internal ID and not exposed to user.
>>>>>
>>>>> I guess many entry users will have such questions. Just raise this for
>>>>> a
>>>>> broad discussion; Any comment and suggestion is welcomed. Thanks for
>>>>> your
>>>>> time!
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>>
--
Best regards,
Shaofeng Shi
Re: Mapping from a resource partition to a logical data partition
Posted by kishore g <g....@gmail.com>.
Ha ave you defined offline to dropped state transition in your state
transition handler.
On Feb 3, 2016 11:31 PM, "ShaoFeng Shi" <sh...@apache.org> wrote:
> I found an issue: the numberOfPartitions can only be increased, and
> couldn't be decreased; Even if I set it to 0, and then do a rebalance, the
> old partitions still exist in idea view and external view; So, how can
> descrease or drop a partition?
>
>
>
> 2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
>> Nice, that you Kishore!
>>
>> 2016-02-04 10:35 GMT+08:00 kishore g <g....@gmail.com>:
>>
>>> This is needed for another project I am working on. There is no reason
>>> for Helix to depend on this convention. I will fix this.
>>>
>>>
>>>
>>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <sh...@apache.org>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm trying to use Helix (0.7.1) to manage our resource partitions on a
>>>> cluster. My scenario is, every 5 minutes a partition will be added;
>>>> What I
>>>> do now is, get the ideal state, +1 for the partition number and then
>>>> update
>>>> it with ZKHelixAdmin. With a rebalance action, the new partition will be
>>>> created and assigned to an instance. What the instance can get from
>>>> Helix
>>>> is the resource name and partition id. To map the parition to my logical
>>>> data, I maintained a mapping table in our datastore, which looks like:
>>>>
>>>> {
>>>> "resource_0": 201601010000_ 201601010005,
>>>> "resource_1": 201601010005_ 201601010010,
>>>> ...
>>>> "resource_11": 201601010055_ 201601010100
>>>> }
>>>>
>>>> Here it has 12 partitions; Now I want to discard some old partitions,
>>>> say
>>>> the first 6 partitions; It seems in Helix the partitions must start
>>>> from
>>>> 0, so with an update on the IdealState, set # of partitions to 6, the
>>>> new
>>>> partitions on the cluster would become to:
>>>>
>>>> resource_0,resource_1, ..., resource_5,
>>>>
>>>> To make sure the partitions wouldn't be wrongly mapped, I need update my
>>>> mapping table before the rebalance. While that may not ensure the atomic
>>>> between the two updates.
>>>>
>>>> So my question is, what's the suggested way to do the resource partition
>>>> mapping? does Helix allow user to specify additional information on a
>>>> partition (if have this, I don't need maintain the mapping outside)?
>>>> Can we
>>>> have some simple APIs like addPartition(String parititionName),
>>>> dropParitition(String partitionName), just like that for resource? The
>>>> numeric paritition id can be an internal ID and not exposed to user.
>>>>
>>>> I guess many entry users will have such questions. Just raise this for a
>>>> broad discussion; Any comment and suggestion is welcomed. Thanks for
>>>> your
>>>> time!
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi
>>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>
Re: Mapping from a resource partition to a logical data partition
Posted by ShaoFeng Shi <sh...@apache.org>.
I found an issue: the numberOfPartitions can only be increased, and
couldn't be decreased; Even if I set it to 0, and then do a rebalance, the
old partitions still exist in idea view and external view; So, how can
descrease or drop a partition?
2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> Nice, that you Kishore!
>
> 2016-02-04 10:35 GMT+08:00 kishore g <g....@gmail.com>:
>
>> This is needed for another project I am working on. There is no reason
>> for Helix to depend on this convention. I will fix this.
>>
>>
>>
>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <sh...@apache.org>
>> wrote:
>>
>>> Hello,
>>>
>>> I'm trying to use Helix (0.7.1) to manage our resource partitions on a
>>> cluster. My scenario is, every 5 minutes a partition will be added; What
>>> I
>>> do now is, get the ideal state, +1 for the partition number and then
>>> update
>>> it with ZKHelixAdmin. With a rebalance action, the new partition will be
>>> created and assigned to an instance. What the instance can get from Helix
>>> is the resource name and partition id. To map the parition to my logical
>>> data, I maintained a mapping table in our datastore, which looks like:
>>>
>>> {
>>> "resource_0": 201601010000_ 201601010005,
>>> "resource_1": 201601010005_ 201601010010,
>>> ...
>>> "resource_11": 201601010055_ 201601010100
>>> }
>>>
>>> Here it has 12 partitions; Now I want to discard some old partitions, say
>>> the first 6 partitions; It seems in Helix the partitions must start from
>>> 0, so with an update on the IdealState, set # of partitions to 6, the new
>>> partitions on the cluster would become to:
>>>
>>> resource_0,resource_1, ..., resource_5,
>>>
>>> To make sure the partitions wouldn't be wrongly mapped, I need update my
>>> mapping table before the rebalance. While that may not ensure the atomic
>>> between the two updates.
>>>
>>> So my question is, what's the suggested way to do the resource partition
>>> mapping? does Helix allow user to specify additional information on a
>>> partition (if have this, I don't need maintain the mapping outside)? Can
>>> we
>>> have some simple APIs like addPartition(String parititionName),
>>> dropParitition(String partitionName), just like that for resource? The
>>> numeric paritition id can be an internal ID and not exposed to user.
>>>
>>> I guess many entry users will have such questions. Just raise this for a
>>> broad discussion; Any comment and suggestion is welcomed. Thanks for your
>>> time!
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi
>>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>
--
Best regards,
Shaofeng Shi
Re: Mapping from a resource partition to a logical data partition
Posted by ShaoFeng Shi <sh...@apache.org>.
Nice, that you Kishore!
2016-02-04 10:35 GMT+08:00 kishore g <g....@gmail.com>:
> This is needed for another project I am working on. There is no reason for
> Helix to depend on this convention. I will fix this.
>
>
>
> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> Hello,
>>
>> I'm trying to use Helix (0.7.1) to manage our resource partitions on a
>> cluster. My scenario is, every 5 minutes a partition will be added; What I
>> do now is, get the ideal state, +1 for the partition number and then
>> update
>> it with ZKHelixAdmin. With a rebalance action, the new partition will be
>> created and assigned to an instance. What the instance can get from Helix
>> is the resource name and partition id. To map the parition to my logical
>> data, I maintained a mapping table in our datastore, which looks like:
>>
>> {
>> "resource_0": 201601010000_ 201601010005,
>> "resource_1": 201601010005_ 201601010010,
>> ...
>> "resource_11": 201601010055_ 201601010100
>> }
>>
>> Here it has 12 partitions; Now I want to discard some old partitions, say
>> the first 6 partitions; It seems in Helix the partitions must start from
>> 0, so with an update on the IdealState, set # of partitions to 6, the new
>> partitions on the cluster would become to:
>>
>> resource_0,resource_1, ..., resource_5,
>>
>> To make sure the partitions wouldn't be wrongly mapped, I need update my
>> mapping table before the rebalance. While that may not ensure the atomic
>> between the two updates.
>>
>> So my question is, what's the suggested way to do the resource partition
>> mapping? does Helix allow user to specify additional information on a
>> partition (if have this, I don't need maintain the mapping outside)? Can
>> we
>> have some simple APIs like addPartition(String parititionName),
>> dropParitition(String partitionName), just like that for resource? The
>> numeric paritition id can be an internal ID and not exposed to user.
>>
>> I guess many entry users will have such questions. Just raise this for a
>> broad discussion; Any comment and suggestion is welcomed. Thanks for your
>> time!
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>
>
--
Best regards,
Shaofeng Shi
Re: Mapping from a resource partition to a logical data partition
Posted by kishore g <g....@gmail.com>.
This is needed for another project I am working on. There is no reason for
Helix to depend on this convention. I will fix this.
On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <sh...@apache.org> wrote:
> Hello,
>
> I'm trying to use Helix (0.7.1) to manage our resource partitions on a
> cluster. My scenario is, every 5 minutes a partition will be added; What I
> do now is, get the ideal state, +1 for the partition number and then update
> it with ZKHelixAdmin. With a rebalance action, the new partition will be
> created and assigned to an instance. What the instance can get from Helix
> is the resource name and partition id. To map the parition to my logical
> data, I maintained a mapping table in our datastore, which looks like:
>
> {
> "resource_0": 201601010000_ 201601010005,
> "resource_1": 201601010005_ 201601010010,
> ...
> "resource_11": 201601010055_ 201601010100
> }
>
> Here it has 12 partitions; Now I want to discard some old partitions, say
> the first 6 partitions; It seems in Helix the partitions must start from
> 0, so with an update on the IdealState, set # of partitions to 6, the new
> partitions on the cluster would become to:
>
> resource_0,resource_1, ..., resource_5,
>
> To make sure the partitions wouldn't be wrongly mapped, I need update my
> mapping table before the rebalance. While that may not ensure the atomic
> between the two updates.
>
> So my question is, what's the suggested way to do the resource partition
> mapping? does Helix allow user to specify additional information on a
> partition (if have this, I don't need maintain the mapping outside)? Can we
> have some simple APIs like addPartition(String parititionName),
> dropParitition(String partitionName), just like that for resource? The
> numeric paritition id can be an internal ID and not exposed to user.
>
> I guess many entry users will have such questions. Just raise this for a
> broad discussion; Any comment and suggestion is welcomed. Thanks for your
> time!
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
Re: Mapping from a resource partition to a logical data partition
Posted by kishore g <g....@gmail.com>.
This is needed for another project I am working on. There is no reason for
Helix to depend on this convention. I will fix this.
On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <sh...@apache.org> wrote:
> Hello,
>
> I'm trying to use Helix (0.7.1) to manage our resource partitions on a
> cluster. My scenario is, every 5 minutes a partition will be added; What I
> do now is, get the ideal state, +1 for the partition number and then update
> it with ZKHelixAdmin. With a rebalance action, the new partition will be
> created and assigned to an instance. What the instance can get from Helix
> is the resource name and partition id. To map the parition to my logical
> data, I maintained a mapping table in our datastore, which looks like:
>
> {
> "resource_0": 201601010000_ 201601010005,
> "resource_1": 201601010005_ 201601010010,
> ...
> "resource_11": 201601010055_ 201601010100
> }
>
> Here it has 12 partitions; Now I want to discard some old partitions, say
> the first 6 partitions; It seems in Helix the partitions must start from
> 0, so with an update on the IdealState, set # of partitions to 6, the new
> partitions on the cluster would become to:
>
> resource_0,resource_1, ..., resource_5,
>
> To make sure the partitions wouldn't be wrongly mapped, I need update my
> mapping table before the rebalance. While that may not ensure the atomic
> between the two updates.
>
> So my question is, what's the suggested way to do the resource partition
> mapping? does Helix allow user to specify additional information on a
> partition (if have this, I don't need maintain the mapping outside)? Can we
> have some simple APIs like addPartition(String parititionName),
> dropParitition(String partitionName), just like that for resource? The
> numeric paritition id can be an internal ID and not exposed to user.
>
> I guess many entry users will have such questions. Just raise this for a
> broad discussion; Any comment and suggestion is welcomed. Thanks for your
> time!
>
>
> --
> Best regards,
>
> Shaofeng Shi
>