You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Varun Sharma <va...@pinterest.com> on 2015/02/23 10:51:48 UTC
Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to
do so. I am setting the bucketSize in the idealState and using
addResource() and setResourceIdealState and it ends up complaining that ZK
size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state
with a valid bucket size. However, when I tried to set the ideal state on
that bucket (this ideal state did not have the bucket size set), it failed
as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun
RE: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <zz...@linkedin.com>.
that should be fine. also i think we can have a tool for changing bucket size if you can tolerate disabling the cluster for a short period. basically, we can 1) disable the cluster, 2) change the bucket size, 3) re-bucketize all existing buckets, and 4) re-enable the cluster.
________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Tuesday, February 24, 2015 10:30 AM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4
I see and is it okay to set the bucket size for a resource which previously did not have buckets - would that also lead to bad behaviour (using setResourceIdealState)
Thanks
Varun
On Tue, Feb 24, 2015 at 10:27 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
drop ideal state means drop the resource.
yes. if you set a different bucket size, the old buckets will remains. since the bucket size is changed, helix controller will not be able to read the correct current state of each partition and considers them all be in initial state. This will lead to controller sending incorrect transitions to existing partitions.
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Tuesday, February 24, 2015 12:37 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Do I need to drop the resource in order to drop the ideal state - also wondering what the behaviour is if i set the bucket size using setResourceIdealState - would it just keep the previous bucketing or bad things could happen ?
On Tue, Feb 24, 2015 at 12:20 AM, Zhen Zhang <ne...@gmail.com>> wrote:
To get bucket size, get the ideal state and then getBucketSize() on it. If you want to change the bucket size, you may first drop the ideal state and re-create using the new bucket size.
On Mon, Feb 23, 2015 at 3:10 PM, Varun Sharma <va...@pinterest.com>> wrote:
Also, would changing the bucketSize throw an exception or would it cause undefined behaviour or would the new bucket size simply be ignored ? This is assuming the number of partitions remain the same in the setResourceIdealState call ?
Thanks !
Varun
On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <va...@pinterest.com>> wrote:
Cool - thanks - for a pre existing resource, how can we retrieve the bucket size - do we need to pull in the whole ideal state and then run a getBucketSize() on it ?
On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zz...@linkedin.com>> wrote:
bucktSize is the size of each bucket, i.e. the number of partitions each bucket will hold. bucket size can't be changed dynamically.
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 2:37 PM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
OK - thanks. Btw, is bucket size the number of buckets or is it the size of one bucket ? Also can the bucket size be changed dynamically using setResourceIdealState ?
On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
The partition number is mainly for calculating the buckets. Partition name will remain the same. You can always get and build Helix source code from here:
https://github.com/apache/helix
Changing the partition naming is not hard, but we need to come up with a general way to achieve this. We will think about this requirement. If you have any good idea for this, you can always submit a pull request on github.
Thanks,
Jason
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 12:04 PM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
How hard is this to change to '$' - its difficult for us to now change the partition names now. Is there a place to get the helix 0.6.4 source code, make changes and built it ?
Another related question - is the partition number mainly needed for hashing into the buckets - the partition name would still continue to be "resource_name_partition_num", right ?
On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
The bucketize logic is expecting partition name following the convention {resource_name}_{partition_number}. In your case, you are using $ ?
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 11:48 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Note that we had to prefix each partition with the resource name to make the partitions unique.
On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>> wrote:
Here is the latest error I am getting:
2015-02-23 19:44:48,153 [controller-pool-0] (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for $terrapin$data$pinpin_board_join$1424377884722, rolling back.
java.lang.IllegalArgumentException: Could NOT parse partition# (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0
at org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)
at org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)
at org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
at org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Not really. The bucketize logic just creates child znode under the original ideal state, and each of the child znode holds a subset of the total partitions.
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:37 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
One other question - would bucketizing add anything to the partition name - just worried about any special symbols (we use _ and $) causing issues there ?
On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
HelixAdmin#addResource was initially used for adding an empty resource only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so the bucketize logic is missing from addResource. We should definitely fix this. We are also considering merging addResource and rebalance, since separating them are confusing. For now you may use setResourceIdealState as a workaround.
Thanks,
Zhen
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:20 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Thanks, Zhen.
What about just calling addResource with an idealState with the bucket size - is that expected to not work on its own ?
Varun
On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size
When you set the ideal state, you still need set the bucket size in the ideal state. This value will be used to bucketize the ideal state.
Thanks,
Jason
________________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting the bucketSize in the idealState and using addResource() and setResourceIdealState and it ends up complaining that ZK size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state with a valid bucket size. However, when I tried to set the ideal state on that bucket (this ideal state did not have the bucket size set), it failed as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
I see and is it okay to set the bucket size for a resource which previously
did not have buckets - would that also lead to bad behaviour (using
setResourceIdealState)
Thanks
Varun
On Tue, Feb 24, 2015 at 10:27 AM, Zhen Zhang <zz...@linkedin.com> wrote:
> drop ideal state means drop the resource.
>
> yes. if you set a different bucket size, the old buckets will remains.
> since the bucket size is changed, helix controller will not be able to read
> the correct current state of each partition and considers them all be in
> initial state. This will lead to controller sending incorrect transitions
> to existing partitions.
>
> ------------------------------
> *From:* Varun Sharma [varun@pinterest.com]
> *Sent:* Tuesday, February 24, 2015 12:37 AM
>
> *To:* user@helix.apache.org
> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>
> Do I need to drop the resource in order to drop the ideal state - also
> wondering what the behaviour is if i set the bucket size using
> setResourceIdealState - would it just keep the previous bucketing or bad
> things could happen ?
>
> On Tue, Feb 24, 2015 at 12:20 AM, Zhen Zhang <ne...@gmail.com> wrote:
>
>> To get bucket size, get the ideal state and then getBucketSize() on it.
>> If you want to change the bucket size, you may first drop the ideal state
>> and re-create using the new bucket size.
>>
>> On Mon, Feb 23, 2015 at 3:10 PM, Varun Sharma <va...@pinterest.com>
>> wrote:
>>
>>> Also, would changing the bucketSize throw an exception or would it cause
>>> undefined behaviour or would the new bucket size simply be ignored ? This
>>> is assuming the number of partitions remain the same in the
>>> setResourceIdealState call ?
>>>
>>> Thanks !
>>> Varun
>>>
>>> On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <va...@pinterest.com>
>>> wrote:
>>>
>>>> Cool - thanks - for a pre existing resource, how can we retrieve the
>>>> bucket size - do we need to pull in the whole ideal state and then run a
>>>> getBucketSize() on it ?
>>>>
>>>> On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zz...@linkedin.com>
>>>> wrote:
>>>>
>>>>> bucktSize is the size of each bucket, i.e. the number of partitions
>>>>> each bucket will hold. bucket size can't be changed dynamically.
>>>>>
>>>>> ------------------------------
>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>> *Sent:* Monday, February 23, 2015 2:37 PM
>>>>>
>>>>> *To:* user@helix.apache.org
>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>
>>>>> OK - thanks. Btw, is bucket size the number of buckets or is it
>>>>> the size of one bucket ? Also can the bucket size be changed dynamically
>>>>> using setResourceIdealState ?
>>>>>
>>>>> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Varun,
>>>>>>
>>>>>> The partition number is mainly for calculating the buckets.
>>>>>> Partition name will remain the same. You can always get and build Helix
>>>>>> source code from here:
>>>>>> https://github.com/apache/helix
>>>>>>
>>>>>> Changing the partition naming is not hard, but we need to come up
>>>>>> with a general way to achieve this. We will think about this requirement.
>>>>>> If you have any good idea for this, you can always submit a pull request on
>>>>>> github.
>>>>>>
>>>>>> Thanks,
>>>>>> Jason
>>>>>> ------------------------------
>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>> *Sent:* Monday, February 23, 2015 12:04 PM
>>>>>>
>>>>>> *To:* user@helix.apache.org
>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>
>>>>>> How hard is this to change to '$' - its difficult for us to now
>>>>>> change the partition names now. Is there a place to get the helix 0.6.4
>>>>>> source code, make changes and built it ?
>>>>>>
>>>>>> Another related question - is the partition number mainly needed
>>>>>> for hashing into the buckets - the partition name would still continue to
>>>>>> be "resource_name_partition_num", right ?
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>> wrote:
>>>>>>
>>>>>>> The bucketize logic is expecting partition name following the
>>>>>>> convention {resource_name}_{partition_number}. In your case, you are using
>>>>>>> $ ?
>>>>>>> ------------------------------
>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>> *Sent:* Monday, February 23, 2015 11:48 AM
>>>>>>>
>>>>>>> *To:* user@helix.apache.org
>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>
>>>>>>> Note that we had to prefix each partition with the resource name
>>>>>>> to make the partitions unique.
>>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Here is the latest error I am getting:
>>>>>>>>
>>>>>>>> 2015-02-23 19:44:48,153 [controller-pool-0]
>>>>>>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
>>>>>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>>>>>>>
>>>>>>>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>>>>>>>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>>>>>>>
>>>>>>>> *at
>>>>>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>>>>>>>
>>>>>>>> *at
>>>>>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>>>>>>>
>>>>>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Not really. The bucketize logic just creates child znode under
>>>>>>>>> the original ideal state, and each of the child znode holds a subset of the
>>>>>>>>> total partitions.
>>>>>>>>> ------------------------------
>>>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>>>>>>>
>>>>>>>>> *To:* user@helix.apache.org
>>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>>>
>>>>>>>>> One other question - would bucketizing add anything to the
>>>>>>>>> partition name - just worried about any special symbols (we use _ and $)
>>>>>>>>> causing issues there ?
>>>>>>>>>
>>>>>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Varun,
>>>>>>>>>>
>>>>>>>>>> HelixAdmin#addResource was initially used for adding an empty
>>>>>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal
>>>>>>>>>> state, so the bucketize logic is missing from addResource. We should
>>>>>>>>>> definitely fix this. We are also considering merging addResource and
>>>>>>>>>> rebalance, since separating them are confusing. For now you may use
>>>>>>>>>> setResourceIdealState as a workaround.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Zhen
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>>>>>>>> *To:* user@helix.apache.org
>>>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>>>>
>>>>>>>>>> Thanks, Zhen.
>>>>>>>>>>
>>>>>>>>>> What about just calling addResource with an idealState with the
>>>>>>>>>> bucket size - is that expected to not work on its own ?
>>>>>>>>>>
>>>>>>>>>> Varun
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zzhang@linkedin.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Varun,
>>>>>>>>>>>
>>>>>>>>>>> Did you do the following?
>>>>>>>>>>> HelixAdmin#addResource() // set bucket size
>>>>>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>>>>>>>
>>>>>>>>>>> When you set the ideal state, you still need set the bucket size
>>>>>>>>>>> in the ideal state. This value will be used to bucketize the ideal state.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Jason
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>>>>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>>>>>>>> To: user@helix.apache.org
>>>>>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am
>>>>>>>>>>> unable to do so. I am setting the bucketSize in the idealState and using
>>>>>>>>>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>>>>>>>>>> size of 1M has been exceeded.
>>>>>>>>>>>
>>>>>>>>>>> I also tried using the addResource() api which creates an empty
>>>>>>>>>>> ideal state with a valid bucket size. However, when I tried to set the
>>>>>>>>>>> ideal state on that bucket (this ideal state did not have the bucket size
>>>>>>>>>>> set), it failed as well.
>>>>>>>>>>>
>>>>>>>>>>> Whats the right way to get bucketizing to work ?
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>> Varun
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
RE: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <zz...@linkedin.com>.
drop ideal state means drop the resource.
yes. if you set a different bucket size, the old buckets will remains. since the bucket size is changed, helix controller will not be able to read the correct current state of each partition and considers them all be in initial state. This will lead to controller sending incorrect transitions to existing partitions.
________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Tuesday, February 24, 2015 12:37 AM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4
Do I need to drop the resource in order to drop the ideal state - also wondering what the behaviour is if i set the bucket size using setResourceIdealState - would it just keep the previous bucketing or bad things could happen ?
On Tue, Feb 24, 2015 at 12:20 AM, Zhen Zhang <ne...@gmail.com>> wrote:
To get bucket size, get the ideal state and then getBucketSize() on it. If you want to change the bucket size, you may first drop the ideal state and re-create using the new bucket size.
On Mon, Feb 23, 2015 at 3:10 PM, Varun Sharma <va...@pinterest.com>> wrote:
Also, would changing the bucketSize throw an exception or would it cause undefined behaviour or would the new bucket size simply be ignored ? This is assuming the number of partitions remain the same in the setResourceIdealState call ?
Thanks !
Varun
On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <va...@pinterest.com>> wrote:
Cool - thanks - for a pre existing resource, how can we retrieve the bucket size - do we need to pull in the whole ideal state and then run a getBucketSize() on it ?
On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zz...@linkedin.com>> wrote:
bucktSize is the size of each bucket, i.e. the number of partitions each bucket will hold. bucket size can't be changed dynamically.
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 2:37 PM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
OK - thanks. Btw, is bucket size the number of buckets or is it the size of one bucket ? Also can the bucket size be changed dynamically using setResourceIdealState ?
On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
The partition number is mainly for calculating the buckets. Partition name will remain the same. You can always get and build Helix source code from here:
https://github.com/apache/helix
Changing the partition naming is not hard, but we need to come up with a general way to achieve this. We will think about this requirement. If you have any good idea for this, you can always submit a pull request on github.
Thanks,
Jason
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 12:04 PM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
How hard is this to change to '$' - its difficult for us to now change the partition names now. Is there a place to get the helix 0.6.4 source code, make changes and built it ?
Another related question - is the partition number mainly needed for hashing into the buckets - the partition name would still continue to be "resource_name_partition_num", right ?
On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
The bucketize logic is expecting partition name following the convention {resource_name}_{partition_number}. In your case, you are using $ ?
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 11:48 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Note that we had to prefix each partition with the resource name to make the partitions unique.
On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>> wrote:
Here is the latest error I am getting:
2015-02-23 19:44:48,153 [controller-pool-0] (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for $terrapin$data$pinpin_board_join$1424377884722, rolling back.
java.lang.IllegalArgumentException: Could NOT parse partition# (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0
at org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)
at org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)
at org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
at org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Not really. The bucketize logic just creates child znode under the original ideal state, and each of the child znode holds a subset of the total partitions.
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:37 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
One other question - would bucketizing add anything to the partition name - just worried about any special symbols (we use _ and $) causing issues there ?
On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
HelixAdmin#addResource was initially used for adding an empty resource only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so the bucketize logic is missing from addResource. We should definitely fix this. We are also considering merging addResource and rebalance, since separating them are confusing. For now you may use setResourceIdealState as a workaround.
Thanks,
Zhen
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:20 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Thanks, Zhen.
What about just calling addResource with an idealState with the bucket size - is that expected to not work on its own ?
Varun
On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size
When you set the ideal state, you still need set the bucket size in the ideal state. This value will be used to bucketize the ideal state.
Thanks,
Jason
________________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting the bucketSize in the idealState and using addResource() and setResourceIdealState and it ends up complaining that ZK size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state with a valid bucket size. However, when I tried to set the ideal state on that bucket (this ideal state did not have the bucket size set), it failed as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
Do I need to drop the resource in order to drop the ideal state - also
wondering what the behaviour is if i set the bucket size using
setResourceIdealState - would it just keep the previous bucketing or bad
things could happen ?
On Tue, Feb 24, 2015 at 12:20 AM, Zhen Zhang <ne...@gmail.com> wrote:
> To get bucket size, get the ideal state and then getBucketSize() on it. If
> you want to change the bucket size, you may first drop the ideal state and
> re-create using the new bucket size.
>
> On Mon, Feb 23, 2015 at 3:10 PM, Varun Sharma <va...@pinterest.com> wrote:
>
>> Also, would changing the bucketSize throw an exception or would it cause
>> undefined behaviour or would the new bucket size simply be ignored ? This
>> is assuming the number of partitions remain the same in the
>> setResourceIdealState call ?
>>
>> Thanks !
>> Varun
>>
>> On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <va...@pinterest.com>
>> wrote:
>>
>>> Cool - thanks - for a pre existing resource, how can we retrieve the
>>> bucket size - do we need to pull in the whole ideal state and then run a
>>> getBucketSize() on it ?
>>>
>>> On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zz...@linkedin.com> wrote:
>>>
>>>> bucktSize is the size of each bucket, i.e. the number of partitions
>>>> each bucket will hold. bucket size can't be changed dynamically.
>>>>
>>>> ------------------------------
>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>> *Sent:* Monday, February 23, 2015 2:37 PM
>>>>
>>>> *To:* user@helix.apache.org
>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>
>>>> OK - thanks. Btw, is bucket size the number of buckets or is it the
>>>> size of one bucket ? Also can the bucket size be changed dynamically using
>>>> setResourceIdealState ?
>>>>
>>>> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com>
>>>> wrote:
>>>>
>>>>> Hi Varun,
>>>>>
>>>>> The partition number is mainly for calculating the buckets.
>>>>> Partition name will remain the same. You can always get and build Helix
>>>>> source code from here:
>>>>> https://github.com/apache/helix
>>>>>
>>>>> Changing the partition naming is not hard, but we need to come up
>>>>> with a general way to achieve this. We will think about this requirement.
>>>>> If you have any good idea for this, you can always submit a pull request on
>>>>> github.
>>>>>
>>>>> Thanks,
>>>>> Jason
>>>>> ------------------------------
>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>> *Sent:* Monday, February 23, 2015 12:04 PM
>>>>>
>>>>> *To:* user@helix.apache.org
>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>
>>>>> How hard is this to change to '$' - its difficult for us to now
>>>>> change the partition names now. Is there a place to get the helix 0.6.4
>>>>> source code, make changes and built it ?
>>>>>
>>>>> Another related question - is the partition number mainly needed for
>>>>> hashing into the buckets - the partition name would still continue to be
>>>>> "resource_name_partition_num", right ?
>>>>>
>>>>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com>
>>>>> wrote:
>>>>>
>>>>>> The bucketize logic is expecting partition name following the
>>>>>> convention {resource_name}_{partition_number}. In your case, you are using
>>>>>> $ ?
>>>>>> ------------------------------
>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>> *Sent:* Monday, February 23, 2015 11:48 AM
>>>>>>
>>>>>> *To:* user@helix.apache.org
>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>
>>>>>> Note that we had to prefix each partition with the resource name
>>>>>> to make the partitions unique.
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Here is the latest error I am getting:
>>>>>>>
>>>>>>> 2015-02-23 19:44:48,153 [controller-pool-0]
>>>>>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
>>>>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>>>>>>
>>>>>>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>>>>>>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>>>>>>
>>>>>>> *at
>>>>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>>>>>>
>>>>>>> *at
>>>>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Not really. The bucketize logic just creates child znode under
>>>>>>>> the original ideal state, and each of the child znode holds a subset of the
>>>>>>>> total partitions.
>>>>>>>> ------------------------------
>>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>>>>>>
>>>>>>>> *To:* user@helix.apache.org
>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>>
>>>>>>>> One other question - would bucketizing add anything to the
>>>>>>>> partition name - just worried about any special symbols (we use _ and $)
>>>>>>>> causing issues there ?
>>>>>>>>
>>>>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Varun,
>>>>>>>>>
>>>>>>>>> HelixAdmin#addResource was initially used for adding an empty
>>>>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal
>>>>>>>>> state, so the bucketize logic is missing from addResource. We should
>>>>>>>>> definitely fix this. We are also considering merging addResource and
>>>>>>>>> rebalance, since separating them are confusing. For now you may use
>>>>>>>>> setResourceIdealState as a workaround.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Zhen
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>>>>>>> *To:* user@helix.apache.org
>>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>>>
>>>>>>>>> Thanks, Zhen.
>>>>>>>>>
>>>>>>>>> What about just calling addResource with an idealState with the
>>>>>>>>> bucket size - is that expected to not work on its own ?
>>>>>>>>>
>>>>>>>>> Varun
>>>>>>>>>
>>>>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Varun,
>>>>>>>>>>
>>>>>>>>>> Did you do the following?
>>>>>>>>>> HelixAdmin#addResource() // set bucket size
>>>>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>>>>>>
>>>>>>>>>> When you set the ideal state, you still need set the bucket size
>>>>>>>>>> in the ideal state. This value will be used to bucketize the ideal state.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Jason
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>>>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>>>>>>> To: user@helix.apache.org
>>>>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am
>>>>>>>>>> unable to do so. I am setting the bucketSize in the idealState and using
>>>>>>>>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>>>>>>>>> size of 1M has been exceeded.
>>>>>>>>>>
>>>>>>>>>> I also tried using the addResource() api which creates an empty
>>>>>>>>>> ideal state with a valid bucket size. However, when I tried to set the
>>>>>>>>>> ideal state on that bucket (this ideal state did not have the bucket size
>>>>>>>>>> set), it failed as well.
>>>>>>>>>>
>>>>>>>>>> Whats the right way to get bucketizing to work ?
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>> Varun
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <ne...@gmail.com>.
To get bucket size, get the ideal state and then getBucketSize() on it. If
you want to change the bucket size, you may first drop the ideal state and
re-create using the new bucket size.
On Mon, Feb 23, 2015 at 3:10 PM, Varun Sharma <va...@pinterest.com> wrote:
> Also, would changing the bucketSize throw an exception or would it cause
> undefined behaviour or would the new bucket size simply be ignored ? This
> is assuming the number of partitions remain the same in the
> setResourceIdealState call ?
>
> Thanks !
> Varun
>
> On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <va...@pinterest.com> wrote:
>
>> Cool - thanks - for a pre existing resource, how can we retrieve the
>> bucket size - do we need to pull in the whole ideal state and then run a
>> getBucketSize() on it ?
>>
>> On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zz...@linkedin.com> wrote:
>>
>>> bucktSize is the size of each bucket, i.e. the number of partitions
>>> each bucket will hold. bucket size can't be changed dynamically.
>>>
>>> ------------------------------
>>> *From:* Varun Sharma [varun@pinterest.com]
>>> *Sent:* Monday, February 23, 2015 2:37 PM
>>>
>>> *To:* user@helix.apache.org
>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>
>>> OK - thanks. Btw, is bucket size the number of buckets or is it the
>>> size of one bucket ? Also can the bucket size be changed dynamically using
>>> setResourceIdealState ?
>>>
>>> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com> wrote:
>>>
>>>> Hi Varun,
>>>>
>>>> The partition number is mainly for calculating the buckets. Partition
>>>> name will remain the same. You can always get and build Helix source code
>>>> from here:
>>>> https://github.com/apache/helix
>>>>
>>>> Changing the partition naming is not hard, but we need to come up
>>>> with a general way to achieve this. We will think about this requirement.
>>>> If you have any good idea for this, you can always submit a pull request on
>>>> github.
>>>>
>>>> Thanks,
>>>> Jason
>>>> ------------------------------
>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>> *Sent:* Monday, February 23, 2015 12:04 PM
>>>>
>>>> *To:* user@helix.apache.org
>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>
>>>> How hard is this to change to '$' - its difficult for us to now
>>>> change the partition names now. Is there a place to get the helix 0.6.4
>>>> source code, make changes and built it ?
>>>>
>>>> Another related question - is the partition number mainly needed for
>>>> hashing into the buckets - the partition name would still continue to be
>>>> "resource_name_partition_num", right ?
>>>>
>>>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com>
>>>> wrote:
>>>>
>>>>> The bucketize logic is expecting partition name following the
>>>>> convention {resource_name}_{partition_number}. In your case, you are using
>>>>> $ ?
>>>>> ------------------------------
>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>> *Sent:* Monday, February 23, 2015 11:48 AM
>>>>>
>>>>> *To:* user@helix.apache.org
>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>
>>>>> Note that we had to prefix each partition with the resource name
>>>>> to make the partitions unique.
>>>>>
>>>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>
>>>>> wrote:
>>>>>
>>>>>> Here is the latest error I am getting:
>>>>>>
>>>>>> 2015-02-23 19:44:48,153 [controller-pool-0]
>>>>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
>>>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>>>>>
>>>>>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>>>>>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>>>>>
>>>>>> *at
>>>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>>>>>
>>>>>> *at
>>>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>>>>>
>>>>>> at
>>>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>>>>>
>>>>>> at
>>>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Not really. The bucketize logic just creates child znode under the
>>>>>>> original ideal state, and each of the child znode holds a subset of the
>>>>>>> total partitions.
>>>>>>> ------------------------------
>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>>>>>
>>>>>>> *To:* user@helix.apache.org
>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>
>>>>>>> One other question - would bucketizing add anything to the
>>>>>>> partition name - just worried about any special symbols (we use _ and $)
>>>>>>> causing issues there ?
>>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Varun,
>>>>>>>>
>>>>>>>> HelixAdmin#addResource was initially used for adding an empty
>>>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal
>>>>>>>> state, so the bucketize logic is missing from addResource. We should
>>>>>>>> definitely fix this. We are also considering merging addResource and
>>>>>>>> rebalance, since separating them are confusing. For now you may use
>>>>>>>> setResourceIdealState as a workaround.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Zhen
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>>>>>> *To:* user@helix.apache.org
>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>>
>>>>>>>> Thanks, Zhen.
>>>>>>>>
>>>>>>>> What about just calling addResource with an idealState with the
>>>>>>>> bucket size - is that expected to not work on its own ?
>>>>>>>>
>>>>>>>> Varun
>>>>>>>>
>>>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Varun,
>>>>>>>>>
>>>>>>>>> Did you do the following?
>>>>>>>>> HelixAdmin#addResource() // set bucket size
>>>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>>>>>
>>>>>>>>> When you set the ideal state, you still need set the bucket size
>>>>>>>>> in the ideal state. This value will be used to bucketize the ideal state.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Jason
>>>>>>>>>
>>>>>>>>> ________________________________________
>>>>>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>>>>>> To: user@helix.apache.org
>>>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am
>>>>>>>>> unable to do so. I am setting the bucketSize in the idealState and using
>>>>>>>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>>>>>>>> size of 1M has been exceeded.
>>>>>>>>>
>>>>>>>>> I also tried using the addResource() api which creates an empty
>>>>>>>>> ideal state with a valid bucket size. However, when I tried to set the
>>>>>>>>> ideal state on that bucket (this ideal state did not have the bucket size
>>>>>>>>> set), it failed as well.
>>>>>>>>>
>>>>>>>>> Whats the right way to get bucketizing to work ?
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>> Varun
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
Also, would changing the bucketSize throw an exception or would it cause
undefined behaviour or would the new bucket size simply be ignored ? This
is assuming the number of partitions remain the same in the
setResourceIdealState call ?
Thanks !
Varun
On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <va...@pinterest.com> wrote:
> Cool - thanks - for a pre existing resource, how can we retrieve the
> bucket size - do we need to pull in the whole ideal state and then run a
> getBucketSize() on it ?
>
> On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zz...@linkedin.com> wrote:
>
>> bucktSize is the size of each bucket, i.e. the number of partitions
>> each bucket will hold. bucket size can't be changed dynamically.
>>
>> ------------------------------
>> *From:* Varun Sharma [varun@pinterest.com]
>> *Sent:* Monday, February 23, 2015 2:37 PM
>>
>> *To:* user@helix.apache.org
>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>
>> OK - thanks. Btw, is bucket size the number of buckets or is it the
>> size of one bucket ? Also can the bucket size be changed dynamically using
>> setResourceIdealState ?
>>
>> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com> wrote:
>>
>>> Hi Varun,
>>>
>>> The partition number is mainly for calculating the buckets. Partition
>>> name will remain the same. You can always get and build Helix source code
>>> from here:
>>> https://github.com/apache/helix
>>>
>>> Changing the partition naming is not hard, but we need to come up with
>>> a general way to achieve this. We will think about this requirement. If you
>>> have any good idea for this, you can always submit a pull request on github.
>>>
>>> Thanks,
>>> Jason
>>> ------------------------------
>>> *From:* Varun Sharma [varun@pinterest.com]
>>> *Sent:* Monday, February 23, 2015 12:04 PM
>>>
>>> *To:* user@helix.apache.org
>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>
>>> How hard is this to change to '$' - its difficult for us to now
>>> change the partition names now. Is there a place to get the helix 0.6.4
>>> source code, make changes and built it ?
>>>
>>> Another related question - is the partition number mainly needed for
>>> hashing into the buckets - the partition name would still continue to be
>>> "resource_name_partition_num", right ?
>>>
>>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com>
>>> wrote:
>>>
>>>> The bucketize logic is expecting partition name following the
>>>> convention {resource_name}_{partition_number}. In your case, you are using
>>>> $ ?
>>>> ------------------------------
>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>> *Sent:* Monday, February 23, 2015 11:48 AM
>>>>
>>>> *To:* user@helix.apache.org
>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>
>>>> Note that we had to prefix each partition with the resource name to
>>>> make the partitions unique.
>>>>
>>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>
>>>> wrote:
>>>>
>>>>> Here is the latest error I am getting:
>>>>>
>>>>> 2015-02-23 19:44:48,153 [controller-pool-0]
>>>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
>>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>>>>
>>>>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>>>>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>>>>
>>>>> *at
>>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>>>>
>>>>> *at
>>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>>>>
>>>>> at
>>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>>>>
>>>>> at
>>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>>>>
>>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>
>>>>> wrote:
>>>>>
>>>>>> Not really. The bucketize logic just creates child znode under the
>>>>>> original ideal state, and each of the child znode holds a subset of the
>>>>>> total partitions.
>>>>>> ------------------------------
>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>>>>
>>>>>> *To:* user@helix.apache.org
>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>
>>>>>> One other question - would bucketizing add anything to the
>>>>>> partition name - just worried about any special symbols (we use _ and $)
>>>>>> causing issues there ?
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Varun,
>>>>>>>
>>>>>>> HelixAdmin#addResource was initially used for adding an empty
>>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal
>>>>>>> state, so the bucketize logic is missing from addResource. We should
>>>>>>> definitely fix this. We are also considering merging addResource and
>>>>>>> rebalance, since separating them are confusing. For now you may use
>>>>>>> setResourceIdealState as a workaround.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Zhen
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>>>>> *To:* user@helix.apache.org
>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>
>>>>>>> Thanks, Zhen.
>>>>>>>
>>>>>>> What about just calling addResource with an idealState with the
>>>>>>> bucket size - is that expected to not work on its own ?
>>>>>>>
>>>>>>> Varun
>>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Varun,
>>>>>>>>
>>>>>>>> Did you do the following?
>>>>>>>> HelixAdmin#addResource() // set bucket size
>>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>>>>
>>>>>>>> When you set the ideal state, you still need set the bucket size in
>>>>>>>> the ideal state. This value will be used to bucketize the ideal state.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jason
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>>>>> To: user@helix.apache.org
>>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am
>>>>>>>> unable to do so. I am setting the bucketSize in the idealState and using
>>>>>>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>>>>>>> size of 1M has been exceeded.
>>>>>>>>
>>>>>>>> I also tried using the addResource() api which creates an empty
>>>>>>>> ideal state with a valid bucket size. However, when I tried to set the
>>>>>>>> ideal state on that bucket (this ideal state did not have the bucket size
>>>>>>>> set), it failed as well.
>>>>>>>>
>>>>>>>> Whats the right way to get bucketizing to work ?
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>> Varun
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
Cool - thanks - for a pre existing resource, how can we retrieve the bucket
size - do we need to pull in the whole ideal state and then run a
getBucketSize() on it ?
On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zz...@linkedin.com> wrote:
> bucktSize is the size of each bucket, i.e. the number of partitions each
> bucket will hold. bucket size can't be changed dynamically.
>
> ------------------------------
> *From:* Varun Sharma [varun@pinterest.com]
> *Sent:* Monday, February 23, 2015 2:37 PM
>
> *To:* user@helix.apache.org
> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>
> OK - thanks. Btw, is bucket size the number of buckets or is it the
> size of one bucket ? Also can the bucket size be changed dynamically using
> setResourceIdealState ?
>
> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com> wrote:
>
>> Hi Varun,
>>
>> The partition number is mainly for calculating the buckets. Partition
>> name will remain the same. You can always get and build Helix source code
>> from here:
>> https://github.com/apache/helix
>>
>> Changing the partition naming is not hard, but we need to come up with
>> a general way to achieve this. We will think about this requirement. If you
>> have any good idea for this, you can always submit a pull request on github.
>>
>> Thanks,
>> Jason
>> ------------------------------
>> *From:* Varun Sharma [varun@pinterest.com]
>> *Sent:* Monday, February 23, 2015 12:04 PM
>>
>> *To:* user@helix.apache.org
>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>
>> How hard is this to change to '$' - its difficult for us to now
>> change the partition names now. Is there a place to get the helix 0.6.4
>> source code, make changes and built it ?
>>
>> Another related question - is the partition number mainly needed for
>> hashing into the buckets - the partition name would still continue to be
>> "resource_name_partition_num", right ?
>>
>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com> wrote:
>>
>>> The bucketize logic is expecting partition name following the
>>> convention {resource_name}_{partition_number}. In your case, you are using
>>> $ ?
>>> ------------------------------
>>> *From:* Varun Sharma [varun@pinterest.com]
>>> *Sent:* Monday, February 23, 2015 11:48 AM
>>>
>>> *To:* user@helix.apache.org
>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>
>>> Note that we had to prefix each partition with the resource name to
>>> make the partitions unique.
>>>
>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>
>>> wrote:
>>>
>>>> Here is the latest error I am getting:
>>>>
>>>> 2015-02-23 19:44:48,153 [controller-pool-0]
>>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>>>
>>>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>>>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>>>
>>>> *at
>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>>>
>>>> *at
>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>>>
>>>> at
>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>>>
>>>> at
>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>>>
>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>
>>>> wrote:
>>>>
>>>>> Not really. The bucketize logic just creates child znode under the
>>>>> original ideal state, and each of the child znode holds a subset of the
>>>>> total partitions.
>>>>> ------------------------------
>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>>>
>>>>> *To:* user@helix.apache.org
>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>
>>>>> One other question - would bucketizing add anything to the
>>>>> partition name - just worried about any special symbols (we use _ and $)
>>>>> causing issues there ?
>>>>>
>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Varun,
>>>>>>
>>>>>> HelixAdmin#addResource was initially used for adding an empty
>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal
>>>>>> state, so the bucketize logic is missing from addResource. We should
>>>>>> definitely fix this. We are also considering merging addResource and
>>>>>> rebalance, since separating them are confusing. For now you may use
>>>>>> setResourceIdealState as a workaround.
>>>>>>
>>>>>> Thanks,
>>>>>> Zhen
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>>>> *To:* user@helix.apache.org
>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>
>>>>>> Thanks, Zhen.
>>>>>>
>>>>>> What about just calling addResource with an idealState with the
>>>>>> bucket size - is that expected to not work on its own ?
>>>>>>
>>>>>> Varun
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Varun,
>>>>>>>
>>>>>>> Did you do the following?
>>>>>>> HelixAdmin#addResource() // set bucket size
>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>>>
>>>>>>> When you set the ideal state, you still need set the bucket size in
>>>>>>> the ideal state. This value will be used to bucketize the ideal state.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jason
>>>>>>>
>>>>>>> ________________________________________
>>>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>>>> To: user@helix.apache.org
>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am
>>>>>>> unable to do so. I am setting the bucketSize in the idealState and using
>>>>>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>>>>>> size of 1M has been exceeded.
>>>>>>>
>>>>>>> I also tried using the addResource() api which creates an empty
>>>>>>> ideal state with a valid bucket size. However, when I tried to set the
>>>>>>> ideal state on that bucket (this ideal state did not have the bucket size
>>>>>>> set), it failed as well.
>>>>>>>
>>>>>>> Whats the right way to get bucketizing to work ?
>>>>>>>
>>>>>>> Thanks !
>>>>>>> Varun
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
RE: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <zz...@linkedin.com>.
bucktSize is the size of each bucket, i.e. the number of partitions each bucket will hold. bucket size can't be changed dynamically.
________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 2:37 PM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4
OK - thanks. Btw, is bucket size the number of buckets or is it the size of one bucket ? Also can the bucket size be changed dynamically using setResourceIdealState ?
On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
The partition number is mainly for calculating the buckets. Partition name will remain the same. You can always get and build Helix source code from here:
https://github.com/apache/helix
Changing the partition naming is not hard, but we need to come up with a general way to achieve this. We will think about this requirement. If you have any good idea for this, you can always submit a pull request on github.
Thanks,
Jason
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 12:04 PM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
How hard is this to change to '$' - its difficult for us to now change the partition names now. Is there a place to get the helix 0.6.4 source code, make changes and built it ?
Another related question - is the partition number mainly needed for hashing into the buckets - the partition name would still continue to be "resource_name_partition_num", right ?
On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
The bucketize logic is expecting partition name following the convention {resource_name}_{partition_number}. In your case, you are using $ ?
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 11:48 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Note that we had to prefix each partition with the resource name to make the partitions unique.
On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>> wrote:
Here is the latest error I am getting:
2015-02-23 19:44:48,153 [controller-pool-0] (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for $terrapin$data$pinpin_board_join$1424377884722, rolling back.
java.lang.IllegalArgumentException: Could NOT parse partition# (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0
at org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)
at org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)
at org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
at org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Not really. The bucketize logic just creates child znode under the original ideal state, and each of the child znode holds a subset of the total partitions.
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:37 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
One other question - would bucketizing add anything to the partition name - just worried about any special symbols (we use _ and $) causing issues there ?
On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
HelixAdmin#addResource was initially used for adding an empty resource only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so the bucketize logic is missing from addResource. We should definitely fix this. We are also considering merging addResource and rebalance, since separating them are confusing. For now you may use setResourceIdealState as a workaround.
Thanks,
Zhen
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:20 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Thanks, Zhen.
What about just calling addResource with an idealState with the bucket size - is that expected to not work on its own ?
Varun
On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size
When you set the ideal state, you still need set the bucket size in the ideal state. This value will be used to bucketize the ideal state.
Thanks,
Jason
________________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting the bucketSize in the idealState and using addResource() and setResourceIdealState and it ends up complaining that ZK size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state with a valid bucket size. However, when I tried to set the ideal state on that bucket (this ideal state did not have the bucket size set), it failed as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
OK - thanks. Btw, is bucket size the number of buckets or is it the size of
one bucket ? Also can the bucket size be changed dynamically using
setResourceIdealState ?
On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zz...@linkedin.com> wrote:
> Hi Varun,
>
> The partition number is mainly for calculating the buckets. Partition
> name will remain the same. You can always get and build Helix source code
> from here:
> https://github.com/apache/helix
>
> Changing the partition naming is not hard, but we need to come up with a
> general way to achieve this. We will think about this requirement. If you
> have any good idea for this, you can always submit a pull request on github.
>
> Thanks,
> Jason
> ------------------------------
> *From:* Varun Sharma [varun@pinterest.com]
> *Sent:* Monday, February 23, 2015 12:04 PM
>
> *To:* user@helix.apache.org
> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>
> How hard is this to change to '$' - its difficult for us to now change
> the partition names now. Is there a place to get the helix 0.6.4 source
> code, make changes and built it ?
>
> Another related question - is the partition number mainly needed for
> hashing into the buckets - the partition name would still continue to be
> "resource_name_partition_num", right ?
>
> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com> wrote:
>
>> The bucketize logic is expecting partition name following the
>> convention {resource_name}_{partition_number}. In your case, you are using
>> $ ?
>> ------------------------------
>> *From:* Varun Sharma [varun@pinterest.com]
>> *Sent:* Monday, February 23, 2015 11:48 AM
>>
>> *To:* user@helix.apache.org
>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>
>> Note that we had to prefix each partition with the resource name to
>> make the partitions unique.
>>
>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>
>> wrote:
>>
>>> Here is the latest error I am getting:
>>>
>>> 2015-02-23 19:44:48,153 [controller-pool-0]
>>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>>
>>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>>
>>> *at
>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>>
>>> *at
>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>>
>>> at
>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>>
>>> at
>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>>
>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>
>>> wrote:
>>>
>>>> Not really. The bucketize logic just creates child znode under the
>>>> original ideal state, and each of the child znode holds a subset of the
>>>> total partitions.
>>>> ------------------------------
>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>>
>>>> *To:* user@helix.apache.org
>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>
>>>> One other question - would bucketizing add anything to the
>>>> partition name - just worried about any special symbols (we use _ and $)
>>>> causing issues there ?
>>>>
>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>
>>>> wrote:
>>>>
>>>>> Hi Varun,
>>>>>
>>>>> HelixAdmin#addResource was initially used for adding an empty
>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal
>>>>> state, so the bucketize logic is missing from addResource. We should
>>>>> definitely fix this. We are also considering merging addResource and
>>>>> rebalance, since separating them are confusing. For now you may use
>>>>> setResourceIdealState as a workaround.
>>>>>
>>>>> Thanks,
>>>>> Zhen
>>>>>
>>>>> ------------------------------
>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>>> *To:* user@helix.apache.org
>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>
>>>>> Thanks, Zhen.
>>>>>
>>>>> What about just calling addResource with an idealState with the
>>>>> bucket size - is that expected to not work on its own ?
>>>>>
>>>>> Varun
>>>>>
>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Varun,
>>>>>>
>>>>>> Did you do the following?
>>>>>> HelixAdmin#addResource() // set bucket size
>>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>>
>>>>>> When you set the ideal state, you still need set the bucket size in
>>>>>> the ideal state. This value will be used to bucketize the ideal state.
>>>>>>
>>>>>> Thanks,
>>>>>> Jason
>>>>>>
>>>>>> ________________________________________
>>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>>> To: user@helix.apache.org
>>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am
>>>>>> unable to do so. I am setting the bucketSize in the idealState and using
>>>>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>>>>> size of 1M has been exceeded.
>>>>>>
>>>>>> I also tried using the addResource() api which creates an empty ideal
>>>>>> state with a valid bucket size. However, when I tried to set the ideal
>>>>>> state on that bucket (this ideal state did not have the bucket size set),
>>>>>> it failed as well.
>>>>>>
>>>>>> Whats the right way to get bucketizing to work ?
>>>>>>
>>>>>> Thanks !
>>>>>> Varun
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
RE: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <zz...@linkedin.com>.
Hi Varun,
The partition number is mainly for calculating the buckets. Partition name will remain the same. You can always get and build Helix source code from here:
https://github.com/apache/helix
Changing the partition naming is not hard, but we need to come up with a general way to achieve this. We will think about this requirement. If you have any good idea for this, you can always submit a pull request on github.
Thanks,
Jason
________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 12:04 PM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4
How hard is this to change to '$' - its difficult for us to now change the partition names now. Is there a place to get the helix 0.6.4 source code, make changes and built it ?
Another related question - is the partition number mainly needed for hashing into the buckets - the partition name would still continue to be "resource_name_partition_num", right ?
On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
The bucketize logic is expecting partition name following the convention {resource_name}_{partition_number}. In your case, you are using $ ?
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 11:48 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Note that we had to prefix each partition with the resource name to make the partitions unique.
On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>> wrote:
Here is the latest error I am getting:
2015-02-23 19:44:48,153 [controller-pool-0] (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for $terrapin$data$pinpin_board_join$1424377884722, rolling back.
java.lang.IllegalArgumentException: Could NOT parse partition# (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0
at org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)
at org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)
at org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
at org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Not really. The bucketize logic just creates child znode under the original ideal state, and each of the child znode holds a subset of the total partitions.
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:37 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
One other question - would bucketizing add anything to the partition name - just worried about any special symbols (we use _ and $) causing issues there ?
On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
HelixAdmin#addResource was initially used for adding an empty resource only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so the bucketize logic is missing from addResource. We should definitely fix this. We are also considering merging addResource and rebalance, since separating them are confusing. For now you may use setResourceIdealState as a workaround.
Thanks,
Zhen
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:20 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Thanks, Zhen.
What about just calling addResource with an idealState with the bucket size - is that expected to not work on its own ?
Varun
On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size
When you set the ideal state, you still need set the bucket size in the ideal state. This value will be used to bucketize the ideal state.
Thanks,
Jason
________________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting the bucketSize in the idealState and using addResource() and setResourceIdealState and it ends up complaining that ZK size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state with a valid bucket size. However, when I tried to set the ideal state on that bucket (this ideal state did not have the bucket size set), it failed as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
How hard is this to change to '$' - its difficult for us to now change the
partition names now. Is there a place to get the helix 0.6.4 source code,
make changes and built it ?
Another related question - is the partition number mainly needed for
hashing into the buckets - the partition name would still continue to be
"resource_name_partition_num", right ?
On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zz...@linkedin.com> wrote:
> The bucketize logic is expecting partition name following the convention
> {resource_name}_{partition_number}. In your case, you are using $ ?
> ------------------------------
> *From:* Varun Sharma [varun@pinterest.com]
> *Sent:* Monday, February 23, 2015 11:48 AM
>
> *To:* user@helix.apache.org
> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>
> Note that we had to prefix each partition with the resource name to
> make the partitions unique.
>
> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>
> wrote:
>
>> Here is the latest error I am getting:
>>
>> 2015-02-23 19:44:48,153 [controller-pool-0]
>> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>
>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>
>> *at
>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>
>> *at
>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>
>> at
>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>
>> at
>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>
>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com> wrote:
>>
>>> Not really. The bucketize logic just creates child znode under the
>>> original ideal state, and each of the child znode holds a subset of the
>>> total partitions.
>>> ------------------------------
>>> *From:* Varun Sharma [varun@pinterest.com]
>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>
>>> *To:* user@helix.apache.org
>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>
>>> One other question - would bucketizing add anything to the partition
>>> name - just worried about any special symbols (we use _ and $) causing
>>> issues there ?
>>>
>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>
>>> wrote:
>>>
>>>> Hi Varun,
>>>>
>>>> HelixAdmin#addResource was initially used for adding an empty
>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal
>>>> state, so the bucketize logic is missing from addResource. We should
>>>> definitely fix this. We are also considering merging addResource and
>>>> rebalance, since separating them are confusing. For now you may use
>>>> setResourceIdealState as a workaround.
>>>>
>>>> Thanks,
>>>> Zhen
>>>>
>>>> ------------------------------
>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>> *To:* user@helix.apache.org
>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>
>>>> Thanks, Zhen.
>>>>
>>>> What about just calling addResource with an idealState with the
>>>> bucket size - is that expected to not work on its own ?
>>>>
>>>> Varun
>>>>
>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>
>>>> wrote:
>>>>
>>>>> Hi Varun,
>>>>>
>>>>> Did you do the following?
>>>>> HelixAdmin#addResource() // set bucket size
>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>
>>>>> When you set the ideal state, you still need set the bucket size in
>>>>> the ideal state. This value will be used to bucketize the ideal state.
>>>>>
>>>>> Thanks,
>>>>> Jason
>>>>>
>>>>> ________________________________________
>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>> To: user@helix.apache.org
>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am
>>>>> unable to do so. I am setting the bucketSize in the idealState and using
>>>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>>>> size of 1M has been exceeded.
>>>>>
>>>>> I also tried using the addResource() api which creates an empty ideal
>>>>> state with a valid bucket size. However, when I tried to set the ideal
>>>>> state on that bucket (this ideal state did not have the bucket size set),
>>>>> it failed as well.
>>>>>
>>>>> Whats the right way to get bucketizing to work ?
>>>>>
>>>>> Thanks !
>>>>> Varun
>>>>>
>>>>
>>>>
>>>
>>
>
RE: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <zz...@linkedin.com>.
The bucketize logic is expecting partition name following the convention {resource_name}_{partition_number}. In your case, you are using $ ?
________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 11:48 AM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4
Note that we had to prefix each partition with the resource name to make the partitions unique.
On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com>> wrote:
Here is the latest error I am getting:
2015-02-23 19:44:48,153 [controller-pool-0] (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for $terrapin$data$pinpin_board_join$1424377884722, rolling back.
java.lang.IllegalArgumentException: Could NOT parse partition# (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0
at org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)
at org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)
at org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
at org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Not really. The bucketize logic just creates child znode under the original ideal state, and each of the child znode holds a subset of the total partitions.
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:37 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
One other question - would bucketizing add anything to the partition name - just worried about any special symbols (we use _ and $) causing issues there ?
On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
HelixAdmin#addResource was initially used for adding an empty resource only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so the bucketize logic is missing from addResource. We should definitely fix this. We are also considering merging addResource and rebalance, since separating them are confusing. For now you may use setResourceIdealState as a workaround.
Thanks,
Zhen
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:20 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Thanks, Zhen.
What about just calling addResource with an idealState with the bucket size - is that expected to not work on its own ?
Varun
On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size
When you set the ideal state, you still need set the bucket size in the ideal state. This value will be used to bucketize the ideal state.
Thanks,
Jason
________________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting the bucketSize in the idealState and using addResource() and setResourceIdealState and it ends up complaining that ZK size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state with a valid bucket size. However, when I tried to set the ideal state on that bucket (this ideal state did not have the bucket size set), it failed as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
Note that we had to prefix each partition with the resource name to make
the partitions unique.
On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <va...@pinterest.com> wrote:
> Here is the latest error I am getting:
>
> 2015-02-23 19:44:48,153 [controller-pool-0]
> (TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>
> *java.lang.IllegalArgumentException: Could NOT parse partition#
> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>
> * at
> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>
> * at
> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>
> at
> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>
> at
> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>
> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com> wrote:
>
>> Not really. The bucketize logic just creates child znode under the
>> original ideal state, and each of the child znode holds a subset of the
>> total partitions.
>> ------------------------------
>> *From:* Varun Sharma [varun@pinterest.com]
>> *Sent:* Monday, February 23, 2015 10:37 AM
>>
>> *To:* user@helix.apache.org
>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>
>> One other question - would bucketizing add anything to the partition
>> name - just worried about any special symbols (we use _ and $) causing
>> issues there ?
>>
>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com> wrote:
>>
>>> Hi Varun,
>>>
>>> HelixAdmin#addResource was initially used for adding an empty resource
>>> only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so
>>> the bucketize logic is missing from addResource. We should definitely fix
>>> this. We are also considering merging addResource and rebalance, since
>>> separating them are confusing. For now you may use setResourceIdealState as
>>> a workaround.
>>>
>>> Thanks,
>>> Zhen
>>>
>>> ------------------------------
>>> *From:* Varun Sharma [varun@pinterest.com]
>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>> *To:* user@helix.apache.org
>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>
>>> Thanks, Zhen.
>>>
>>> What about just calling addResource with an idealState with the bucket
>>> size - is that expected to not work on its own ?
>>>
>>> Varun
>>>
>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>
>>> wrote:
>>>
>>>> Hi Varun,
>>>>
>>>> Did you do the following?
>>>> HelixAdmin#addResource() // set bucket size
>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>
>>>> When you set the ideal state, you still need set the bucket size in the
>>>> ideal state. This value will be used to bucketize the ideal state.
>>>>
>>>> Thanks,
>>>> Jason
>>>>
>>>> ________________________________________
>>>> From: Varun Sharma [varun@pinterest.com]
>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>> To: user@helix.apache.org
>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>
>>>> Hi,
>>>>
>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am unable
>>>> to do so. I am setting the bucketSize in the idealState and using
>>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>>> size of 1M has been exceeded.
>>>>
>>>> I also tried using the addResource() api which creates an empty ideal
>>>> state with a valid bucket size. However, when I tried to set the ideal
>>>> state on that bucket (this ideal state did not have the bucket size set),
>>>> it failed as well.
>>>>
>>>> Whats the right way to get bucketizing to work ?
>>>>
>>>> Thanks !
>>>> Varun
>>>>
>>>
>>>
>>
>
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
Here is the latest error I am getting:
2015-02-23 19:44:48,153 [controller-pool-0]
(TerrapinControllerServiceImpl.java:129) WARN Resource creation failed for
$terrapin$data$pinpin_board_join$1424377884722, rolling back.
*java.lang.IllegalArgumentException: Could NOT parse partition#
(join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
* at
org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
* at
org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
at
org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
at
org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zz...@linkedin.com> wrote:
> Not really. The bucketize logic just creates child znode under the
> original ideal state, and each of the child znode holds a subset of the
> total partitions.
> ------------------------------
> *From:* Varun Sharma [varun@pinterest.com]
> *Sent:* Monday, February 23, 2015 10:37 AM
>
> *To:* user@helix.apache.org
> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>
> One other question - would bucketizing add anything to the partition
> name - just worried about any special symbols (we use _ and $) causing
> issues there ?
>
> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com> wrote:
>
>> Hi Varun,
>>
>> HelixAdmin#addResource was initially used for adding an empty resource
>> only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so
>> the bucketize logic is missing from addResource. We should definitely fix
>> this. We are also considering merging addResource and rebalance, since
>> separating them are confusing. For now you may use setResourceIdealState as
>> a workaround.
>>
>> Thanks,
>> Zhen
>>
>> ------------------------------
>> *From:* Varun Sharma [varun@pinterest.com]
>> *Sent:* Monday, February 23, 2015 10:20 AM
>> *To:* user@helix.apache.org
>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>
>> Thanks, Zhen.
>>
>> What about just calling addResource with an idealState with the bucket
>> size - is that expected to not work on its own ?
>>
>> Varun
>>
>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com> wrote:
>>
>>> Hi Varun,
>>>
>>> Did you do the following?
>>> HelixAdmin#addResource() // set bucket size
>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>
>>> When you set the ideal state, you still need set the bucket size in the
>>> ideal state. This value will be used to bucketize the ideal state.
>>>
>>> Thanks,
>>> Jason
>>>
>>> ________________________________________
>>> From: Varun Sharma [varun@pinterest.com]
>>> Sent: Monday, February 23, 2015 1:51 AM
>>> To: user@helix.apache.org
>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>
>>> Hi,
>>>
>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am unable
>>> to do so. I am setting the bucketSize in the idealState and using
>>> addResource() and setResourceIdealState and it ends up complaining that ZK
>>> size of 1M has been exceeded.
>>>
>>> I also tried using the addResource() api which creates an empty ideal
>>> state with a valid bucket size. However, when I tried to set the ideal
>>> state on that bucket (this ideal state did not have the bucket size set),
>>> it failed as well.
>>>
>>> Whats the right way to get bucketizing to work ?
>>>
>>> Thanks !
>>> Varun
>>>
>>
>>
>
RE: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <zz...@linkedin.com>.
Not really. The bucketize logic just creates child znode under the original ideal state, and each of the child znode holds a subset of the total partitions.
________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 10:37 AM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4
One other question - would bucketizing add anything to the partition name - just worried about any special symbols (we use _ and $) causing issues there ?
On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
HelixAdmin#addResource was initially used for adding an empty resource only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so the bucketize logic is missing from addResource. We should definitely fix this. We are also considering merging addResource and rebalance, since separating them are confusing. For now you may use setResourceIdealState as a workaround.
Thanks,
Zhen
________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 10:20 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4
Thanks, Zhen.
What about just calling addResource with an idealState with the bucket size - is that expected to not work on its own ?
Varun
On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size
When you set the ideal state, you still need set the bucket size in the ideal state. This value will be used to bucketize the ideal state.
Thanks,
Jason
________________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting the bucketSize in the idealState and using addResource() and setResourceIdealState and it ends up complaining that ZK size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state with a valid bucket size. However, when I tried to set the ideal state on that bucket (this ideal state did not have the bucket size set), it failed as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
One other question - would bucketizing add anything to the partition name -
just worried about any special symbols (we use _ and $) causing issues
there ?
On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zz...@linkedin.com> wrote:
> Hi Varun,
>
> HelixAdmin#addResource was initially used for adding an empty resource
> only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so
> the bucketize logic is missing from addResource. We should definitely fix
> this. We are also considering merging addResource and rebalance, since
> separating them are confusing. For now you may use setResourceIdealState as
> a workaround.
>
> Thanks,
> Zhen
>
> ------------------------------
> *From:* Varun Sharma [varun@pinterest.com]
> *Sent:* Monday, February 23, 2015 10:20 AM
> *To:* user@helix.apache.org
> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>
> Thanks, Zhen.
>
> What about just calling addResource with an idealState with the bucket
> size - is that expected to not work on its own ?
>
> Varun
>
> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com> wrote:
>
>> Hi Varun,
>>
>> Did you do the following?
>> HelixAdmin#addResource() // set bucket size
>> HelixAdmin#setResourceIdealState() // no bucket size
>>
>> When you set the ideal state, you still need set the bucket size in the
>> ideal state. This value will be used to bucketize the ideal state.
>>
>> Thanks,
>> Jason
>>
>> ________________________________________
>> From: Varun Sharma [varun@pinterest.com]
>> Sent: Monday, February 23, 2015 1:51 AM
>> To: user@helix.apache.org
>> Subject: Bucketized resources not working in Helix 0.6.4
>>
>> Hi,
>>
>> I am trying to bucketize a resource on Helix 0.6.4 - however, am unable
>> to do so. I am setting the bucketSize in the idealState and using
>> addResource() and setResourceIdealState and it ends up complaining that ZK
>> size of 1M has been exceeded.
>>
>> I also tried using the addResource() api which creates an empty ideal
>> state with a valid bucket size. However, when I tried to set the ideal
>> state on that bucket (this ideal state did not have the bucket size set),
>> it failed as well.
>>
>> Whats the right way to get bucketizing to work ?
>>
>> Thanks !
>> Varun
>>
>
>
RE: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <zz...@linkedin.com>.
Hi Varun,
HelixAdmin#addResource was initially used for adding an empty resource only, followed by HelixAdmin#rebalance, which bucketize the ideal state, so the bucketize logic is missing from addResource. We should definitely fix this. We are also considering merging addResource and rebalance, since separating them are confusing. For now you may use setResourceIdealState as a workaround.
Thanks,
Zhen
________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 10:20 AM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4
Thanks, Zhen.
What about just calling addResource with an idealState with the bucket size - is that expected to not work on its own ?
Varun
On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com>> wrote:
Hi Varun,
Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size
When you set the ideal state, you still need set the bucket size in the ideal state. This value will be used to bucketize the ideal state.
Thanks,
Jason
________________________________________
From: Varun Sharma [varun@pinterest.com<ma...@pinterest.com>]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org<ma...@helix.apache.org>
Subject: Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting the bucketSize in the idealState and using addResource() and setResourceIdealState and it ends up complaining that ZK size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state with a valid bucket size. However, when I tried to set the ideal state on that bucket (this ideal state did not have the bucket size set), it failed as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun
Re: Bucketized resources not working in Helix 0.6.4
Posted by Varun Sharma <va...@pinterest.com>.
Thanks, Zhen.
What about just calling addResource with an idealState with the bucket size
- is that expected to not work on its own ?
Varun
On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zz...@linkedin.com> wrote:
> Hi Varun,
>
> Did you do the following?
> HelixAdmin#addResource() // set bucket size
> HelixAdmin#setResourceIdealState() // no bucket size
>
> When you set the ideal state, you still need set the bucket size in the
> ideal state. This value will be used to bucketize the ideal state.
>
> Thanks,
> Jason
>
> ________________________________________
> From: Varun Sharma [varun@pinterest.com]
> Sent: Monday, February 23, 2015 1:51 AM
> To: user@helix.apache.org
> Subject: Bucketized resources not working in Helix 0.6.4
>
> Hi,
>
> I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to
> do so. I am setting the bucketSize in the idealState and using
> addResource() and setResourceIdealState and it ends up complaining that ZK
> size of 1M has been exceeded.
>
> I also tried using the addResource() api which creates an empty ideal
> state with a valid bucket size. However, when I tried to set the ideal
> state on that bucket (this ideal state did not have the bucket size set),
> it failed as well.
>
> Whats the right way to get bucketizing to work ?
>
> Thanks !
> Varun
>
RE: Bucketized resources not working in Helix 0.6.4
Posted by Zhen Zhang <zz...@linkedin.com>.
Hi Varun,
Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size
When you set the ideal state, you still need set the bucket size in the ideal state. This value will be used to bucketize the ideal state.
Thanks,
Jason
________________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org
Subject: Bucketized resources not working in Helix 0.6.4
Hi,
I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting the bucketSize in the idealState and using addResource() and setResourceIdealState and it ends up complaining that ZK size of 1M has been exceeded.
I also tried using the addResource() api which creates an empty ideal state with a valid bucket size. However, when I tried to set the ideal state on that bucket (this ideal state did not have the bucket size set), it failed as well.
Whats the right way to get bucketizing to work ?
Thanks !
Varun