You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Patrik Modesto <pa...@gmail.com> on 2012/07/19 08:49:06 UTC
random partitioner and key scan
Hi,
I know that RandomPartitioner does MD5 of a key and the MD5 is then
used for key distribution AND key ordering. I was just wondering if
it's possible to have RandomPartitioner just for key distribution and
OrderedPartitioner just for per-node key ordering. That would solve
the often requested key scan feature.
Regards,
Patrik
Re: random partitioner and key scan
Posted by prasenjit mukherjee <pr...@gmail.com>.
Agree to aaron's point. But still I think there are ways to overcome
this problem, as I believe that the row-scan use-case is very
important. One simple ( but expensive ) approach could be to duplicate
the rows with RP tokens on primary node, which will be used only in
case of repairs.
Any thoughts ?
Thanks,
Prasenjit
On 7/20/12, aaron morton <aa...@thelastpickle.com> wrote:
> Repair and token moves work on ranges of Tokens, not row keys. These
> operations need to scan through all the rows in the token range.
>
> Ordering the rows by row key locally would mean that every row on the node
> would have to be scanned to find the ones whose token was in the required
> token range.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/07/2012, at 4:46 AM, prasenjit mukherjee wrote:
>
>> When a request for token change is issued ( via nodetool ) then on
>> what basis a node will move some of its rows to other node, as there
>> will be no way to scan rows based on MD5 hash in a given node ( if the
>> keys are not prefixed with MD5 hash )
>>
>> On Thu, Jul 19, 2012 at 1:43 PM, Patrik Modesto
>> <pa...@gmail.com> wrote:
>>> Hi Prasenjit,
>>>
>>> I don't see the need to recalculate anything. One key has a one MD5
>>> hash, it doesn't change. Just use the hash to select a node, than just
>>> the plain key. Can you elaborate on the redistribution please?
>>>
>>> Regards,
>>> P.
>>>
>>> On Thu, Jul 19, 2012 at 9:09 AM, prasenjit mukherjee
>>> <pr...@gmail.com> wrote:
>>>> The probem could be while redistributing the tokens. In that case the
>>>> hashes has to be recalculated on each fo the candidate node.
>>>>
>>>> -Thanks,
>>>> Prasenjit
>>>>
>>>> On Thu, Jul 19, 2012 at 12:19 PM, Patrik Modesto
>>>> <pa...@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I know that RandomPartitioner does MD5 of a key and the MD5 is then
>>>>> used for key distribution AND key ordering. I was just wondering if
>>>>> it's possible to have RandomPartitioner just for key distribution and
>>>>> OrderedPartitioner just for per-node key ordering. That would solve
>>>>> the often requested key scan feature.
>>>>>
>>>>> Regards,
>>>>> Patrik
>
>
--
Sent from my mobile device
Re: random partitioner and key scan
Posted by prasenjit mukherjee <pr...@gmail.com>.
On Sat, Jul 21, 2012 at 3:37 PM, Sylvain Lebresne <sy...@datastax.com> wrote:
>> I don't know much about Cassandra internals, but from a user point of
>> view, a scan for a range of tokens is not a common use-case.
>
> All of boostrap/move/decommission/repair rely heavily on being able to
> scan efficiently a range of token. Otherwise, a
> boostrap/move/decommission/repair of a node would require the node and
> all the nodes that share some data with it would have to read all
> their data (to extract the correct token range). This would have a
> hugely detrimental impact of the performance of those operations and
> is therefore not an option.
Why cant we keep duplicated rows with md5 hashing, which will be used
in such scenarios ( bootstrap/move etc ). Otherwise the
OrderedPartitining will be used for loookups.
>
> --
> Sylvain
Re: random partitioner and key scan
Posted by Sylvain Lebresne <sy...@datastax.com>.
> I don't know much about Cassandra internals, but from a user point of
> view, a scan for a range of tokens is not a common use-case.
All of boostrap/move/decommission/repair rely heavily on being able to
scan efficiently a range of token. Otherwise, a
boostrap/move/decommission/repair of a node would require the node and
all the nodes that share some data with it would have to read all
their data (to extract the correct token range). This would have a
hugely detrimental impact of the performance of those operations and
is therefore not an option.
--
Sylvain
Re: random partitioner and key scan
Posted by Patrik Modesto <pa...@gmail.com>.
On Fri, Jul 20, 2012 at 11:17 AM, aaron morton <aa...@thelastpickle.com> wrote:
> Ordering the rows by row key locally would mean that every row on the node
> would have to be scanned to find the ones whose token was in the required
> token range.
I don't know much about Cassandra internals, but from a user point of
view, a scan for a range of tokens is not a common use-case. Does
Cassandra need it so much that it would be impossible to implement the
md5_distribution+natural_keys localy?
Regards,
P.
Re: random partitioner and key scan
Posted by aaron morton <aa...@thelastpickle.com>.
Repair and token moves work on ranges of Tokens, not row keys. These operations need to scan through all the rows in the token range.
Ordering the rows by row key locally would mean that every row on the node would have to be scanned to find the ones whose token was in the required token range.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 20/07/2012, at 4:46 AM, prasenjit mukherjee wrote:
> When a request for token change is issued ( via nodetool ) then on
> what basis a node will move some of its rows to other node, as there
> will be no way to scan rows based on MD5 hash in a given node ( if the
> keys are not prefixed with MD5 hash )
>
> On Thu, Jul 19, 2012 at 1:43 PM, Patrik Modesto
> <pa...@gmail.com> wrote:
>> Hi Prasenjit,
>>
>> I don't see the need to recalculate anything. One key has a one MD5
>> hash, it doesn't change. Just use the hash to select a node, than just
>> the plain key. Can you elaborate on the redistribution please?
>>
>> Regards,
>> P.
>>
>> On Thu, Jul 19, 2012 at 9:09 AM, prasenjit mukherjee
>> <pr...@gmail.com> wrote:
>>> The probem could be while redistributing the tokens. In that case the
>>> hashes has to be recalculated on each fo the candidate node.
>>>
>>> -Thanks,
>>> Prasenjit
>>>
>>> On Thu, Jul 19, 2012 at 12:19 PM, Patrik Modesto
>>> <pa...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I know that RandomPartitioner does MD5 of a key and the MD5 is then
>>>> used for key distribution AND key ordering. I was just wondering if
>>>> it's possible to have RandomPartitioner just for key distribution and
>>>> OrderedPartitioner just for per-node key ordering. That would solve
>>>> the often requested key scan feature.
>>>>
>>>> Regards,
>>>> Patrik
Re: random partitioner and key scan
Posted by prasenjit mukherjee <pr...@gmail.com>.
When a request for token change is issued ( via nodetool ) then on
what basis a node will move some of its rows to other node, as there
will be no way to scan rows based on MD5 hash in a given node ( if the
keys are not prefixed with MD5 hash )
On Thu, Jul 19, 2012 at 1:43 PM, Patrik Modesto
<pa...@gmail.com> wrote:
> Hi Prasenjit,
>
> I don't see the need to recalculate anything. One key has a one MD5
> hash, it doesn't change. Just use the hash to select a node, than just
> the plain key. Can you elaborate on the redistribution please?
>
> Regards,
> P.
>
> On Thu, Jul 19, 2012 at 9:09 AM, prasenjit mukherjee
> <pr...@gmail.com> wrote:
>> The probem could be while redistributing the tokens. In that case the
>> hashes has to be recalculated on each fo the candidate node.
>>
>> -Thanks,
>> Prasenjit
>>
>> On Thu, Jul 19, 2012 at 12:19 PM, Patrik Modesto
>> <pa...@gmail.com> wrote:
>>> Hi,
>>>
>>> I know that RandomPartitioner does MD5 of a key and the MD5 is then
>>> used for key distribution AND key ordering. I was just wondering if
>>> it's possible to have RandomPartitioner just for key distribution and
>>> OrderedPartitioner just for per-node key ordering. That would solve
>>> the often requested key scan feature.
>>>
>>> Regards,
>>> Patrik
Re: random partitioner and key scan
Posted by Patrik Modesto <pa...@gmail.com>.
Hi Prasenjit,
I don't see the need to recalculate anything. One key has a one MD5
hash, it doesn't change. Just use the hash to select a node, than just
the plain key. Can you elaborate on the redistribution please?
Regards,
P.
On Thu, Jul 19, 2012 at 9:09 AM, prasenjit mukherjee
<pr...@gmail.com> wrote:
> The probem could be while redistributing the tokens. In that case the
> hashes has to be recalculated on each fo the candidate node.
>
> -Thanks,
> Prasenjit
>
> On Thu, Jul 19, 2012 at 12:19 PM, Patrik Modesto
> <pa...@gmail.com> wrote:
>> Hi,
>>
>> I know that RandomPartitioner does MD5 of a key and the MD5 is then
>> used for key distribution AND key ordering. I was just wondering if
>> it's possible to have RandomPartitioner just for key distribution and
>> OrderedPartitioner just for per-node key ordering. That would solve
>> the often requested key scan feature.
>>
>> Regards,
>> Patrik
Re: random partitioner and key scan
Posted by prasenjit mukherjee <pr...@gmail.com>.
I agree.
The probem could be while redistributing the tokens. In that case the
hashes has to be recalculated on each fo the candidate node.
-Thanks,
Prasenjit
On Thu, Jul 19, 2012 at 12:19 PM, Patrik Modesto
<pa...@gmail.com> wrote:
> Hi,
>
> I know that RandomPartitioner does MD5 of a key and the MD5 is then
> used for key distribution AND key ordering. I was just wondering if
> it's possible to have RandomPartitioner just for key distribution and
> OrderedPartitioner just for per-node key ordering. That would solve
> the often requested key scan feature.
>
> Regards,
> Patrik