ordered partitioner

["Desimpel, Ignace" <Ig...@nuance.com>]

Hi,

I was trying to implement my own ordered partitioner and got into problems.
The current DecoratedKey is using a ByteBufferUtil.compareUnsigned for comparing the key. I was thinking of having a signed comparison, so I thought of making my own DecoratedKey, Token and Partitioner. That way I would have complete control...
So  made a partitioner whith a function decorateKey(...) returning MyDecoratedKey in stead of DecoratedKey
But when making my own MyDecoratedKey, the database get into trouble when adding a key space due to the fact that some code in Cassandra is using the 'new DecoratedKey(...)' statement and is not using the partitioner function decorateKey(...).

Would it be logical to always call the partitioner function decorateKey such that the creation of an own partitioner and key decoration is possible?

Ignace Desimpel

RE: ordered partitioner

["Desimpel, Ignace" <Ig...@nuance.com>]

I got into problems on starting a new database. That starts up ok. Then I add a keyspace and it goes wrong. The error came from DefsTable.mergeSchema (working on version 1.2.1). It starts by fetching the current keyspace and stores it in a map of DecoratedKey elements. Then it applies the mutation and fetches all the keyspaces again. But now the fetch gets a map of the new OwnDecoratedKey elements. So the difference of these elements contains all the 'old' DecoratedKey and the 'new' OwnDecoratedKey elements, each kind of DecoratedKey contains all the old existing keyspaces. So the the mergeKeyspaces routine thinks that it has to recreate the already existing keyspaces like system_auth , etc...

Currently in my schema, I don't use indexes, so don't know the effect. Also, the change (local of course) I did, was done without really knowing what to do (kind of blind replacement) ...
But the one reference in SecondaryIndex was left as it is, since it was using a LocalToken object.

What I changed to make it work is :
------------------------------------------
1) Function Memtable:: resolve(DecoratedKey key, ColumnFamily cf, SecondaryIndexManager.Updater indexer)
  With line "previous = columnFamilies.putIfAbsent(new DecoratedKey(key.token, allocator.clone(key.key)), empty);"
  Changed to "previous = columnFamilies.putIfAbsent(StorageService.getPartitioner().decorateKey(allocator.clone(key.key)), empty);"
2) Function SSTable:: getMinimalKey(DecoratedKey key)
  With line " new DecoratedKey(key.token, HeapAllocator.instance.clone(key.key))"
  Changed to " StorageService.getPartitioner().decorateKey(HeapAllocator.instance.clone(key.key)) "
3) ColumnFamilyStore:: getSSTablesForKey(String key)
  With line "DecoratedKey dk = new DecoratedKey(partitioner.getToken(ByteBuffer.wrap(key.getBytes())), ByteBuffer.wrap(key.getBytes()));"
  Changed to DecoratedKey dk = partitioner.decorateKey(ByteBuffer.wrap(key.getBytes()));


Regards,

Ignace

-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com] 
Sent: maandag 22 april 2013 20:12
To: user
Subject: Re: ordered partitioner

Not in general, no.  There are places, like indexing, that need to use a local partitioner rather than the global one.

Which uses of the DK constructor looked erroneous to you?

On Mon, Apr 22, 2013 at 10:54 AM, Desimpel, Ignace <Ig...@nuance.com> wrote:
> Hi,
>
>
>
> I was trying to implement my own ordered partitioner and got into problems.
>
> The current DecoratedKey is using a ByteBufferUtil.compareUnsigned for 
> comparing the key. I was thinking of having a signed comparison, so I 
> thought of making my own DecoratedKey, Token and Partitioner. That way 
> I would have complete control...
>
> So  made a partitioner whith a function decorateKey(...) returning 
> MyDecoratedKey in stead of DecoratedKey
>
> But when making my own MyDecoratedKey, the database get into trouble 
> when adding a key space due to the fact that some code in Cassandra is 
> using the 'new DecoratedKey(...)' statement and is not using the 
> partitioner function decorateKey(...).
>
>
>
> Would it be logical to always call the partitioner function 
> decorateKey such that the creation of an own partitioner and key decoration is possible?
>
>
>
> Ignace Desimpel
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

Re: ordered partitioner

[Jonathan Ellis <jb...@gmail.com>]

Not in general, no.  There are places, like indexing, that need to use
a local partitioner rather than the global one.

Which uses of the DK constructor looked erroneous to you?

On Mon, Apr 22, 2013 at 10:54 AM, Desimpel, Ignace
<Ig...@nuance.com> wrote:
> Hi,
>
>
>
> I was trying to implement my own ordered partitioner and got into problems.
>
> The current DecoratedKey is using a ByteBufferUtil.compareUnsigned for
> comparing the key. I was thinking of having a signed comparison, so I
> thought of making my own DecoratedKey, Token and Partitioner. That way I
> would have complete control…
>
> So  made a partitioner whith a function decorateKey(…) returning
> MyDecoratedKey in stead of DecoratedKey
>
> But when making my own MyDecoratedKey, the database get into trouble when
> adding a key space due to the fact that some code in Cassandra is using the
> ‘new DecoratedKey(…)’ statement and is not using the partitioner function
> decorateKey(…).
>
>
>
> Would it be logical to always call the partitioner function decorateKey such
> that the creation of an own partitioner and key decoration is possible?
>
>
>
> Ignace Desimpel
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced