You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "John R. Frank" <jr...@mit.edu> on 2013/04/22 14:15:59 UTC

loading all rows from cassandra using multiple (python) clients in parallel

Cassandra Experts,

I understand that when using Cassandra's recommended RandomPartitioner (or 
Murmur3Partitioner), it is not possible to do meaningful range queries on 
keys, because the rows are distributed around the cluster using the md5 
hash of the key.  These hashes are called "tokens."

Nonetheless, it would be very useful to split up a large table amongst 
many compute workers by assigning each a range of tokens.  Using CQL3, it 
appears possible to issue queries directly against the tokens, however the 
following python does not work:

http://stackoverflow.com/questions/16137944/loading-all-rows-from-cassandra-using-multiple-python-clients-in-parallel

I would ideally like to make this work with pycassa, because I prefer its 
more pythonic interface.

Am I just not invoking CQL3 correctly through the cql package?

Is there a better way to do this?


Thanks for any pointers!

John





Re: loading all rows from cassandra using multiple (python) clients in parallel

Posted by "John R. Frank" <jr...@mit.edu>.
On Wed, 24 Apr 2013, aaron morton wrote:
>
>        EDIT: works after switching to testing against the lastest 
> version of the cassandra database (doh!), and also updating the syntax 
> per notes below:
> 
> http://stackoverflow.com/questions/16137944/loading-all-rows-from-cassandra-using-multiple-python-clients-in-parallel
> 
> Is this still a problem?


The exceptions raised when trying to use CQL v3.0.2 against an older 
version of Cassandra could be improved.  Once I figured that out, the 
basic question was answered.

More importantly, it would be really helpful to expose this functionality 
in pycassa so that we don't have to insert a chunk of code based on the 
cql python module into an app that already uses pycassa everywhere else.

I have seen the posts about CQL2 versus CQL3 being an "implementation 
choice" in pycassa.  That may be true for much of the functionality that 
pycassa exposes.  It does not appear to be true if you consider all of the 
functionality that is available from cassandra through other clients --- 
for example, naming row keys and querying tokens directly (IIUC).


jrf

Re: loading all rows from cassandra using multiple (python) clients in parallel

Posted by aaron morton <aa...@thelastpickle.com>.
> 
>  EDIT: works after switching to testing against the lastest version of the cassandra database (doh!), and also updating the syntax per notes below:
http://stackoverflow.com/questions/16137944/loading-all-rows-from-cassandra-using-multiple-python-clients-in-parallel

Is this still a problem?

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 23/04/2013, at 12:15 AM, John R. Frank <jr...@mit.edu> wrote:

> Cassandra Experts,
> 
> I understand that when using Cassandra's recommended RandomPartitioner (or Murmur3Partitioner), it is not possible to do meaningful range queries on keys, because the rows are distributed around the cluster using the md5 hash of the key.  These hashes are called "tokens."
> 
> Nonetheless, it would be very useful to split up a large table amongst many compute workers by assigning each a range of tokens.  Using CQL3, it appears possible to issue queries directly against the tokens, however the following python does not work:
> 
> http://stackoverflow.com/questions/16137944/loading-all-rows-from-cassandra-using-multiple-python-clients-in-parallel
> 
> I would ideally like to make this work with pycassa, because I prefer its more pythonic interface.
> 
> Am I just not invoking CQL3 correctly through the cql package?
> 
> Is there a better way to do this?
> 
> 
> Thanks for any pointers!
> 
> John
> 
> 
> 
>