You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "John R. Frank" <jr...@mit.edu> on 2013/04/22 14:15:59 UTC
loading all rows from cassandra using multiple (python) clients in
parallel
Cassandra Experts,
I understand that when using Cassandra's recommended RandomPartitioner (or
Murmur3Partitioner), it is not possible to do meaningful range queries on
keys, because the rows are distributed around the cluster using the md5
hash of the key. These hashes are called "tokens."
Nonetheless, it would be very useful to split up a large table amongst
many compute workers by assigning each a range of tokens. Using CQL3, it
appears possible to issue queries directly against the tokens, however the
following python does not work:
http://stackoverflow.com/questions/16137944/loading-all-rows-from-cassandra-using-multiple-python-clients-in-parallel
I would ideally like to make this work with pycassa, because I prefer its
more pythonic interface.
Am I just not invoking CQL3 correctly through the cql package?
Is there a better way to do this?
Thanks for any pointers!
John
Re: loading all rows from cassandra using multiple (python) clients
in parallel
Posted by "John R. Frank" <jr...@mit.edu>.
On Wed, 24 Apr 2013, aaron morton wrote:
>
> EDIT: works after switching to testing against the lastest
> version of the cassandra database (doh!), and also updating the syntax
> per notes below:
>
> http://stackoverflow.com/questions/16137944/loading-all-rows-from-cassandra-using-multiple-python-clients-in-parallel
>
> Is this still a problem?
The exceptions raised when trying to use CQL v3.0.2 against an older
version of Cassandra could be improved. Once I figured that out, the
basic question was answered.
More importantly, it would be really helpful to expose this functionality
in pycassa so that we don't have to insert a chunk of code based on the
cql python module into an app that already uses pycassa everywhere else.
I have seen the posts about CQL2 versus CQL3 being an "implementation
choice" in pycassa. That may be true for much of the functionality that
pycassa exposes. It does not appear to be true if you consider all of the
functionality that is available from cassandra through other clients ---
for example, naming row keys and querying tokens directly (IIUC).
jrf
Re: loading all rows from cassandra using multiple (python) clients in parallel
Posted by aaron morton <aa...@thelastpickle.com>.
>
> EDIT: works after switching to testing against the lastest version of the cassandra database (doh!), and also updating the syntax per notes below:
http://stackoverflow.com/questions/16137944/loading-all-rows-from-cassandra-using-multiple-python-clients-in-parallel
Is this still a problem?
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 23/04/2013, at 12:15 AM, John R. Frank <jr...@mit.edu> wrote:
> Cassandra Experts,
>
> I understand that when using Cassandra's recommended RandomPartitioner (or Murmur3Partitioner), it is not possible to do meaningful range queries on keys, because the rows are distributed around the cluster using the md5 hash of the key. These hashes are called "tokens."
>
> Nonetheless, it would be very useful to split up a large table amongst many compute workers by assigning each a range of tokens. Using CQL3, it appears possible to issue queries directly against the tokens, however the following python does not work:
>
> http://stackoverflow.com/questions/16137944/loading-all-rows-from-cassandra-using-multiple-python-clients-in-parallel
>
> I would ideally like to make this work with pycassa, because I prefer its more pythonic interface.
>
> Am I just not invoking CQL3 correctly through the cql package?
>
> Is there a better way to do this?
>
>
> Thanks for any pointers!
>
> John
>
>
>
>