How to retrieve keys from Cassandra ?

[Sébastien Pierre <se...@gmail.com>]

Hi all,

I would like to know how to retrieve the list of available keys available
for a specific column. There is the get_key_range method, but it is only
available when using the OrderPreservingPartitioner -- I use a
RandomPartitioner.

Does this mean that when using a RandomPartitioner, you cannot see which
keys are available in the database ?

 -- Sébastien

Re: How to retrieve keys from Cassandra ?

[Jonathan Ellis <jb...@gmail.com>]

On Tue, Feb 2, 2010 at 12:51 PM, Jean-Denis Greze <je...@6coders.com> wrote:
> Anyway, partially to address the efficiency concern, I've been playing
> around with the idea of having 745-like functionality on a per-node basis: a
> call to get all of the keys on a particular node as opposed to the entire
> cluster.  It just seems like with a very large cluster with billions, tens
> of billions, or hundreds of billions of keys 745 would just get overwhelmed.

That's why 745 is really there for hadoop support
(https://issues.apache.org/jira/browse/CASSANDRA-342), not something
intended to be used manually.

-Jonathan

Re: How to retrieve keys from Cassandra ?

[Jean-Denis Greze <je...@6coders.com>]

Ok, so 0.6's https://issues.apache.org/jira/browse/CASSANDRA-745 permits
"someone using RandomPartitioner to pass start="" and finish="" to get all
of the rows in their cluster, although in an extremely inefficient way."

We are in a situation like Pierre's, where we need to know what's currently
in the DB so to speak -- except that we have a hundreds of millions of rows
(and increasing) and that maintaining an index of the keys in another CF, as
Brandon suggests, is becoming difficult (we also don't like the double write
on initial key inserts, in terms of transactionality especially).

Also, every once in a while, we need to enhance our data as part of some
functionality upgrade or refactoring.  So far, what we do is enhance on
reads (i.e., whenever we read a particular record, see if it's not up to the
latest version, and if so enhance), but there are many problems with this
approach. We've been considering doing background process enhancing by
running through all of the keys, which is why 745 is pretty exciting.  We'd
rather go through the inefficient operation once in a while as opposed to
doing a check on every read.

Anyway, partially to address the efficiency concern, I've been playing
around with the idea of having 745-like functionality on a per-node basis: a
call to get all of the keys on a particular node as opposed to the entire
cluster.  It just seems like with a very large cluster with billions, tens
of billions, or hundreds of billions of keys 745 would just get overwhelmed.
 Just a thought.







On Tue, Feb 2, 2010 at 7:31 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>
> More or less (but see
> https://issues.apache.org/jira/browse/CASSANDRA-745, in 0.6).
>
> Think of it this way: when you have a few billion keys, how useful is
> it to list them?
>
> -Jonathan
>
> 2010/2/2 Sébastien Pierre <se...@gmail.com>:
> > Hi all,
> > I would like to know how to retrieve the list of available keys
available
> > for a specific column. There is the get_key_range method, but it is only
> > available when using the OrderPreservingPartitioner -- I use a
> > RandomPartitioner.
> > Does this mean that when using a RandomPartitioner, you cannot see which
> > keys are available in the database ?
> >  -- Sébastien



--
jeandenis@6coders.com
(917) 951-0636

This email and any files transmitted with it are confidential and intended
solely for the use of the individual to whom they are addressed. If you have
received this email in error please notify the system manager. This message
contains confidential information and is intended only for the individual
named. If you are not the named addressee you should not disseminate,
distribute or copy this e-mail. Please notify the sender immediately by
e-mail if you have received this e-mail by mistake and delete this e-mail
from your system. If you are not the intended recipient you are notified
that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.

Re: How to retrieve keys from Cassandra ?

[Sébastien Pierre <se...@gmail.com>]

Yes, that's a good idea ! I've considered doing that at some point, but I'm
still learning the basics ;)

 -- Sébastien

2010/2/2 Brandon Williams <dr...@gmail.com>

> 2010/2/2 Sébastien Pierre <se...@gmail.com>
>
>> Hi Jonathan,
>>
>> In my case, I'll have much more columns (thousands to millions) than keys
>> in logs (campaign x days), so it's not an issue to retrieve all of them.
>>
>
> If that's the case, your dataset is small enough that you could maintain an
> index of the keys in another CF.   If it needs to scale further, you can
> segment the index keys by year, month, etc.
>
> -Brandon
>

Re: How to retrieve keys from Cassandra ?

[Brandon Williams <dr...@gmail.com>]

2010/2/2 Sébastien Pierre <se...@gmail.com>

> Hi Jonathan,
>
> In my case, I'll have much more columns (thousands to millions) than keys
> in logs (campaign x days), so it's not an issue to retrieve all of them.
>

If that's the case, your dataset is small enough that you could maintain an
index of the keys in another CF.   If it needs to scale further, you can
segment the index keys by year, month, etc.

-Brandon

Re: How to retrieve keys from Cassandra ?

[Sébastien Pierre <se...@gmail.com>]

Hi Jonathan,

In my case, I'll have much more columns (thousands to millions) than keys in
logs (campaign x days), so it's not an issue to retrieve all of them.

Also, if you assume that you can't retrieve values from Cassandra, just
because you're using the wrong key (say your using "user/10" instead of
"user:10") without the ability to list the keys, you'd have no way to find
out the error.

I'm glad to see this implemented :)

 -- Sébastien

2010/2/2 Jonathan Ellis <jb...@gmail.com>

> More or less (but see
> https://issues.apache.org/jira/browse/CASSANDRA-745, in 0.6).
>
> Think of it this way: when you have a few billion keys, how useful is
> it to list them?
>
> -Jonathan
>
> 2010/2/2 Sébastien Pierre <se...@gmail.com>:
> > Hi all,
> > I would like to know how to retrieve the list of available keys available
> > for a specific column. There is the get_key_range method, but it is only
> > available when using the OrderPreservingPartitioner -- I use a
> > RandomPartitioner.
> > Does this mean that when using a RandomPartitioner, you cannot see which
> > keys are available in the database ?
> >  -- Sébastien
>

Re: How to retrieve keys from Cassandra ?

[Jonathan Ellis <jb...@gmail.com>]

More or less (but see
https://issues.apache.org/jira/browse/CASSANDRA-745, in 0.6).

Think of it this way: when you have a few billion keys, how useful is
it to list them?

-Jonathan

2010/2/2 Sébastien Pierre <se...@gmail.com>:
> Hi all,
> I would like to know how to retrieve the list of available keys available
> for a specific column. There is the get_key_range method, but it is only
> available when using the OrderPreservingPartitioner -- I use a
> RandomPartitioner.
> Does this mean that when using a RandomPartitioner, you cannot see which
> keys are available in the database ?
>  -- Sébastien

Re: How to retrieve keys from Cassandra ?

[Sébastien Pierre <se...@gmail.com>]

Hi all,

It's basically for "knowing what's inside the db", as I've been toying with
Cassandra for some time, I have keys that are no longer useful and should be
removed.

I'm also storing HTTP logs in cassandra, where keys follow this convention
"campaign:<CAMPAIGN_ID>:<YYYYMMDD>". So for instance, if I'd like to know
what logs are available I just have to do:

   client.get_keys("Keyspace1", "Logs", "", "", 100, ConsistencyLevel.ONE)

However, I have to use an OrderPreservingPartitioner to do so, which is
(from my understanding) bad for load in this case.

 -- Sébastien


2010/2/2 Erik Holstad <er...@gmail.com>

> Hi Sebastien!
> I'm totally new to Cassandra, but as far as I know there is no way of
> getting just the keys that are in the
> database, they are not stored separately but only with the data itself.
>
> Why do you want a list of keys, what are you going to use them for? Maybe
> there is another way of solving
> your problem.
>
> What you are describing, getting all the keys/rows for a given column
> sounds like you have to fetch all the
> data that you have and then filter every key on your column, I don't think
> that get_key_range will do that for
> you even, says that it takes column_family, but like I said I'm totally new
>
> Erik
>
> 2010/2/2 Sébastien Pierre <se...@gmail.com>
>
>> Hi all,
>>
>> I would like to know how to retrieve the list of available keys available
>> for a specific column. There is the get_key_range method, but it is only
>> available when using the OrderPreservingPartitioner -- I use a
>> RandomPartitioner.
>>
>> Does this mean that when using a RandomPartitioner, you cannot see which
>> keys are available in the database ?
>>
>>  -- Sébastien
>>
>
>
>
> --
> Regards Erik
>

Re: How to retrieve keys from Cassandra ?

[Erik Holstad <er...@gmail.com>]

Hi Sebastien!
I'm totally new to Cassandra, but as far as I know there is no way of
getting just the keys that are in the
database, they are not stored separately but only with the data itself.

Why do you want a list of keys, what are you going to use them for? Maybe
there is another way of solving
your problem.

What you are describing, getting all the keys/rows for a given column sounds
like you have to fetch all the
data that you have and then filter every key on your column, I don't think
that get_key_range will do that for
you even, says that it takes column_family, but like I said I'm totally new

Erik

2010/2/2 Sébastien Pierre <se...@gmail.com>

> Hi all,
>
> I would like to know how to retrieve the list of available keys available
> for a specific column. There is the get_key_range method, but it is only
> available when using the OrderPreservingPartitioner -- I use a
> RandomPartitioner.
>
> Does this mean that when using a RandomPartitioner, you cannot see which
> keys are available in the database ?
>
>  -- Sébastien
>



-- 
Regards Erik