pagination through slices with deleted keys

[Ian Kallen <sp...@gmail.com>]

I read the DistributedDeletes and the range_ghosts FAQ entry on the wiki
which do a good job describing how difficult deletion is in an eventually
consistent system. But practical application strategies for dealing with it
aren't there (that I saw). I'm wondering how folks implement pagination in
their applications; if you want to render N results in an application, is
the only solution to over-fetch and filter out the tombstones? Or is there
something simpler that I overlooked? I'd like to be able to count (even if
the counts are approximate) and fetch rows with the deleted ones filtered
out (without waiting for the GCGraceSeconds interval + compaction) but from
what I see so far, the burden is on the app to deal with the tombstones.
-Ian

Re: pagination through slices with deleted keys

[Ian Kallen <sp...@gmail.com>]

Thanks Mark, great illustration. I'm already splitting my time developing
directly with hector and a vastly simplified jython wrapper around it; I
guess I'll address it at some wrapping layer (patch hector or let the jython
layer deal).

My grumpy editorial about this stuff is that on the cassandra server, the
Column has an isMarkedForDelete():boolean and the thrift (+avro) API could
(should) expose the ability to filter the true cases. It seems pretty grotty
to allow such low level data bookkeeping to leak into the application space;
analogous to the ConsistencyLevel and timestamp stuff that most folks don't
need on the app layer, slicing via the service API should do the filtering
unless your abstraction actually *needs* the tombstones. OK, done being
grumpy :)
-Ian

On Thu, May 6, 2010 at 9:26 AM, Mark Greene <gr...@gmail.com> wrote:

> Hey Ian,
>
> I actually just wrote a quick example of how to iterate over a CF that may
> have tombstones. This may help you out:
> http://markjgreene.wordpress.com/2010/05/05/iterate-over-entire-cassandra-column-family/
>
>
> On Thu, May 6, 2010 at 12:17 PM, Ian Kallen <sp...@gmail.com>wrote:
>
>> I read the DistributedDeletes and the range_ghosts FAQ entry on the wiki
>> which do a good job describing how difficult deletion is in an eventually
>> consistent system. But practical application strategies for dealing with it
>> aren't there (that I saw). I'm wondering how folks implement pagination in
>> their applications; if you want to render N results in an application, is
>> the only solution to over-fetch and filter out the tombstones? Or is there
>> something simpler that I overlooked? I'd like to be able to count (even if
>> the counts are approximate) and fetch rows with the deleted ones filtered
>> out (without waiting for the GCGraceSeconds interval + compaction) but from
>> what I see so far, the burden is on the app to deal with the tombstones.
>>  -Ian
>>
>
>

Re: pagination through slices with deleted keys

[Mark Greene <gr...@gmail.com>]

Hey Ian,

I actually just wrote a quick example of how to iterate over a CF that may
have tombstones. This may help you out:
http://markjgreene.wordpress.com/2010/05/05/iterate-over-entire-cassandra-column-family/

On Thu, May 6, 2010 at 12:17 PM, Ian Kallen <sp...@gmail.com> wrote:

> I read the DistributedDeletes and the range_ghosts FAQ entry on the wiki
> which do a good job describing how difficult deletion is in an eventually
> consistent system. But practical application strategies for dealing with it
> aren't there (that I saw). I'm wondering how folks implement pagination in
> their applications; if you want to render N results in an application, is
> the only solution to over-fetch and filter out the tombstones? Or is there
> something simpler that I overlooked? I'd like to be able to count (even if
> the counts are approximate) and fetch rows with the deleted ones filtered
> out (without waiting for the GCGraceSeconds interval + compaction) but from
> what I see so far, the burden is on the app to deal with the tombstones.
>  -Ian
>

Re: pagination through slices with deleted keys

[Mike Malone <mi...@simplegeo.com>]

On Fri, May 7, 2010 at 5:29 AM, Joost Ouwerkerk <jo...@openplaces.org>wrote:

> +1.  There is some disagreement on whether or not the API should
> return empty columns or skip rows when no data is found.  In all of
> our use cases, we would prefer skipped rows.  And based on how
> frequently new cassandra users appear to be confused about the current
> behaviour, this might be a more common use case than the need for
> empty cols.  Perhaps this could be added as an option on
> SlicePredicate ?  (e.g. skipEmpty=true).
>

That's exactly how we implemented it:

struct SlicePredicate {
    1: optional list<binary> column_names,
    2: optional SliceRange   slice_range,
    3: optional bool ignore_empty_rows=0,
}

Mike

Re: pagination through slices with deleted keys

[Mark Greene <gr...@gmail.com>]

I like your idea about specifying it at the SP level.

On Fri, May 7, 2010 at 8:29 AM, Joost Ouwerkerk <jo...@openplaces.org>wrote:

> +1.  There is some disagreement on whether or not the API should
> return empty columns or skip rows when no data is found.  In all of
> our use cases, we would prefer skipped rows.  And based on how
> frequently new cassandra users appear to be confused about the current
> behaviour, this might be a more common use case than the need for
> empty cols.  Perhaps this could be added as an option on
> SlicePredicate ?  (e.g. skipEmpty=true).
>
> On Fri, May 7, 2010 at 12:59 AM, Mike Malone <mi...@simplegeo.com> wrote:
> > On Thu, May 6, 2010 at 3:27 PM, Ian Kallen <sp...@gmail.com>
> wrote:
> >>
> >> Cool, is this a patch you've applied on the server side? Are you running
> >> 0.6.x? I'm wondering if this kind of thing can make it into future
> versions
> >> of Cassandra.
> >
> > Yea, server side. It's basically doing the same thing clients typically
> want
> > to do (again, at least for our use cases) but doing it closer to the
> data.
> > Our patch is kind of janky though. I can probably get some version of it
> > pushed back upstream - or at least on github or something - if there's
> any
> > interest.
> > Mike
>

Re: pagination through slices with deleted keys

[Joost Ouwerkerk <jo...@openplaces.org>]

+1.  There is some disagreement on whether or not the API should
return empty columns or skip rows when no data is found.  In all of
our use cases, we would prefer skipped rows.  And based on how
frequently new cassandra users appear to be confused about the current
behaviour, this might be a more common use case than the need for
empty cols.  Perhaps this could be added as an option on
SlicePredicate ?  (e.g. skipEmpty=true).

On Fri, May 7, 2010 at 12:59 AM, Mike Malone <mi...@simplegeo.com> wrote:
> On Thu, May 6, 2010 at 3:27 PM, Ian Kallen <sp...@gmail.com> wrote:
>>
>> Cool, is this a patch you've applied on the server side? Are you running
>> 0.6.x? I'm wondering if this kind of thing can make it into future versions
>> of Cassandra.
>
> Yea, server side. It's basically doing the same thing clients typically want
> to do (again, at least for our use cases) but doing it closer to the data.
> Our patch is kind of janky though. I can probably get some version of it
> pushed back upstream - or at least on github or something - if there's any
> interest.
> Mike

Re: pagination through slices with deleted keys

[Mike Malone <mi...@simplegeo.com>]

On Thu, May 6, 2010 at 3:27 PM, Ian Kallen <sp...@gmail.com> wrote:

> Cool, is this a patch you've applied on the server side? Are you running
> 0.6.x? I'm wondering if this kind of thing can make it into future versions
> of Cassandra.
>

Yea, server side. It's basically doing the same thing clients typically want
to do (again, at least for our use cases) but doing it closer to the data.
Our patch is kind of janky though. I can probably get some version of it
pushed back upstream - or at least on github or something - if there's any
interest.

Mike

Re: pagination through slices with deleted keys

[Ian Kallen <sp...@gmail.com>]

Cool, is this a patch you've applied on the server side? Are you running
0.6.x? I'm wondering if this kind of thing can make it into future versions
of Cassandra.
-Ian

On Thu, May 6, 2010 at 2:56 PM, Mike Malone <mi...@simplegeo.com> wrote:

> Our solution at SimpleGeo has been to hack Cassandra to (optionally, at
> least) be sensible and drop Rows that don't have any Columns. The claim from
> the FAQ that "Cassandra would have to check if there are any other columns
> in the row" is inaccurate. The common case for us at least is that we're
> only interested in Rows that have Columns matching our predicate. So if
> there aren't any, we just don't return that row. No need to check if the
> entire row is deleted.
>
> Mike
>
>
> On Thu, May 6, 2010 at 9:17 AM, Ian Kallen <sp...@gmail.com>wrote:
>
>> I read the DistributedDeletes and the range_ghosts FAQ entry on the wiki
>> which do a good job describing how difficult deletion is in an eventually
>> consistent system. But practical application strategies for dealing with it
>> aren't there (that I saw). I'm wondering how folks implement pagination in
>> their applications; if you want to render N results in an application, is
>> the only solution to over-fetch and filter out the tombstones? Or is there
>> something simpler that I overlooked? I'd like to be able to count (even if
>> the counts are approximate) and fetch rows with the deleted ones filtered
>> out (without waiting for the GCGraceSeconds interval + compaction) but from
>> what I see so far, the burden is on the app to deal with the tombstones.
>>  -Ian
>>
>
>

Re: pagination through slices with deleted keys

[Mike Malone <mi...@simplegeo.com>]

Our solution at SimpleGeo has been to hack Cassandra to (optionally, at
least) be sensible and drop Rows that don't have any Columns. The claim from
the FAQ that "Cassandra would have to check if there are any other columns
in the row" is inaccurate. The common case for us at least is that we're
only interested in Rows that have Columns matching our predicate. So if
there aren't any, we just don't return that row. No need to check if the
entire row is deleted.

Mike

On Thu, May 6, 2010 at 9:17 AM, Ian Kallen <sp...@gmail.com> wrote:

> I read the DistributedDeletes and the range_ghosts FAQ entry on the wiki
> which do a good job describing how difficult deletion is in an eventually
> consistent system. But practical application strategies for dealing with it
> aren't there (that I saw). I'm wondering how folks implement pagination in
> their applications; if you want to render N results in an application, is
> the only solution to over-fetch and filter out the tombstones? Or is there
> something simpler that I overlooked? I'd like to be able to count (even if
> the counts are approximate) and fetch rows with the deleted ones filtered
> out (without waiting for the GCGraceSeconds interval + compaction) but from
> what I see so far, the burden is on the app to deal with the tombstones.
> -Ian
>