You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Paul Brown <pa...@gmail.com> on 2010/07/16 05:35:28 UTC
Re: Seeing very weird results on 0.6.2 when paginating through a ColumnFamily with get_slice()
You should make sure that your directions and interval endpoints are chosen correctly. I recall the semantics of the call being like an old-school for with the descending flag as a step of +1 or -1.
--
Spelling by mobile.
On Jul 15, 2010, at 20:19, Ilya Maykov <iv...@gmail.com> wrote:
> Hi all,
>
> I'm trying to debug some pretty weird behavior when paginating through
> a ColumnFamily with get_slice(). It basically looks like Cassandra
> does not respect the limit parameter in the SlicePredicate, sometimes
> returning more than limit columns. It also sometimes silently drops
> columns. I'm reading using QUORUM, and all data was written using
> QUORUM as well. The client is in Ruby.
>
> I am seeing basically non-deterministic behavior when paginating
> through a column family (which is not being written concurrently).
> Here is example output from my pagination method (code below) with
> some extra debug prints added:
>
> irb(main):005:0> blah = get_entire_column_family(@cassandra,
> "some_key", "some_cf", 100)
> get_entire_column_family(@cass, "some_key", "some_cf", 100) ...
> 100/6648 ... 199/6648 ... 354/6648 ... 453/6648 ... 552/6648 ...
> 689/6648 ... 788/6648 ... 887/6648 ... 1048/6648 ... 1147/6648 ...
> 1246/6648 ... 1377/6648 ... 1476/6648 ... 1575/6648 ... 1674/6648 ...
> 1773/6648 ... 1908/6648 ... 2051/6648 ... 2150/6648 ... 2249/6648 ...
> 2348/6648 ... <snip> ... 6127/6648 ... 6127/6648 ... 6127/6648 ...
> 6127/6648 ... 6127/6648 ...
>
> The N/6648 is just printing the retrieved columns / total columns at
> each step of the pagination loop. It should be going up by 99 on each
> iteration after the first (because the start of the next slice == the
> last value in the current slice). But sometimes it jumps, indicating
> that more than 100 values were returned from a single get_slice() call
> (i.e. ... 199/6648 ... 354/6648 ...). And when it gets to the end of
> the column family, we end up with fewer than 6648 columns on the
> client side and the code gets stuck in an infinite loop.
>
> I've tried this several times from an interactive Ruby session and got
> a different number of columns each time:
> 6536/6648
> 6127/6648
> 6514/6648
> However, once I set the limit to be > num_columns and read the entire
> row as a single page, everything worked. And follow-up paginated reads
> also return the entire row successfully. Not sure if that's because
> the entire row is now in cache, or because something was wrong and
> read-repair has fixed it. But since all of our reads and writes are
> done using QUORUM, read-repair shouldn't matter, right?
>
> Here is the pagination code:
>
> def get_entire_column_family(cassandra, row_key, column_family,
> limit_per_slice)
> column_parent = CassandraThrift::ColumnParent.new(:column_family
> => column_family, :super_column => nil)
> num_columns = cassandra.get_count(@keyspace, row_key,
> column_parent, CassandraThrift::ConsistencyLevel::QUORUM)
> predicate = CassandraThrift::SlicePredicate.new
> predicate.slice_range = CassandraThrift::SliceRange.new(:start
> => "", :finish => "", :reversed => false, :count => limit_per_slice)
> slice = cassandra.get_slice(@keyspace, row_key, column_parent,
> predicate, CassandraThrift::ConsistencyLevel::QUORUM)
> result = slice
> while result.size < num_columns
> predicate = CassandraThrift::SlicePredicate.new
> predicate.slice_range = CassandraThrift::SliceRange.new(:start
> => result.last.column.name,
> :finish => "", :reversed => false, :count => limit_per_slice)
> slice = cassandra.get_slice(@keyspace, row_key, column_parent,
> predicate, CassandraThrift::ConsistencyLevel::QUORUM)
> # Because the start parameter to get_slice() is inclusive, we
> should already have the first column of the
> # new slice in our result. We don't want to have 2 copies of
> it, so drop it from the slice before concatenating.
> unless slice.nil? || slice.empty?
> if result.last.column.name == slice.first.column.name
> result.concat slice[1 .. slice.size-1]
> else
> result.concat slice
> end
> end
> end # while
> return result
> end
>
> I guess I have several questions:
> 1) Is this the proper way to paginate through a large column family
> for a single row key? If not, what is the proper way? Some of our rows
> are very big (hundreds of thousands of columns in the worst case), and
> pagination is a must.
> 2) Could this behavior be expected under some conditions (i.e. maybe
> presence of tombstones or hints from when a node was down or other
> weirdness?)
> 2) Is this a known bug? (maybe related to
> https://issues.apache.org/jira/browse/CASSANDRA-1145 and/or
> https://issues.apache.org/jira/browse/CASSANDRA-1042 ?)
> 3) If this is not a known bug, how should I proceed with investigating it?
>
> Thanks,
>
> -- Ilya
Re: Seeing very weird results on 0.6.2 when paginating through a
ColumnFamily with get_slice()
Posted by Ilya Maykov <iv...@gmail.com>.
The column names are arbitrary strings, so it's not obvious what the
"next" value should be at any step. So, I just set the start of the
next page to the end of the last page and eliminate the duplicate
value when joining the 2 pages together.
The paging direction does not matter in my case, as I just want to get
the entire column family. And in the vast majority of cases (probably
99%+) the paginated get method returns the correct results. But
sometimes it does not.
-- Ilya
On Thu, Jul 15, 2010 at 8:35 PM, Paul Brown <pa...@gmail.com> wrote:
>
> You should make sure that your directions and interval endpoints are chosen correctly. I recall the semantics of the call being like an old-school for with the descending flag as a step of +1 or -1.
>
> --
> Spelling by mobile.
>
> On Jul 15, 2010, at 20:19, Ilya Maykov <iv...@gmail.com> wrote:
>
>> Hi all,
>>
>> I'm trying to debug some pretty weird behavior when paginating through
>> a ColumnFamily with get_slice(). It basically looks like Cassandra
>> does not respect the limit parameter in the SlicePredicate, sometimes
>> returning more than limit columns. It also sometimes silently drops
>> columns. I'm reading using QUORUM, and all data was written using
>> QUORUM as well. The client is in Ruby.
>>
>> I am seeing basically non-deterministic behavior when paginating
>> through a column family (which is not being written concurrently).
>> Here is example output from my pagination method (code below) with
>> some extra debug prints added:
>>
>> irb(main):005:0> blah = get_entire_column_family(@cassandra,
>> "some_key", "some_cf", 100)
>> get_entire_column_family(@cass, "some_key", "some_cf", 100) ...
>> 100/6648 ... 199/6648 ... 354/6648 ... 453/6648 ... 552/6648 ...
>> 689/6648 ... 788/6648 ... 887/6648 ... 1048/6648 ... 1147/6648 ...
>> 1246/6648 ... 1377/6648 ... 1476/6648 ... 1575/6648 ... 1674/6648 ...
>> 1773/6648 ... 1908/6648 ... 2051/6648 ... 2150/6648 ... 2249/6648 ...
>> 2348/6648 ... <snip> ... 6127/6648 ... 6127/6648 ... 6127/6648 ...
>> 6127/6648 ... 6127/6648 ...
>>
>> The N/6648 is just printing the retrieved columns / total columns at
>> each step of the pagination loop. It should be going up by 99 on each
>> iteration after the first (because the start of the next slice == the
>> last value in the current slice). But sometimes it jumps, indicating
>> that more than 100 values were returned from a single get_slice() call
>> (i.e. ... 199/6648 ... 354/6648 ...). And when it gets to the end of
>> the column family, we end up with fewer than 6648 columns on the
>> client side and the code gets stuck in an infinite loop.
>>
>> I've tried this several times from an interactive Ruby session and got
>> a different number of columns each time:
>> 6536/6648
>> 6127/6648
>> 6514/6648
>> However, once I set the limit to be > num_columns and read the entire
>> row as a single page, everything worked. And follow-up paginated reads
>> also return the entire row successfully. Not sure if that's because
>> the entire row is now in cache, or because something was wrong and
>> read-repair has fixed it. But since all of our reads and writes are
>> done using QUORUM, read-repair shouldn't matter, right?
>>
>> Here is the pagination code:
>>
>> def get_entire_column_family(cassandra, row_key, column_family,
>> limit_per_slice)
>> column_parent = CassandraThrift::ColumnParent.new(:column_family
>> => column_family, :super_column => nil)
>> num_columns = cassandra.get_count(@keyspace, row_key,
>> column_parent, CassandraThrift::ConsistencyLevel::QUORUM)
>> predicate = CassandraThrift::SlicePredicate.new
>> predicate.slice_range = CassandraThrift::SliceRange.new(:start
>> => "", :finish => "", :reversed => false, :count => limit_per_slice)
>> slice = cassandra.get_slice(@keyspace, row_key, column_parent,
>> predicate, CassandraThrift::ConsistencyLevel::QUORUM)
>> result = slice
>> while result.size < num_columns
>> predicate = CassandraThrift::SlicePredicate.new
>> predicate.slice_range = CassandraThrift::SliceRange.new(:start
>> => result.last.column.name,
>> :finish => "", :reversed => false, :count => limit_per_slice)
>> slice = cassandra.get_slice(@keyspace, row_key, column_parent,
>> predicate, CassandraThrift::ConsistencyLevel::QUORUM)
>> # Because the start parameter to get_slice() is inclusive, we
>> should already have the first column of the
>> # new slice in our result. We don't want to have 2 copies of
>> it, so drop it from the slice before concatenating.
>> unless slice.nil? || slice.empty?
>> if result.last.column.name == slice.first.column.name
>> result.concat slice[1 .. slice.size-1]
>> else
>> result.concat slice
>> end
>> end
>> end # while
>> return result
>> end
>>
>> I guess I have several questions:
>> 1) Is this the proper way to paginate through a large column family
>> for a single row key? If not, what is the proper way? Some of our rows
>> are very big (hundreds of thousands of columns in the worst case), and
>> pagination is a must.
>> 2) Could this behavior be expected under some conditions (i.e. maybe
>> presence of tombstones or hints from when a node was down or other
>> weirdness?)
>> 2) Is this a known bug? (maybe related to
>> https://issues.apache.org/jira/browse/CASSANDRA-1145 and/or
>> https://issues.apache.org/jira/browse/CASSANDRA-1042 ?)
>> 3) If this is not a known bug, how should I proceed with investigating it?
>>
>> Thanks,
>>
>> -- Ilya
>