You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Hayarobi Park <ha...@gmail.com> on 2010/11/17 09:03:59 UTC
Count of SliceRange in get_slice seems not to work
Hello.
I'm using cassandra (currently 0.7.0-beta3) in JAVA; with library
hector.
It seems that cassandra ignore the count of SliceRange when received
get_slice request.
I traced cassandra source code, and the part of code that retrieving
columns does not get count as parameter. See,
getSlice(List<ReadCommand> commands, ConsistencyLevel consistency_level)
method in org.apache.cassandra.thrift.CassandraServer class. (line
224~238 in 0.7.0-beta3)
private Map<ByteBuffer, List<ColumnOrSuperColumn>>
getSlice(List<ReadCommand> commands, ConsistencyLevel consistency_level)
throws InvalidRequestException, UnavailableException,
TimedOutException
{
Map<DecoratedKey, ColumnFamily> columnFamilies =
readColumnFamily(commands, consistency_level);
Map<ByteBuffer, List<ColumnOrSuperColumn>> columnFamiliesMap =
new HashMap<ByteBuffer, List<ColumnOrSuperColumn>>();
for (ReadCommand command: commands)
{
ColumnFamily cf =
columnFamilies.get(StorageService.getPartitioner().decorateKey(command.key));
boolean reverseOrder = command instanceof
SliceFromReadCommand && ((SliceFromReadCommand)command).reversed;
List<ColumnOrSuperColumn> thriftifiedColumns =
thriftifyColumnFamily(cf, command.queryPath.superColumnName != null,
reverseOrder);
columnFamiliesMap.put(command.key, thriftifiedColumns);
}
return columnFamiliesMap;
}
When I inspected in debug mode, the command variable in for loop has the
valid count value. The thriftifyColumnFamily(cf,
command.queryPath.superColumnName != null, reverseOrder) method actually
get columns but it has no way to get count value, and return all value
that were not limit by the count.
Re: Count of SliceRange in get_slice seems not to work
Posted by Hayarobi Park <ha...@gmail.com>.
I found the reason (in the view of client side).
I used unit test of my DAO class. the test class inserted test row and
columns before doing test, and then do test, finally delete inserted
columns after test.
The test was succeeded at first. When I do that test again, the test
code attempt to insert columns as same row key, same super column name,
same sub column name and same value as the those of the test before. The
test failed.
I think it's hard to describe whole example to reproduce that, the code
use custom DAO object to domain specific dao on the layer of JAVA client
API over thrift.
I'll ask this problem to discussion group of that JAVA client api, and
will make new thread later if I find something else.
2010-11-19 (금), 00:23 +1300, aaron morton:
> Sorry, I'm not following your example.
>
> Could you describe the request you sent, what you expected to get back and what you actually got back. Are you able to reproduce the fault in a clean install, e.g. load this data, run these commands and then it goes bang ?
>
> Aaron
>
>
> On 18 Nov 2010, at 23:54, Hayarobi Park wrote:
>
> > I inspected some code of the cluster ring.
> >
> > Thread [ReadStage:6] (Suspended)
> > SuperColumn.isMarkedForDelete() line: 87
> > SliceQueryFilter.collectReducedColumns(IColumnContainer,
> > Iterator<IColumn>, int) line: 138
> > QueryFilter.collectCollatedColumns(ColumnFamily, Iterator<IColumn>,
> > int) line: 146
> > ColumnFamilyStore.getTopLevelColumns(QueryFilter, int) line: 1157
> > ColumnFamilyStore.getColumnFamily(QueryFilter, int) line: 1034
> > ColumnFamilyStore.getColumnFamily(QueryFilter) line: 1004
> > Table.getRow(QueryFilter) line: 359
> > SliceFromReadCommand.getRow(Table) line: 63
> > ReadVerbHandler.doVerb(Message) line: 73
> > MessageDeliveryTask.run() line: 62
> > ThreadPoolExecutor$Worker.runTask(Runnable) line: 886
> > ThreadPoolExecutor$Worker.run() line: 908
> > Thread.run() line: 619
> >
> > In method of SliceQueryFilter.collectReducedColumns(), all columns has
> > same values of SuperColumn.localDeletionTime and
> > SuperColumn.markedForDeleteAt, which seemed to be current time. That
> > caused SuperColumn.isMarkedForDelete() to return true, and then no
> > liveColumns but all non-gc-able columns to be returned.
> >
> > This happened in super column family with LongType super column. (see
> > description below.) SCF with UTF8Type super column worked normally.
> >
> > I'm not sure if the column was inserted with bad deletion time ,by bug
> > of client API or etc, or there was a problem to read super columns with
> > Long Type comparator, or else.
> >
> >
> > ColumnFamily: LongtypeSCF (Super)
> > Columns sorted by:
> > org.apache.cassandra.db.marshal.LongType/org.apache.cassandra.db.marshal.UTF8Type
> > Row cache size / save period: 0.0/0
> > Key cache size / save period: 200000.0/3600
> > Memtable thresholds: 0.19218749999999998/41/60
> > GC grace seconds: 864000
> > Compaction min/max thresholds: 4/32
> > Read repair chance: 1.0
> >
> > ColumnFamily: StringTypeSCF (Super)
> > Columns sorted by:
> > org.apache.cassandra.db.marshal.UTF8Type/org.apache.cassandra.db.marshal.UTF8Type
> > Row cache size / save period: 0.0/0
> > Key cache size / save period: 200000.0/3600
> > Memtable thresholds: 0.19218749999999998/41/60
> > GC grace seconds: 864000
> > Compaction min/max thresholds: 4/32
> > Read repair chance: 1.0
> >
> >
> >
> > 2010-11-18 (목), 04:28 +0000, Aaron Morton:
> >> Just had a quick look at an 0.7b2 install and it appeared to be
> >> working as expected.
> >>
> >>
> >> Here's what I got for a row with 50 super columns, that each have 50
> >> columns. I ran the following get_slice calls .
> >>
> >>
> >> get_slice with no super column specified, count=100
> >> returned 50 super columns, each with 50 columns
> >>
> >>
> >> get_slice with no super column specified, count = 5
> >> returned 5 super column, each with 50 columns
> >>
> >>
> >> If your get_slice does not specify a super column (on the ColumParent
> >> arg) the count applies to the number of SuperColumn objects to return.
> >> Each of those will have all of it's columns. If a super column is
> >> specified on ColumnParent then the count refers to the number of
> >> Columns to return.
> >>
> >>
> >> If you're seeing something else can you send an example.
> >>
> >>
> >> Thanks.
> >> Aaron
> >>
> >>
> >> On 18 Nov, 2010,at 03:04 PM, Hayarobi Park <ha...@gmail.com>
> >> wrote:
> >>
> >>
> >>> It returned all columns within the range of start and end without
> >>> regard
> >>> to the count. the CF is super column family and I send the range of
> >>> super column names of type Long. (and sub column name was UTF8)
> >>>
> >>> I put 2000 super columns in a row, and tried to read the first 50
> >>> columns in some range of columns. I inspected
> >>> StorageProxy.readProtocol() after read your reply, and got the
> >>> command
> >>> object, of class SliceFromReadCommand, has the 'count' member
> >>> variable
> >>> having int value 50.
> >>>
> >>> I test get_slice request to super column family of UTF8Type/UTF8Type
> >>> for
> >>> super column name/column name, and this test was successfully return
> >>> the
> >>> columns with requested count.
> >>>
> >>>
> >>> 2010-11-18 (목), 00:35 +1300, aaron morton:
> >>>> The CassandraServer is not doing the read, step through the code
> >>> from the call to readColumnFamily() in getSlice().
> >>>>
> >>>> The read is passed to the StorageProxy.readProtocol() which looks
> >>> at the CL and determines if its a weak or strong read, sends it out
> >>> to all the replicas and manages everything. Eventually the request
> >>> ends up as the ReadVerbHandler() where it will deserialise an
> >>> instance of the SliceFromReadCommand and call it's getRow(). From
> >>> there you can trace through how the count is used.
> >>>>
> >>>> Do you have a case where a call to the API returned more or less
> >>> data than expected?
> >>>>
> >>>> Hope that helps.
> >>>> Aaron
> >>>>
> >>>> On 17 Nov 2010, at 21:03, Hayarobi Park wrote:
> >>>>
> >>>>> Hello.
> >>>>>
> >>>>> I'm using cassandra (currently 0.7.0-beta3) in JAVA; with
> >>> library
> >>>>> hector.
> >>>>>
> >>>>> It seems that cassandra ignore the count of SliceRange when
> >>> received
> >>>>> get_slice request.
> >>>>>
> >>>>>
> >>>>> I traced cassandra source code, and the part of code that
> >>> retrieving
> >>>>> columns does not get count as parameter. See,
> >>>>> getSlice(List<ReadCommand> commands, ConsistencyLevel
> >>> consistency_level)
> >>>>> method in org.apachecassandra.thrift.CassandraServer class.
> >>> (line
> >>>>> 224~238 in 0.7.0-beta3)
> >>>>>
> >>>>>
> >>>>> private Map<ByteBuffer, List<ColumnOrSuperColumn>>
> >>>>> getSlice(List<ReadCommand> commands, ConsistencyLevel
> >>> consistency_level)
> >>>>> throws InvalidRequestException, UnavailableException,
> >>>>> TimedOutException
> >>>>> {
> >>>>> Map<DecoratedKey, ColumnFamily> columnFamilies =
> >>>>> readColumnFamily(commands, consistency_level);
> >>>>> Map<ByteBuffer, List<ColumnOrSuperColumn>> columnFamiliesMap =
> >>>>> new HashMap<ByteBuffer, List<ColumnOrSuperColumn>>();
> >>>>> for (ReadCommand command: commands)
> >>>>> {
> >>>>> ColumnFamily cf =
> >>>>>
> >>> columnFamilies.get(StorageService.getPartitioner().decorateKey(command.key));
> >>>>> boolean reverseOrder = command instanceof
> >>>>> SliceFromReadCommand &&
> >>> ((SliceFromReadCommand)command).reversed;
> >>>>> List<ColumnOrSuperColumn> thriftifiedColumns =
> >>>>> thriftifyColumnFamily(cf, command.queryPath.superColumnName !=
> >>> null,
> >>>>> reverseOrder);
> >>>>> columnFamiliesMap.put(command.key, thriftifiedColumns);
> >>>>> }
> >>>>>
> >>>>> return columnFamiliesMap;
> >>>>> }
> >>>>>
> >>>>> When I inspected in debug mode, the command variable in for loop
> >>> has the
> >>>>> valid count value. The thriftifyColumnFamily(cf,
> >>>>> command.queryPath.superColumnName != null, reverseOrder) method
> >>> actually
> >>>>> get columns but it has no way to get count value, and return all
> >>> value
> >>>>> that were not limit by the count.
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >
> >
>
Re: Count of SliceRange in get_slice seems not to work
Posted by aaron morton <aa...@thelastpickle.com>.
Sorry, I'm not following your example.
Could you describe the request you sent, what you expected to get back and what you actually got back. Are you able to reproduce the fault in a clean install, e.g. load this data, run these commands and then it goes bang ?
Aaron
On 18 Nov 2010, at 23:54, Hayarobi Park wrote:
> I inspected some code of the cluster ring.
>
> Thread [ReadStage:6] (Suspended)
> SuperColumn.isMarkedForDelete() line: 87
> SliceQueryFilter.collectReducedColumns(IColumnContainer,
> Iterator<IColumn>, int) line: 138
> QueryFilter.collectCollatedColumns(ColumnFamily, Iterator<IColumn>,
> int) line: 146
> ColumnFamilyStore.getTopLevelColumns(QueryFilter, int) line: 1157
> ColumnFamilyStore.getColumnFamily(QueryFilter, int) line: 1034
> ColumnFamilyStore.getColumnFamily(QueryFilter) line: 1004
> Table.getRow(QueryFilter) line: 359
> SliceFromReadCommand.getRow(Table) line: 63
> ReadVerbHandler.doVerb(Message) line: 73
> MessageDeliveryTask.run() line: 62
> ThreadPoolExecutor$Worker.runTask(Runnable) line: 886
> ThreadPoolExecutor$Worker.run() line: 908
> Thread.run() line: 619
>
> In method of SliceQueryFilter.collectReducedColumns(), all columns has
> same values of SuperColumn.localDeletionTime and
> SuperColumn.markedForDeleteAt, which seemed to be current time. That
> caused SuperColumn.isMarkedForDelete() to return true, and then no
> liveColumns but all non-gc-able columns to be returned.
>
> This happened in super column family with LongType super column. (see
> description below.) SCF with UTF8Type super column worked normally.
>
> I'm not sure if the column was inserted with bad deletion time ,by bug
> of client API or etc, or there was a problem to read super columns with
> Long Type comparator, or else.
>
>
> ColumnFamily: LongtypeSCF (Super)
> Columns sorted by:
> org.apache.cassandra.db.marshal.LongType/org.apache.cassandra.db.marshal.UTF8Type
> Row cache size / save period: 0.0/0
> Key cache size / save period: 200000.0/3600
> Memtable thresholds: 0.19218749999999998/41/60
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
>
> ColumnFamily: StringTypeSCF (Super)
> Columns sorted by:
> org.apache.cassandra.db.marshal.UTF8Type/org.apache.cassandra.db.marshal.UTF8Type
> Row cache size / save period: 0.0/0
> Key cache size / save period: 200000.0/3600
> Memtable thresholds: 0.19218749999999998/41/60
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
>
>
>
> 2010-11-18 (목), 04:28 +0000, Aaron Morton:
>> Just had a quick look at an 0.7b2 install and it appeared to be
>> working as expected.
>>
>>
>> Here's what I got for a row with 50 super columns, that each have 50
>> columns. I ran the following get_slice calls .
>>
>>
>> get_slice with no super column specified, count=100
>> returned 50 super columns, each with 50 columns
>>
>>
>> get_slice with no super column specified, count = 5
>> returned 5 super column, each with 50 columns
>>
>>
>> If your get_slice does not specify a super column (on the ColumParent
>> arg) the count applies to the number of SuperColumn objects to return.
>> Each of those will have all of it's columns. If a super column is
>> specified on ColumnParent then the count refers to the number of
>> Columns to return.
>>
>>
>> If you're seeing something else can you send an example.
>>
>>
>> Thanks.
>> Aaron
>>
>>
>> On 18 Nov, 2010,at 03:04 PM, Hayarobi Park <ha...@gmail.com>
>> wrote:
>>
>>
>>> It returned all columns within the range of start and end without
>>> regard
>>> to the count. the CF is super column family and I send the range of
>>> super column names of type Long. (and sub column name was UTF8)
>>>
>>> I put 2000 super columns in a row, and tried to read the first 50
>>> columns in some range of columns. I inspected
>>> StorageProxy.readProtocol() after read your reply, and got the
>>> command
>>> object, of class SliceFromReadCommand, has the 'count' member
>>> variable
>>> having int value 50.
>>>
>>> I test get_slice request to super column family of UTF8Type/UTF8Type
>>> for
>>> super column name/column name, and this test was successfully return
>>> the
>>> columns with requested count.
>>>
>>>
>>> 2010-11-18 (목), 00:35 +1300, aaron morton:
>>>> The CassandraServer is not doing the read, step through the code
>>> from the call to readColumnFamily() in getSlice().
>>>>
>>>> The read is passed to the StorageProxy.readProtocol() which looks
>>> at the CL and determines if its a weak or strong read, sends it out
>>> to all the replicas and manages everything. Eventually the request
>>> ends up as the ReadVerbHandler() where it will deserialise an
>>> instance of the SliceFromReadCommand and call it's getRow(). From
>>> there you can trace through how the count is used.
>>>>
>>>> Do you have a case where a call to the API returned more or less
>>> data than expected?
>>>>
>>>> Hope that helps.
>>>> Aaron
>>>>
>>>> On 17 Nov 2010, at 21:03, Hayarobi Park wrote:
>>>>
>>>>> Hello.
>>>>>
>>>>> I'm using cassandra (currently 0.7.0-beta3) in JAVA; with
>>> library
>>>>> hector.
>>>>>
>>>>> It seems that cassandra ignore the count of SliceRange when
>>> received
>>>>> get_slice request.
>>>>>
>>>>>
>>>>> I traced cassandra source code, and the part of code that
>>> retrieving
>>>>> columns does not get count as parameter. See,
>>>>> getSlice(List<ReadCommand> commands, ConsistencyLevel
>>> consistency_level)
>>>>> method in org.apachecassandra.thrift.CassandraServer class.
>>> (line
>>>>> 224~238 in 0.7.0-beta3)
>>>>>
>>>>>
>>>>> private Map<ByteBuffer, List<ColumnOrSuperColumn>>
>>>>> getSlice(List<ReadCommand> commands, ConsistencyLevel
>>> consistency_level)
>>>>> throws InvalidRequestException, UnavailableException,
>>>>> TimedOutException
>>>>> {
>>>>> Map<DecoratedKey, ColumnFamily> columnFamilies =
>>>>> readColumnFamily(commands, consistency_level);
>>>>> Map<ByteBuffer, List<ColumnOrSuperColumn>> columnFamiliesMap =
>>>>> new HashMap<ByteBuffer, List<ColumnOrSuperColumn>>();
>>>>> for (ReadCommand command: commands)
>>>>> {
>>>>> ColumnFamily cf =
>>>>>
>>> columnFamilies.get(StorageService.getPartitioner().decorateKey(command.key));
>>>>> boolean reverseOrder = command instanceof
>>>>> SliceFromReadCommand &&
>>> ((SliceFromReadCommand)command).reversed;
>>>>> List<ColumnOrSuperColumn> thriftifiedColumns =
>>>>> thriftifyColumnFamily(cf, command.queryPath.superColumnName !=
>>> null,
>>>>> reverseOrder);
>>>>> columnFamiliesMap.put(command.key, thriftifiedColumns);
>>>>> }
>>>>>
>>>>> return columnFamiliesMap;
>>>>> }
>>>>>
>>>>> When I inspected in debug mode, the command variable in for loop
>>> has the
>>>>> valid count value. The thriftifyColumnFamily(cf,
>>>>> command.queryPath.superColumnName != null, reverseOrder) method
>>> actually
>>>>> get columns but it has no way to get count value, and return all
>>> value
>>>>> that were not limit by the count.
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>
>
Re:Re: Count of SliceRange in get_slice seems not to work
Posted by Hayarobi Park <ha...@gmail.com>.
I inspected some code of the cluster ring.
Thread [ReadStage:6] (Suspended)
SuperColumn.isMarkedForDelete() line: 87
SliceQueryFilter.collectReducedColumns(IColumnContainer,
Iterator<IColumn>, int) line: 138
QueryFilter.collectCollatedColumns(ColumnFamily, Iterator<IColumn>,
int) line: 146
ColumnFamilyStore.getTopLevelColumns(QueryFilter, int) line: 1157
ColumnFamilyStore.getColumnFamily(QueryFilter, int) line: 1034
ColumnFamilyStore.getColumnFamily(QueryFilter) line: 1004
Table.getRow(QueryFilter) line: 359
SliceFromReadCommand.getRow(Table) line: 63
ReadVerbHandler.doVerb(Message) line: 73
MessageDeliveryTask.run() line: 62
ThreadPoolExecutor$Worker.runTask(Runnable) line: 886
ThreadPoolExecutor$Worker.run() line: 908
Thread.run() line: 619
In method of SliceQueryFilter.collectReducedColumns(), all columns has
same values of SuperColumn.localDeletionTime and
SuperColumn.markedForDeleteAt, which seemed to be current time. That
caused SuperColumn.isMarkedForDelete() to return true, and then no
liveColumns but all non-gc-able columns to be returned.
This happened in super column family with LongType super column. (see
description below.) SCF with UTF8Type super column worked normally.
I'm not sure if the column was inserted with bad deletion time ,by bug
of client API or etc, or there was a problem to read super columns with
Long Type comparator, or else.
ColumnFamily: LongtypeSCF (Super)
Columns sorted by:
org.apache.cassandra.db.marshal.LongType/org.apache.cassandra.db.marshal.UTF8Type
Row cache size / save period: 0.0/0
Key cache size / save period: 200000.0/3600
Memtable thresholds: 0.19218749999999998/41/60
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
ColumnFamily: StringTypeSCF (Super)
Columns sorted by:
org.apache.cassandra.db.marshal.UTF8Type/org.apache.cassandra.db.marshal.UTF8Type
Row cache size / save period: 0.0/0
Key cache size / save period: 200000.0/3600
Memtable thresholds: 0.19218749999999998/41/60
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
2010-11-18 (목), 04:28 +0000, Aaron Morton:
> Just had a quick look at an 0.7b2 install and it appeared to be
> working as expected.
>
>
> Here's what I got for a row with 50 super columns, that each have 50
> columns. I ran the following get_slice calls .
>
>
> get_slice with no super column specified, count=100
> returned 50 super columns, each with 50 columns
>
>
> get_slice with no super column specified, count = 5
> returned 5 super column, each with 50 columns
>
>
> If your get_slice does not specify a super column (on the ColumParent
> arg) the count applies to the number of SuperColumn objects to return.
> Each of those will have all of it's columns. If a super column is
> specified on ColumnParent then the count refers to the number of
> Columns to return.
>
>
> If you're seeing something else can you send an example.
>
>
> Thanks.
> Aaron
>
>
> On 18 Nov, 2010,at 03:04 PM, Hayarobi Park <ha...@gmail.com>
> wrote:
>
>
> > It returned all columns within the range of start and end without
> > regard
> > to the count. the CF is super column family and I send the range of
> > super column names of type Long. (and sub column name was UTF8)
> >
> > I put 2000 super columns in a row, and tried to read the first 50
> > columns in some range of columns. I inspected
> > StorageProxy.readProtocol() after read your reply, and got the
> > command
> > object, of class SliceFromReadCommand, has the 'count' member
> > variable
> > having int value 50.
> >
> > I test get_slice request to super column family of UTF8Type/UTF8Type
> > for
> > super column name/column name, and this test was successfully return
> > the
> > columns with requested count.
> >
> >
> > 2010-11-18 (목), 00:35 +1300, aaron morton:
> > > The CassandraServer is not doing the read, step through the code
> > from the call to readColumnFamily() in getSlice().
> > >
> > > The read is passed to the StorageProxy.readProtocol() which looks
> > at the CL and determines if its a weak or strong read, sends it out
> > to all the replicas and manages everything. Eventually the request
> > ends up as the ReadVerbHandler() where it will deserialise an
> > instance of the SliceFromReadCommand and call it's getRow(). From
> > there you can trace through how the count is used.
> > >
> > > Do you have a case where a call to the API returned more or less
> > data than expected?
> > >
> > > Hope that helps.
> > > Aaron
> > >
> > > On 17 Nov 2010, at 21:03, Hayarobi Park wrote:
> > >
> > > > Hello.
> > > >
> > > > I'm using cassandra (currently 0.7.0-beta3) in JAVA; with
> > library
> > > > hector.
> > > >
> > > > It seems that cassandra ignore the count of SliceRange when
> > received
> > > > get_slice request.
> > > >
> > > >
> > > > I traced cassandra source code, and the part of code that
> > retrieving
> > > > columns does not get count as parameter. See,
> > > > getSlice(List<ReadCommand> commands, ConsistencyLevel
> > consistency_level)
> > > > method in org.apachecassandra.thrift.CassandraServer class.
> > (line
> > > > 224~238 in 0.7.0-beta3)
> > > >
> > > >
> > > > private Map<ByteBuffer, List<ColumnOrSuperColumn>>
> > > > getSlice(List<ReadCommand> commands, ConsistencyLevel
> > consistency_level)
> > > > throws InvalidRequestException, UnavailableException,
> > > > TimedOutException
> > > > {
> > > > Map<DecoratedKey, ColumnFamily> columnFamilies =
> > > > readColumnFamily(commands, consistency_level);
> > > > Map<ByteBuffer, List<ColumnOrSuperColumn>> columnFamiliesMap =
> > > > new HashMap<ByteBuffer, List<ColumnOrSuperColumn>>();
> > > > for (ReadCommand command: commands)
> > > > {
> > > > ColumnFamily cf =
> > > >
> > columnFamilies.get(StorageService.getPartitioner().decorateKey(command.key));
> > > > boolean reverseOrder = command instanceof
> > > > SliceFromReadCommand &&
> > ((SliceFromReadCommand)command).reversed;
> > > > List<ColumnOrSuperColumn> thriftifiedColumns =
> > > > thriftifyColumnFamily(cf, command.queryPath.superColumnName !=
> > null,
> > > > reverseOrder);
> > > > columnFamiliesMap.put(command.key, thriftifiedColumns);
> > > > }
> > > >
> > > > return columnFamiliesMap;
> > > > }
> > > >
> > > > When I inspected in debug mode, the command variable in for loop
> > has the
> > > > valid count value. The thriftifyColumnFamily(cf,
> > > > command.queryPath.superColumnName != null, reverseOrder) method
> > actually
> > > > get columns but it has no way to get count value, and return all
> > value
> > > > that were not limit by the count.
> > > >
> > > >
> > >
> >
> >
> >
Re: Count of SliceRange in get_slice seems not to work
Posted by Aaron Morton <aa...@thelastpickle.com>.
Just had a quick look at an 0.7b2 install and it appeared to be working as expected.
Here's what I got for a row with 50 super columns, that each have 50 columns. I ran the following get_slice calls .
get_slice with no super column specified, count=100
returned 50 super columns, each with 50 columns
get_slice with no super column specified, count = 5
returned 5 super column, each with 50 columns
If your get_slice does not specify a super column (on the ColumParent arg) the count applies to the number of SuperColumn objects to return. Each of those will have all of it's columns. If a super column is specified on ColumnParent then the count refers to the number of Columns to return.
If you're seeing something else can you send an example.
Thanks
Aaron
On 18 Nov, 2010,at 03:04 PM, Hayarobi Park <ha...@gmail.com> wrote:
It returned all columns within the range of start and end without regard
to the count. the CF is super column family and I send the range of
super column names of type Long. (and sub column name was UTF8)
I put 2000 super columns in a row, and tried to read the first 50
columns in some range of columns. I inspected
StorageProxy.readProtocol() after read your reply, and got the command
object, of class SliceFromReadCommand, has the 'count' member variable
having int value 50.
I test get_slice request to super column family of UTF8Type/UTF8Type for
super column name/column name, and this test was successfully return the
columns with requested count.
2010-11-18 (목), 00:35 +1300, aaron morton:
> The CassandraServer is not doing the read, step through the code from the call to readColumnFamily() in getSlice()
>
> The read is passed to the StorageProxy.readProtocol() which looks at the CL and determines if its a weak or strong read, sends it out to all the replicas and manages everything. Eventually the request ends up as the ReadVerbHandler() where it will deserialise an instance of the SliceFromReadCommand and call it's getRow(). From there you can trace through how the count is used.
>
> Do you have a case where a call to the API returned more or less data than expected?
>
> Hope that helps.
> Aaron
>
> On 17 Nov 2010, at 21:03, Hayarobi Park wrote:
>
> > Hello.
> >
> > I'm using cassandra (currently 0.7.0-beta3) in JAVA; with library
> > hector.
> >
> > It seems that cassandra ignore the count of SliceRange when received
> > get_slice request.
> >
> >
> > I traced cassandra source code, and the part of code that retrieving
> > columns does not get count as parameter. See,
> > getSlice(List<ReadCommand> commands, ConsistencyLevel consistency_level)
> > method in org.apache.cassandra.thrift.CassandraServer class. (line
> > 224~238 in 0.7.0-beta3)
> >
> >
> > private Map<ByteBuffer, List<ColumnOrSuperColumn>>
> > getSlice(List<ReadCommand> commands, ConsistencyLevel consistency_level)
> > throws InvalidRequestException, UnavailableException,
> > TimedOutException
> > {
> > Map<DecoratedKey, ColumnFamily> columnFamilies =
> > readColumnFamily(commands, consistency_level);
> > Map<ByteBuffer, List<ColumnOrSuperColumn>> columnFamiliesMap =
> > new HashMap<ByteBuffer, List<ColumnOrSuperColumn>>();
> > for (ReadCommand command: commands)
> > {
> > ColumnFamily cf =
> > columnFamilies.get(StorageService.getPartitioner().decorateKey(command.key));
> > boolean reverseOrder = command instanceof
> > SliceFromReadCommand && ((SliceFromReadCommand)command).reversed;
> > List<ColumnOrSuperColumn> thriftifiedColumns =
> > thriftifyColumnFamily(cf, command.queryPath.superColumnName != null,
> > reverseOrder);
> > columnFamiliesMap.put(command.key, thriftifiedColumns);
> > }
> >
> > return columnFamiliesMap;
> > }
> >
> > When I inspected in debug mode, the command variable in for loop has the
> > valid count value. The thriftifyColumnFamily(cf,
> > command.queryPath.superColumnName != null, reverseOrder) method actually
> > get columns but it has no way to get count value, and return all value
> > that were not limit by the count.
> >
> >
>
Re: Count of SliceRange in get_slice seems not to work
Posted by Hayarobi Park <ha...@gmail.com>.
It returned all columns within the range of start and end without regard
to the count. the CF is super column family and I send the range of
super column names of type Long. (and sub column name was UTF8)
I put 2000 super columns in a row, and tried to read the first 50
columns in some range of columns. I inspected
StorageProxy.readProtocol() after read your reply, and got the command
object, of class SliceFromReadCommand, has the 'count' member variable
having int value 50.
I test get_slice request to super column family of UTF8Type/UTF8Type for
super column name/column name, and this test was successfully return the
columns with requested count.
2010-11-18 (목), 00:35 +1300, aaron morton:
> The CassandraServer is not doing the read, step through the code from the call to readColumnFamily() in getSlice().
>
> The read is passed to the StorageProxy.readProtocol() which looks at the CL and determines if its a weak or strong read, sends it out to all the replicas and manages everything. Eventually the request ends up as the ReadVerbHandler() where it will deserialise an instance of the SliceFromReadCommand and call it's getRow(). From there you can trace through how the count is used.
>
> Do you have a case where a call to the API returned more or less data than expected?
>
> Hope that helps.
> Aaron
>
> On 17 Nov 2010, at 21:03, Hayarobi Park wrote:
>
> > Hello.
> >
> > I'm using cassandra (currently 0.7.0-beta3) in JAVA; with library
> > hector.
> >
> > It seems that cassandra ignore the count of SliceRange when received
> > get_slice request.
> >
> >
> > I traced cassandra source code, and the part of code that retrieving
> > columns does not get count as parameter. See,
> > getSlice(List<ReadCommand> commands, ConsistencyLevel consistency_level)
> > method in org.apache.cassandra.thrift.CassandraServer class. (line
> > 224~238 in 0.7.0-beta3)
> >
> >
> > private Map<ByteBuffer, List<ColumnOrSuperColumn>>
> > getSlice(List<ReadCommand> commands, ConsistencyLevel consistency_level)
> > throws InvalidRequestException, UnavailableException,
> > TimedOutException
> > {
> > Map<DecoratedKey, ColumnFamily> columnFamilies =
> > readColumnFamily(commands, consistency_level);
> > Map<ByteBuffer, List<ColumnOrSuperColumn>> columnFamiliesMap =
> > new HashMap<ByteBuffer, List<ColumnOrSuperColumn>>();
> > for (ReadCommand command: commands)
> > {
> > ColumnFamily cf =
> > columnFamilies.get(StorageService.getPartitioner().decorateKey(command.key));
> > boolean reverseOrder = command instanceof
> > SliceFromReadCommand && ((SliceFromReadCommand)command).reversed;
> > List<ColumnOrSuperColumn> thriftifiedColumns =
> > thriftifyColumnFamily(cf, command.queryPath.superColumnName != null,
> > reverseOrder);
> > columnFamiliesMap.put(command.key, thriftifiedColumns);
> > }
> >
> > return columnFamiliesMap;
> > }
> >
> > When I inspected in debug mode, the command variable in for loop has the
> > valid count value. The thriftifyColumnFamily(cf,
> > command.queryPath.superColumnName != null, reverseOrder) method actually
> > get columns but it has no way to get count value, and return all value
> > that were not limit by the count.
> >
> >
>
Re: Count of SliceRange in get_slice seems not to work
Posted by aaron morton <aa...@thelastpickle.com>.
The CassandraServer is not doing the read, step through the code from the call to readColumnFamily() in getSlice().
The read is passed to the StorageProxy.readProtocol() which looks at the CL and determines if its a weak or strong read, sends it out to all the replicas and manages everything. Eventually the request ends up as the ReadVerbHandler() where it will deserialise an instance of the SliceFromReadCommand and call it's getRow(). From there you can trace through how the count is used.
Do you have a case where a call to the API returned more or less data than expected?
Hope that helps.
Aaron
On 17 Nov 2010, at 21:03, Hayarobi Park wrote:
> Hello.
>
> I'm using cassandra (currently 0.7.0-beta3) in JAVA; with library
> hector.
>
> It seems that cassandra ignore the count of SliceRange when received
> get_slice request.
>
>
> I traced cassandra source code, and the part of code that retrieving
> columns does not get count as parameter. See,
> getSlice(List<ReadCommand> commands, ConsistencyLevel consistency_level)
> method in org.apache.cassandra.thrift.CassandraServer class. (line
> 224~238 in 0.7.0-beta3)
>
>
> private Map<ByteBuffer, List<ColumnOrSuperColumn>>
> getSlice(List<ReadCommand> commands, ConsistencyLevel consistency_level)
> throws InvalidRequestException, UnavailableException,
> TimedOutException
> {
> Map<DecoratedKey, ColumnFamily> columnFamilies =
> readColumnFamily(commands, consistency_level);
> Map<ByteBuffer, List<ColumnOrSuperColumn>> columnFamiliesMap =
> new HashMap<ByteBuffer, List<ColumnOrSuperColumn>>();
> for (ReadCommand command: commands)
> {
> ColumnFamily cf =
> columnFamilies.get(StorageService.getPartitioner().decorateKey(command.key));
> boolean reverseOrder = command instanceof
> SliceFromReadCommand && ((SliceFromReadCommand)command).reversed;
> List<ColumnOrSuperColumn> thriftifiedColumns =
> thriftifyColumnFamily(cf, command.queryPath.superColumnName != null,
> reverseOrder);
> columnFamiliesMap.put(command.key, thriftifiedColumns);
> }
>
> return columnFamiliesMap;
> }
>
> When I inspected in debug mode, the command variable in for loop has the
> valid count value. The thriftifyColumnFamily(cf,
> command.queryPath.superColumnName != null, reverseOrder) method actually
> get columns but it has no way to get count value, and return all value
> that were not limit by the count.
>
>