You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yang Yang (Created) (JIRA)" <ji...@apache.org> on 2011/10/20 21:46:10 UTC
[jira] [Created] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
ReadResponseSerializer.serializedSize() calculation is wrong
------------------------------------------------------------
Key: CASSANDRA-3390
URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
Project: Cassandra
Issue Type: Bug
Reporter: Yang Yang
Attachments: 3390.patch
in ReadResponse.java
the following code
public long serializedSize(ReadResponse response, int version)
{
int size = DBConstants.intSize;
size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
size += DBConstants.boolSize;
if (response.isDigestQuery())
size += response.digest().remaining();
else
size += Row.serializer().serializedSize(response.row(), version);
return size;
}
adds the digest size 2 times
this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133584#comment-13133584 ]
Yang Yang edited comment on CASSANDRA-3390 at 10/23/11 7:31 AM:
----------------------------------------------------------------
Jonathan: I see what you mean:
the returned CF from cachedRow, in CFS.java:1169 :
/**
* Filter a cached row, which will not be modified by the filter, but may be modified by throwing out
* tombstones that are no longer relevant.
* The returned column family won't be thread safe.
*/
ColumnFamily filterColumnFamily(ColumnFamily cached, QueryFilter filter, int gcBefore)
{
it seems that for some CFs, the buffer.length() was calculated, then another thread did the
removeDeletedColumns() for expiring columns in filterColumnFamily(), then serializedSize() was calculated again, something like that.
if that is the case, would it actually cause correctness problems, or is it just an annoying discrepancy between 2 reports on the size (which reflect the size at 2 times in history)?
was (Author: yangyangyyy):
Jonathan: I see what you mean:
the returned CF from cachedRow, in CFS.java:1169 :
/**
* Filter a cached row, which will not be modified by the filter, but may be modified by throwing out
* tombstones that are no longer relevant.
* The returned column family won't be thread safe.
*/
ColumnFamily filterColumnFamily(ColumnFamily cached, QueryFilter filter, int gcBefore)
{
it seems that for some CFs, the buffer.length() was calculated, then another thread did the
removeDeletedColumns() for expiring columns in filterColumnFamily(), then serializedSize() was calculated again,
if that is the case, would it actually cause correctness problems, or is it just an annoying discrepancy between 2 reports on the size (which reflect the size at 2 times in history)?
something like that.
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132552#comment-13132552 ]
Sylvain Lebresne commented on CASSANDRA-3390:
---------------------------------------------
Do you have an easy way to reproduce that remaining error ? Do you use something specific: super columns ? counters ? expiring columns ?
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133067#comment-13133067 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
I did this too myself
doesn't seem particularly helpful, except for telling us that it's a normal column, with TTL
I'm intrigued by why the difference is always 81 or 77 bytes
ERROR [ReadStage:176] 2011-10-21 14:45:04,099 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1
76,5,main]
java.lang.AssertionError: row:Row(key=DecoratedKey(58613415544222752604144955634601649391, 3164323233656134), cf=ColumnFamily(mea
suredSession [0000013327cc2ad7303030303030303030303030303030303030303030303030303030303030303030303164323233656135:false:4@1319222
703553!600,0000013327cc2ad7303030303030303030303030303030303030303030303030303030303030303030303164323233656137:false:4@1319222703
556!600,0000013327cc2ad7303030303030303030303030303030303030303030303030303030303030303030303164323233656139:false:4@1319222703562
!600,0000013327cc2ad7303030303030303030303030303030303030303030303030303030303030303030303164323233656162:false:4@1319222703567!60
0,0000013327cc2ad7303030303030303030303030303030303030303030303030303030303030303030303164323233656164:false:4@1319222703571!600,0
000013327cc2ad7303030303030303030303030303030303030303030303030303030303030303030303164323233656166:false:4@1319222703584!600,0000
013327cc2ad7303030303030303030303030303030303030303030303030303030303030303030303164323233656231:false:4@1319222703589!600,0000013
327cc2ad7303030303030303030303030303030303030303030303030303030303030303030303164323233656233:false:4@1319222703592!600,]))Final b
uffer length 1152 to accomodate data size of 652 (predicted 575)
at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:683)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:56)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133054#comment-13133054 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
I mean
buffer.getData().length returns the underlying byte[] length, which is always doubled on expansion, so this will not be the same as the actual bytes written
buffer.getLength() returns the "count" var, yes, this should be the same as the serializedSize() calculation
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133079#comment-13133079 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
yes, let me disable rowcache and see if it goes away...
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133584#comment-13133584 ]
Yang Yang edited comment on CASSANDRA-3390 at 10/23/11 7:26 AM:
----------------------------------------------------------------
Jonathan: I see what you mean:
the returned CF from cachedRow, in CFS.java:1169 :
/**
* Filter a cached row, which will not be modified by the filter, but may be modified by throwing out
* tombstones that are no longer relevant.
* The returned column family won't be thread safe.
*/
ColumnFamily filterColumnFamily(ColumnFamily cached, QueryFilter filter, int gcBefore)
{
it seems that for some CFs, the buffer.length() was calculated, then another thread did the
removeDeletedColumns() for expiring columns in filterColumnFamily(), then serializedSize() was calculated again,
if that is the case, would it actually cause correctness problems, or is it just an annoying discrepancy between 2 reports on the size (which reflect the size at 2 times in history)?
something like that.
was (Author: yangyangyyy):
Jonathan: I see what you mean:
the returned CF from cachedRow, in CFS.java:1169 :
/**
* Filter a cached row, which will not be modified by the filter, but may be modified by throwing out
* tombstones that are no longer relevant.
* The returned column family won't be thread safe.
*/
ColumnFamily filterColumnFamily(ColumnFamily cached, QueryFilter filter, int gcBefore)
{
in ColumnSerializer.java:serialize()--->ByteBufferUtils.write()
out.write(buffer.array(), buffer.arrayOffset() + buffer.position(), buffer.remaining());
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133886#comment-13133886 ]
Sylvain Lebresne commented on CASSANDRA-3390:
---------------------------------------------
bq. We could try to preserve it by only skipping thie copy when the requested count == MAXINT
I don't think that would work either because If I understand correctly the problem, it's that when we serialize, we may add a column between when we compute the serialized size and when we do the actual serialization. The request count shouldn't make any difference here (besides, I agree with the contrary to best-practices argument).
I'll note that I think this optimization has always been wrong, because the cf can also change between when the column count is written in the serialization form and the actual write of the columns. If columns are added, we're kind of fine, we'll serialize more columns that advertised but the code will be ok. But it's possible to have a race where we actually remove columns, because a tombstone could be gced by another thread, in which case we could have a EOFException or something like that during deserialization (it's sufficiently unlikely that either nobody ran into it, or got it only once and never again and so didn't report it or something).
Anyway +1 on the patch, but I believe it would be correct to commit to 0.8 too for the above reason.
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Assignee: Jonathan Ellis
> Fix For: 1.0.1
>
> Attachments: 3390.patch, 3390.txt
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133007#comment-13133007 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
I went through the serialize() and serializedSize() code and they seem to agree,
now the most likely cause is thread issue, would the serialize() ever be called from multiple threads?
the buffer.data().length should not be in the assert check, that discrepancy comes from the result of expand(), really that is an internal thing, should not be checked.
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134613#comment-13134613 ]
Hudson commented on CASSANDRA-3390:
-----------------------------------
Integrated in Cassandra-0.8 #387 (See [https://builds.apache.org/job/Cassandra-0.8/387/])
remove incorrect optimization from slice read path
patch by jbellis; reviewed by slebresne for CASSANDRA-3390
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188353
Files :
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Assignee: Jonathan Ellis
> Fix For: 0.8.8, 1.0.1
>
> Attachments: 3390.patch, 3390.txt
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133076#comment-13133076 ]
Jonathan Ellis commented on CASSANDRA-3390:
-------------------------------------------
I think you're right about the threading being the cause. Are you using row cache?
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Sylvain Lebresne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3390:
----------------------------------------
Affects Version/s: 1.0.1
Fix Version/s: 1.0.1
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133584#comment-13133584 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
Jonathan: I see what you mean:
the returned CF from cachedRow, in CFS.java:1169 :
/**
* Filter a cached row, which will not be modified by the filter, but may be modified by throwing out
* tombstones that are no longer relevant.
* The returned column family won't be thread safe.
*/
ColumnFamily filterColumnFamily(ColumnFamily cached, QueryFilter filter, int gcBefore)
{
in ColumnSerializer.java:serialize()--->ByteBufferUtils.write()
out.write(buffer.array(), buffer.arrayOffset() + buffer.position(), buffer.remaining());
the position() can be non-thread-safe, as well as others.
but that's a bit weird since my application code seems to work , i.e. without cassandra failing on parsing the ReadResponse, which we would see if only partial byteBuffer is written
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132072#comment-13132072 ]
Brandon Williams commented on CASSANDRA-3390:
---------------------------------------------
This is the wrong place to ask, but just switch to the 1.0.0 branch: https://github.com/apache/cassandra/tree/cassandra-1.0.0
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133584#comment-13133584 ]
Yang Yang edited comment on CASSANDRA-3390 at 10/23/11 7:10 AM:
----------------------------------------------------------------
Jonathan: I see what you mean:
the returned CF from cachedRow, in CFS.java:1169 :
/**
* Filter a cached row, which will not be modified by the filter, but may be modified by throwing out
* tombstones that are no longer relevant.
* The returned column family won't be thread safe.
*/
ColumnFamily filterColumnFamily(ColumnFamily cached, QueryFilter filter, int gcBefore)
{
in ColumnSerializer.java:serialize()--->ByteBufferUtils.write()
out.write(buffer.array(), buffer.arrayOffset() + buffer.position(), buffer.remaining());
was (Author: yangyangyyy):
Jonathan: I see what you mean:
the returned CF from cachedRow, in CFS.java:1169 :
/**
* Filter a cached row, which will not be modified by the filter, but may be modified by throwing out
* tombstones that are no longer relevant.
* The returned column family won't be thread safe.
*/
ColumnFamily filterColumnFamily(ColumnFamily cached, QueryFilter filter, int gcBefore)
{
in ColumnSerializer.java:serialize()--->ByteBufferUtils.write()
out.write(buffer.array(), buffer.arrayOffset() + buffer.position(), buffer.remaining());
the position() can be non-thread-safe, as well as others.
but that's a bit weird since my application code seems to work , i.e. without cassandra failing on parsing the ReadResponse, which we would see if only partial byteBuffer is written
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131974#comment-13131974 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
it seems that with the remaining error
buffer.getData().length = size*2 + 1
buffer.getLength() = size + 81
this may be of some help in debugging
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133131#comment-13133131 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
yeah, disabled row cache, with multiple threads running, no assertion error so far
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133054#comment-13133054 ]
Yang Yang edited comment on CASSANDRA-3390 at 10/21/11 8:54 PM:
----------------------------------------------------------------
I mean
buffer.getData().length returns the underlying byte[] length, which is always doubled on expansion, so this will not be the same as the actual bytes written. actually this agrees with what we are seeing, getData().length() == (size + 1 ) * 2, because the buffer is doubled on the last write of one byte.
buffer.getLength() returns the "count" var, yes, this should be the same as the serializedSize() calculation
was (Author: yangyangyyy):
I mean
buffer.getData().length returns the underlying byte[] length, which is always doubled on expansion, so this will not be the same as the actual bytes written
buffer.getLength() returns the "count" var, yes, this should be the same as the serializedSize() calculation
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133066#comment-13133066 ]
Jonathan Ellis commented on CASSANDRA-3390:
-------------------------------------------
I added the object being serialized to the assertion failure message in r1187539. Does that give us anything useful?
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133590#comment-13133590 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
well, expiring column would lead to the size to be bigger than buffer.length(),
but what we saw is the reverse, so it could be caused because new columns are added to the cached CF.
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133063#comment-13133063 ]
Jonathan Ellis commented on CASSANDRA-3390:
-------------------------------------------
But we pre-allocate the buffer size:
{code}
int size = (int) serializer.serializedSize(object, version);
DataOutputBuffer buffer = new DataOutputBuffer(size);
{code}
The point is to avoid copies during buffer re-allocations, so again, this is a good check to have.
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133047#comment-13133047 ]
Jonathan Ellis commented on CASSANDRA-3390:
-------------------------------------------
Sure it should be, because that's what tells us if serializedSize has a bug.
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131961#comment-13131961 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
seems there are still other places where the count is wrong
ERROR [ReadStage:129] 2011-10-20 15:54:50,932 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:1
29,5,main]
java.lang.AssertionError: Final buffer length 1778 to accomodate data size of 969 (predicted 888)
at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:56)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131984#comment-13131984 ]
Jonathan Ellis commented on CASSANDRA-3390:
-------------------------------------------
The first one was fixed in CASSANDRA-3373. You should probably retest on latest trunk to make sure the other isn't fixed too.
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132017#comment-13132017 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
tested with latest on 1.0 branch (I checked that it does have the 3373 fix)
still same:
at java.lang.Thread.run(Thread.java:662)
ERROR [ReadStage:57] 2011-10-20 16:48:50,117 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:57,5,main]
java.lang.AssertionError: Final buffer length 560 to accomodate data size of 360 (predicted 279)
at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:56)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [ReadStage:45] 2011-10-20 16:48:50,200 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:45,5,main]
java.lang.AssertionError: Final buffer length 1002 to accomodate data size of 574 (predicted 500)
at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:56)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Yang updated CASSANDRA-3390:
---------------------------------
Attachment: 3390.patch
should add the digest size only once
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133673#comment-13133673 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
tried the latest patch, seems to work fine. but understandably, the performance is a bit slower due to the extra copy (even with arraybackedSortedColumns )
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Assignee: Jonathan Ellis
> Fix For: 1.0.1
>
> Attachments: 3390.patch, 3390.txt
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132901#comment-13132901 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
standard columns
not counter,
yes they are expiring columns
I found this seems to have to do with thread issues, since it works fine with single client
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132019#comment-13132019 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
btw how can I find the github commit corresponding to the 1.0.0 official release?
thanks
Yang
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Yang Yang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131975#comment-13131975 ]
Yang Yang commented on CASSANDRA-3390:
--------------------------------------
the part i fixed is on the isDigest() path,
I'm suspecting that the remaining error is on the non-digest path, since the sizes are pretty large
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yang Yang
> Attachments: 3390.patch
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134092#comment-13134092 ]
Jonathan Ellis commented on CASSANDRA-3390:
-------------------------------------------
bq. we may add a column between when we compute the serialized size and when we do the actual serialization
Right, I meant that if we were willing to drop the assert and allow the buffer to expand in the rare case of a race here.
I'll commit to 0.8 and 1.0.
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Assignee: Jonathan Ellis
> Fix For: 1.0.1
>
> Attachments: 3390.patch, 3390.txt
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3390)
ReadResponseSerializer.serializedSize() calculation is wrong
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3390:
--------------------------------------
Attachment: 3390.txt
It looks like the culprit is this part:
{code}
if (sliceFilter.count >= cached.getColumnCount())
{code}
... then we don't clone the cached CF. But this is broken as-written; it's possible for new columns to be inserted, increasing the cached's row size after the check.
I think the right thing to do here is jut get rid of this special case. We could try to preserve it by only skipping thie copy when the requested count == MAXINT, but that's contrary to best-practices (we strongly discourage using MAXINT as the limit).
> ReadResponseSerializer.serializedSize() calculation is wrong
> ------------------------------------------------------------
>
> Key: CASSANDRA-3390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3390
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.1
> Reporter: Yang Yang
> Fix For: 1.0.1
>
> Attachments: 3390.patch, 3390.txt
>
>
> in ReadResponse.java
> the following code
> public long serializedSize(ReadResponse response, int version)
> {
> int size = DBConstants.intSize;
> size += (response.isDigestQuery() ? response.digest() : ByteBufferUtil.EMPTY_BYTE_BUFFER).remaining();
> size += DBConstants.boolSize;
> if (response.isDigestQuery())
> size += response.digest().remaining();
> else
> size += Row.serializer().serializedSize(response.row(), version);
> return size;
> }
> adds the digest size 2 times
> this triggers assertion error in at least ReadVerbHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira