You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sylvain Lebresne (Updated) (JIRA)" <ji...@apache.org> on 2012/03/07 17:20:57 UTC
[jira] [Updated] (CASSANDRA-3957) Supercolumn serialization
assertion failure
[ https://issues.apache.org/jira/browse/CASSANDRA-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3957:
----------------------------------------
Attachment: 3957.txt
Here's my theory.
The stacktrace concerning the sblocks is due to a bug in DSE's CFS. This happened because CFS sometimes communicate with C* in the same JVM without going trough the wire, so the re-use of ByteBuffer was problematic.
The other stacktrace however can't be the same problem. Whatever Hector does, it goes over the wire and thus can't change the size of a ByteBuffer in C*.
However, if we eliminate the sblocks trace, all the other trace uses SuperColumn. And as it turns out, I think we have a race with SC. As Jonathan pointed out, with the period commit log, if we happen to modify a mutation that was sent to the commit log at any time, that would be a bug. This means not modifying the CF object in the RowMutation, but for SuperColumn, this also mean we shouldn't modify those super columns. However, when we update the row cache (and if it's not a SerializingCache), we do potentially store references to the original SCs in the cached row, which could then lead to the stack trace on this issue. I'm attaching a patch that fixes this.
To make sure this could be the issue here, it would help to know if the stacktrace comes from column families where row cache was used and was not set to the serializing cache.
I'll note however that the very small difference in the stacktraces between the predicted size and the actual serialized side support the theory of having say just one column of a super column updated, while in the sblocks case, the difference was much bigger, supporting the theory of the ByteBuffer content being changed to something completely different.
> Supercolumn serialization assertion failure
> -------------------------------------------
>
> Key: CASSANDRA-3957
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3957
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 3957.txt
>
>
> As reported at http://mail-archives.apache.org/mod_mbox/cassandra-user/201202.mbox/%3CCADJL=w5kH5TEQXOwhTn5Jm3cmR4Rj=NfjcqLryXV7pLyASi95A@mail.gmail.com%3E,
> {noformat}
> ERROR 10:51:44,282 Fatal exception in thread
> Thread[COMMIT-LOG-WRITER,5,main]
> java.lang.AssertionError: Final buffer length 4690 to accomodate data size
> of 2347 (predicted 2344) for RowMutation(keyspace='Player',
> key='36336138643338652d366162302d343334392d383466302d356166643863353133356465',
> modifications=[ColumnFamily(PlayerCity [SuperColumn(owneditem_1019
> []),SuperColumn(owneditem_1024 []),SuperColumn(owneditem_1026
> []),SuperColumn(owneditem_1074 []),SuperColumn(owneditem_1077
> []),SuperColumn(owneditem_1084 []),SuperColumn(owneditem_1094
> []),SuperColumn(owneditem_1130 []),SuperColumn(owneditem_1136
> []),SuperColumn(owneditem_1141 []),SuperColumn(owneditem_1142
> []),SuperColumn(owneditem_1145 []),SuperColumn(owneditem_1218
> [636f6e6e6563746564:false:5@1329648704269002
> ,63757272656e744865616c7468:false:3@1329648704269006
> ,656e64436f6e737472756374696f6e54696d65:false:13@1329648704269007
> ,6964:false:4@1329648704269000,6974656d4964:false:15@1329648704269001
> ,6c61737444657374726f79656454696d65:false:1@1329648704269008
> ,6c61737454696d65436f6c6c6563746564:false:13@1329648704269005
> ,736b696e4964:false:7@1329648704269009,78:false:4@1329648704269003
> ,79:false:3@1329648704269004,]),SuperColumn(owneditem_133
> []),SuperColumn(owneditem_134 []),SuperColumn(owneditem_135
> []),SuperColumn(owneditem_141 []),SuperColumn(owneditem_147
> []),SuperColumn(owneditem_154 []),SuperColumn(owneditem_159
> []),SuperColumn(owneditem_171 []),SuperColumn(owneditem_253
> []),SuperColumn(owneditem_422 []),SuperColumn(owneditem_438
> []),SuperColumn(owneditem_515 []),SuperColumn(owneditem_521
> []),SuperColumn(owneditem_523 []),SuperColumn(owneditem_525
> []),SuperColumn(owneditem_562 []),SuperColumn(owneditem_61
> []),SuperColumn(owneditem_634 []),SuperColumn(owneditem_636
> []),SuperColumn(owneditem_71 []),SuperColumn(owneditem_712
> []),SuperColumn(owneditem_720 []),SuperColumn(owneditem_728
> []),SuperColumn(owneditem_787 []),SuperColumn(owneditem_797
> []),SuperColumn(owneditem_798 []),SuperColumn(owneditem_838
> []),SuperColumn(owneditem_842 []),SuperColumn(owneditem_847
> []),SuperColumn(owneditem_849 []),SuperColumn(owneditem_851
> []),SuperColumn(owneditem_852 []),SuperColumn(owneditem_853
> []),SuperColumn(owneditem_854 []),SuperColumn(owneditem_857
> []),SuperColumn(owneditem_858 []),SuperColumn(owneditem_874
> []),SuperColumn(owneditem_884 []),SuperColumn(owneditem_886
> []),SuperColumn(owneditem_908 []),SuperColumn(owneditem_91
> []),SuperColumn(owneditem_911 []),SuperColumn(owneditem_930
> []),SuperColumn(owneditem_934 []),SuperColumn(owneditem_937
> []),SuperColumn(owneditem_944 []),SuperColumn(owneditem_945
> []),SuperColumn(owneditem_962 []),SuperColumn(owneditem_963
> []),SuperColumn(owneditem_964 []),])])
> at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
> at org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:279)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122)
> at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:599)
> at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira