You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (Created) (JIRA)" <ji...@apache.org> on 2011/10/07 21:16:29 UTC
[jira] [Created] (CASSANDRA-3333) remove more copies from
read/write network path
remove more copies from read/write network path
-----------------------------------------------
Key: CASSANDRA-3333
URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 1.0.1
RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3333:
--------------------------------------
Attachment: 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch
updated 02 w/ fixes
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3333:
--------------------------------------
Attachment: 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch
New version of patch 2, covering the rest of the serializations on the read and write path. Extracted serialize-and-assert-it-matched-claimed-size code into FBUtilities.serialize.
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch, 0002-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3333:
--------------------------------------
Attachment: (was: 0002-reduce-copies.patch)
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126039#comment-13126039 ]
Brandon Williams commented on CASSANDRA-3333:
---------------------------------------------
+1, approximate 10% gain on both reads and writes.
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123225#comment-13123225 ]
Brandon Williams commented on CASSANDRA-3333:
---------------------------------------------
+1
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3333:
--------------------------------------
Attachment: 0002-reduce-copies.patch
0001-clean-up-Serializer-mess.patch
01 cleans up the serializer snafu we've been living with
02 actually does the optimization
Prior to this patch we had three generic serializers:
ICompactSerializer, which is used by Messages, and whose de/serialize methods have version parameters
ICompactSerializer2, which does not have versions and whose methods have DataInput/Output parameters instead of Input/OutputStreams. (The former is a superset of the later.)
ICompactSerializer3, which extends ICS2 and adds serializedSize.
To do this optimization I'd need ICS4, since I want serializedSize AND versioning.
Instead, I've replaced the three old interfaces with just ISerializer and IVersionedSerializer. Both use DataInput/Output, and both have a serializedSize method; the difference is that the former does not have a version parameter. Implementers that don't care about precomputing size simply implement as UnsupportedOperation.
A few serializers (in LegacyBloomFilter and MerkleTree) still needed a Stream parameter because they are doing JDK serialization under the hood. None of that code actually cares about using a generic Serializer interface, so I just made them one-off classes and everything was happy.
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125011#comment-13125011 ]
Brandon Williams commented on CASSANDRA-3333:
---------------------------------------------
Doesn't quite work:
{noformat}
INFO 08:19:01,875 Creating new commitlog segment /var/lib/cassandra/commitlog/CommitLog-1318339141875.log
ERROR 08:19:01,884 Fatal exception in thread Thread[COMMIT-LOG-WRITER,5,main]
java.lang.AssertionError
at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:782)
at org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:278)
at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122)
at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:605)
at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:662)
{noformat}
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch, 0002-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3333:
--------------------------------------
Attachment: 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch, 0002-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3333:
--------------------------------------
Attachment: (was: 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch)
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3333) remove more copies from
read/write network path
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3333:
--------------------------------------
Attachment: (was: 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch)
> remove more copies from read/write network path
> -----------------------------------------------
>
> Key: CASSANDRA-3333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3333
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: network
> Fix For: 1.0.1
>
> Attachments: 0001-clean-up-Serializer-mess.patch, 0002-introduce-FBUtilities.serialize-to-reduce-copies.patch, 0002-reduce-copies.patch
>
>
> RowMutation.serializedBuffer and ReadVerbHandler both do an extra copy of the serialized data. We can avoid this be pre-computing the serialized size and allocating an appropriate buffer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira