You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2012/05/15 23:07:09 UTC
[jira] [Created] (CASSANDRA-4247) More-efficient serialization for
IndexScanCommand and RangeSliceCommand
Jonathan Ellis created CASSANDRA-4247:
-----------------------------------------
Summary: More-efficient serialization for IndexScanCommand and RangeSliceCommand
Key: CASSANDRA-4247
URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
Project: Cassandra
Issue Type: Improvement
Reporter: Jonathan Ellis
Assignee: Yuki Morishita
Priority: Minor
These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4247) More-efficient serialization for
IndexScanCommand and RangeSliceCommand
Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4247:
--------------------------------------
Attachment: 0002-Fix-SerializationsTest.patch
0001-De-serialize-thrift-object-directly-from-to-stream.patch
Patch attached to directly de/serialize from/to stream.
Not related to this ticket, but I also attached SerializationsTest fixes.
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4247) More-efficient serialization
for IndexScanCommand and RangeSliceCommand
Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280226#comment-13280226 ]
Yuki Morishita commented on CASSANDRA-4247:
-------------------------------------------
Ah, your approach is better. :)
But I think there is a problem in serialization order. When performing range slice, first we check whether SlicePredicate's column_names is specified, and if it is we ignore slice_range. So we need to check column_names != null and serialize column_names first instead of range != null and serialize range.
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4247) More-efficient serialization
for IndexScanCommand and RangeSliceCommand
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279621#comment-13279621 ]
Jonathan Ellis commented on CASSANDRA-4247:
-------------------------------------------
Pushed to https://github.com/jbellis/cassandra/branches/4247-3 with some additional cleanup of versioning that is obsolete post-CASSANDRA-3617
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4247) More-efficient serialization for
IndexScanCommand and RangeSliceCommand
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4247:
--------------------------------------
Attachment: (was: 4247-v2.txt)
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4247) More-efficient
serialization for IndexScanCommand and RangeSliceCommand
Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280226#comment-13280226 ]
Yuki Morishita edited comment on CASSANDRA-4247 at 5/21/12 3:44 PM:
--------------------------------------------------------------------
Ah, your approach is better. :)
But I think there is a problem in serialization order. When performing range slice, first we check whether SlicePredicate's column_names is specified, and if it is we ignore slice_range. So we need to check column_names != null and serialize column_names first instead of range != null and serialize range. If we specified both range and column names in predicate, we would get unexpected result.
was (Author: yukim):
Ah, your approach is better. :)
But I think there is a problem in serialization order. When performing range slice, first we check whether SlicePredicate's column_names is specified, and if it is we ignore slice_range. So we need to check column_names != null and serialize column_names first instead of range != null and serialize range.
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4247) More-efficient serialization
for IndexScanCommand and RangeSliceCommand
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280270#comment-13280270 ]
Jonathan Ellis commented on CASSANDRA-4247:
-------------------------------------------
Isn't this what we're checking for in ThriftValidation?
{code}
. if (predicate.column_names == null && predicate.slice_range == null)
throw new InvalidRequestException("predicate column_names and slice_range may not both be null");
if (predicate.column_names != null && predicate.slice_range != null)
throw new InvalidRequestException("predicate column_names and slice_range may not both be present");
{code}
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4247) More-efficient serialization
for IndexScanCommand and RangeSliceCommand
Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280296#comment-13280296 ]
Yuki Morishita commented on CASSANDRA-4247:
-------------------------------------------
Oh, I missed that part. +1 then.
You also need to update test/data/1.2/ by doing {{ant msg-ser-gen-test}} since serialized format is changed for RSC. But that ant command fails because there is an error in MerkleTree serializedSize, so I pushed fix and new version of test data here: https://github.com/yukim/cassandra/commits/4247-4
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4247) More-efficient serialization
for IndexScanCommand and RangeSliceCommand
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279585#comment-13279585 ]
Jonathan Ellis commented on CASSANDRA-4247:
-------------------------------------------
v2 wasn't fully baked yet... taking another stab at it
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4247) More-efficient serialization for
IndexScanCommand and RangeSliceCommand
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4247:
--------------------------------------
Attachment: 4247-v2.txt
Committed 0002.
0001 gets us about halfway there; on the write path for RSCS, serialize(expr).length in serializedSize still has to write it out to a byte[] that we promptly discard.
v2 cuts Thrift serialization out entirely so we can calculate size without creating a byte[] copy.
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-4247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch, 4247-v2.txt
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira