You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2012/05/15 23:07:09 UTC

[jira] [Created] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Jonathan Ellis created CASSANDRA-4247:
-----------------------------------------

             Summary: More-efficient serialization for IndexScanCommand and RangeSliceCommand
                 Key: CASSANDRA-4247
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
             Project: Cassandra
          Issue Type: Improvement
            Reporter: Jonathan Ellis
            Assignee: Yuki Morishita
            Priority: Minor


These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.

We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yuki Morishita updated CASSANDRA-4247:
--------------------------------------

    Attachment: 0002-Fix-SerializationsTest.patch
                0001-De-serialize-thrift-object-directly-from-to-stream.patch

Patch attached to directly de/serialize from/to stream.

Not related to this ticket, but I also attached SerializationsTest fixes.
                
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280226#comment-13280226 ] 

Yuki Morishita commented on CASSANDRA-4247:
-------------------------------------------

Ah, your approach is better. :)

But I think there is a problem in serialization order. When performing range slice, first we check whether SlicePredicate's column_names is specified, and if it is we ignore slice_range. So we need to check column_names != null and serialize column_names first instead of range != null and serialize range.
                
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.2
>
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279621#comment-13279621 ] 

Jonathan Ellis commented on CASSANDRA-4247:
-------------------------------------------

Pushed to https://github.com/jbellis/cassandra/branches/4247-3 with some additional cleanup of versioning that is obsolete post-CASSANDRA-3617
                
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.2
>
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-4247:
--------------------------------------

    Attachment:     (was: 4247-v2.txt)
    
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.2
>
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Comment Edited] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280226#comment-13280226 ] 

Yuki Morishita edited comment on CASSANDRA-4247 at 5/21/12 3:44 PM:
--------------------------------------------------------------------

Ah, your approach is better. :)

But I think there is a problem in serialization order. When performing range slice, first we check whether SlicePredicate's column_names is specified, and if it is we ignore slice_range. So we need to check column_names != null and serialize column_names first instead of range != null and serialize range. If we specified both range and column names in predicate, we would get unexpected result.
                
      was (Author: yukim):
    Ah, your approach is better. :)

But I think there is a problem in serialization order. When performing range slice, first we check whether SlicePredicate's column_names is specified, and if it is we ignore slice_range. So we need to check column_names != null and serialize column_names first instead of range != null and serialize range.
                  
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.2
>
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280270#comment-13280270 ] 

Jonathan Ellis commented on CASSANDRA-4247:
-------------------------------------------

Isn't this what we're checking for in ThriftValidation?

{code}
.       if (predicate.column_names == null && predicate.slice_range == null)
            throw new InvalidRequestException("predicate column_names and slice_range may not both be null");
        if (predicate.column_names != null && predicate.slice_range != null)
            throw new InvalidRequestException("predicate column_names and slice_range may not both be present");
{code}
                
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.2
>
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280296#comment-13280296 ] 

Yuki Morishita commented on CASSANDRA-4247:
-------------------------------------------

Oh, I missed that part. +1 then.

You also need to update test/data/1.2/ by doing {{ant msg-ser-gen-test}} since serialized format is changed for RSC. But that ant command fails because there is an error in MerkleTree serializedSize, so I pushed fix and new version of test data here: https://github.com/yukim/cassandra/commits/4247-4
                
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.2
>
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279585#comment-13279585 ] 

Jonathan Ellis commented on CASSANDRA-4247:
-------------------------------------------

v2 wasn't fully baked yet... taking another stab at it
                
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.2
>
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (CASSANDRA-4247) More-efficient serialization for IndexScanCommand and RangeSliceCommand

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-4247:
--------------------------------------

    Attachment: 4247-v2.txt

Committed 0002.

0001 gets us about halfway there; on the write path for RSCS, serialize(expr).length in serializedSize still has to write it out to a byte[] that we promptly discard.

v2 cuts Thrift serialization out entirely so we can calculate size without creating a byte[] copy.
                
> More-efficient serialization for IndexScanCommand and RangeSliceCommand
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-4247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4247
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 1.2
>
>         Attachments: 0001-De-serialize-thrift-object-directly-from-to-stream.patch, 0002-Fix-SerializationsTest.patch, 4247-v2.txt
>
>
> These two classes embed a Thrift struct to simplify de/serialization. This means we serialize to byte[] twice on the write path: once to get the size, and again to actually send the message.
> We should serialize manually like the other commands in VERSION_12.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira