You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Vijay (Created) (JIRA)" <ji...@apache.org> on 2012/04/11 21:31:16 UTC
[jira] [Created] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Add varint encoding to Serializing Cache
----------------------------------------
Key: CASSANDRA-4138
URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
Project: Cassandra
Issue Type: Sub-task
Reporter: Vijay
Assignee: Vijay
Priority: Minor
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4138) Add varint encoding
to Serializing Cache
Posted by "Vijay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255193#comment-13255193 ]
Vijay edited comment on CASSANDRA-4138 at 4/16/12 11:57 PM:
------------------------------------------------------------
Hi Pavel, attached patch has recommended changes except
{quote}
I think the DBContants class is now should be changed to only share sizeof(type) methods and become something like DBContants.{native, vint}.sizeof(type)
{quote}
I will mark it private once parent ticket is complete (Messaging and SSTable formats), currently we have it called in other places too.
was (Author: vijay2win@yahoo.com):
Hi Pavel, attached patch has recommended changes except
{comment}
I think the DBContants class is now should be changed to only share sizeof(type) methods and become something like DBContants.{native, vint}.sizeof(type)
{comment}
I will mark it private once parent ticket is complete (Messaging and SSTable formats), currently we have it called in other places too.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4138) Add varint encoding
to Serializing Cache
Posted by "Vijay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255193#comment-13255193 ]
Vijay edited comment on CASSANDRA-4138 at 4/16/12 11:58 PM:
------------------------------------------------------------
Hi Pavel, attached patch has recommended changes except
>>> I think the DBContants class is now should be changed to only share sizeof(type) methods and become something like DBContants.{native, vint}.sizeof(type)
I will mark it private once parent ticket is complete (Messaging and SSTable formats), currently we have it called in other places too.
was (Author: vijay2win@yahoo.com):
Hi Pavel, attached patch has recommended changes except
{quote}
I think the DBContants class is now should be changed to only share sizeof(type) methods and become something like DBContants.{native, vint}.sizeof(type)
{quote}
I will mark it private once parent ticket is complete (Messaging and SSTable formats), currently we have it called in other places too.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264084#comment-13264084 ]
Jonathan Ellis commented on CASSANDRA-4138:
-------------------------------------------
It's not immediately clear to me what the changes in ByteBufferUtil are doing -- EDOS doesn't change writeByte so what is breaking? is this backwards compatible?
Have you done any smoke tests to see what kind of savings you get on typical cached data? In other words: is our intuition correct that this is worth the extra complexity?
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255195#comment-13255195 ]
Pavel Yaskevich commented on CASSANDRA-4138:
--------------------------------------------
You can drop changing *_SIZE constants to private in this one but nothing is keeping us from implementing the later part of the proposal, all your changes would use sizeof(type) methods without need to change the rest of the code.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4138) Add varint encoding
to Serializing Cache
Posted by "Pavel Yaskevich (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255143#comment-13255143 ]
Pavel Yaskevich edited comment on CASSANDRA-4138 at 4/16/12 10:55 PM:
----------------------------------------------------------------------
To avoid confusion related to naming of {write, read}VLong methods (as it gives a feeling that writeInt doesn't really write an int anymore) in the EDIS and EDOS I propose to rename them to VInt{Encode, Decode}. Furthermore, we could give a better feel of the encoding used by adding {VInt} as a prefix to both classes (as an alternative they could be moved to o.a.c.u.vint package), also I think the DBContants class is now should be changed to only share sizeof(type) methods and become something like DBContants.{native, vint}.sizeof(type)...
was (Author: xedin):
To avoid confusion related to naming of {write, read}VLong methods (as it gives a feeling that writeInt doesn't really write an int anymore) in the EDIS and EDOS I propose to rename them to {encode, decode}VInt. Furthermore, we could give a better feel of the encoding used by adding {VInt} as a prefix to both classes (as an alternative they could be moved to o.a.c.u.vint package), also I think the DBContants class is now should be changed to only share sizeof(type) methods and become something like DBContants.{native, encoded}.sizeof(type)...
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4138:
-----------------------------
Attachment: 0002-sizeof-changes-on-rest-of-the-code.patch
Attached patch changes the DBConstants to private and changes other unrelated code to use sizeof instead of constants.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0002-sizeof-changes-on-rest-of-the-code.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Pavel Yaskevich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4138:
---------------------------------------
Reviewer: jbellis
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4138:
-----------------------------
Attachment: (was: 0001-CASSANDRA-4138-Take1.patch)
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255143#comment-13255143 ]
Pavel Yaskevich commented on CASSANDRA-4138:
--------------------------------------------
To avoid confusion related to naming of {write, read}VLong methods (as it gives a feeling that writeInt doesn't really write an int anymore) in the EDIS and EDOS I propose to rename them to {encode, decode}VInt. Furthermore, we could give a better feel of the encoding used by adding {VInt} as a prefix to both classes (as an alternative they could be moved to o.a.c.u.vint package), also I think the DBContants class is now should be changed to only share sizeof(type) methods and become something like DBContants.{native, encoded}.sizeof(type)...
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253790#comment-13253790 ]
Vijay commented on CASSANDRA-4138:
----------------------------------
I did try it before copying.... the problem is that we cannot override writeInt() writeLong() and writeShort() because they are marked as final.
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/DataOutputStream.java#DataOutputStream.writeInt%28int%29
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Pavel Yaskevich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4138:
---------------------------------------
Attachment: CASSANDRA-4138-v3.patch
Made some refactoring (added license headers, renamed variables etc.)
I have removed o.a.c.util package that you created and moved Encoded* clases to o.a.c.utils.vint, done the same for tests.
Renamed DBContants to DBTypeSizes with two statics NATIVE and VINT to be used for sizeof(type) + all related code.
Moved class EncodedDBConstant to the DBTypeSized and renamed it to VIntEncodedDBTypeSizes, the same done for native - it's NativeDBTypeSizes now.
Changed {encode, decode}VInt methods visibility to private and renamed to vint{Encode, Decode} + added a comment why we use single method for all integer types.
One problem remains - CompactionsTest fails after your modifications.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4138:
-----------------------------
Attachment: 0001-CASSANDRA-4138-V2.patch
Hi Pavel, attached patch has recommended changes except
{comment}
I think the DBContants class is now should be changed to only share sizeof(type) methods and become something like DBContants.{native, vint}.sizeof(type)
{comment}
I will mark it private once parent ticket is complete (Messaging and SSTable formats), currently we have it called in other places too.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4138:
-----------------------------
Attachment: 0001-CASSANDRA-4138-Take1.patch
Attached patch is the first attempt to add VarInt Encoding to cassandra.
It save's us around 10% of the memory compared to normal DataInputStream.
Once this gets committed i will work on the rest of the pieces.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4138:
-----------------------------
Attachment: 0001-CASSANDRA-4138-v4.patch
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4138) Add varint encoding
to Serializing Cache
Posted by "Vijay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253028#comment-13253028 ]
Vijay edited comment on CASSANDRA-4138 at 4/13/12 1:04 AM:
-----------------------------------------------------------
Attached patch is the first attempt to add VarInt Encoding to cassandra.
It save's us around 10% of the memory compared to normal DataInputStream. (based on a simple test via Stress Tool)
Once this gets committed i will work on the rest of the pieces.
was (Author: vijay2win@yahoo.com):
Attached patch is the first attempt to add VarInt Encoding to cassandra.
It save's us around 10% of the memory compared to normal DataInputStream.
Once this gets committed i will work on the rest of the pieces.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264191#comment-13264191 ]
Jonathan Ellis commented on CASSANDRA-4138:
-------------------------------------------
LGTM, +1.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4138:
-----------------------------
Attachment: 0001-CASSANDRA-4138-v4.patch
v4 fixes the test failures (BBU.readShort to readUnsignedShort) and also fixes the autoboxing in the verbs which was there in earlier versions...
Thanks!
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264110#comment-13264110 ]
Vijay commented on CASSANDRA-4138:
----------------------------------
>>> Have you done any smoke tests to see what kind of savings you get on typical cached data?
Yes i do see 10% gain in space.
>>> is our intuition correct that this is worth the extra complexity?
The complexity for the Cache as such is only using EDOS instead of DOS... other complexity are generic to support messaging and SSTable formats.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4138:
-----------------------------
Attachment: 0001-CASSANDRA-4138-Take1.patch
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264118#comment-13264118 ]
Pavel Yaskevich commented on CASSANDRA-4138:
--------------------------------------------
Yeah, this is my intended clarification of why is that needed but I somehow missed it :(
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264115#comment-13264115 ]
Vijay commented on CASSANDRA-4138:
----------------------------------
In addition to Pavel's comment, the reason for the change is to make the writeWithShortLength write a varint while using EDOS (will just use a byte instead of 2 for most of the cases).
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4138:
-----------------------------
Attachment: (was: 0001-CASSANDRA-4138-v4.patch)
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253906#comment-13253906 ]
Vijay commented on CASSANDRA-4138:
----------------------------------
Hi Jonathan, Nope this ticket was just an extension of the parent, i wanted to get the lhf first :)... Once we have EDOS we can use it for
IVersionedSerializer.serialize(T t, EncodedDataOutput dos, int version)
and
OutboundTcpConnection.write(Message message, String id, EncodedDataOutput out)
Which will be easier to deal with the version too (and with SST versioning)... Hope it makes sense.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253895#comment-13253895 ]
Jonathan Ellis commented on CASSANDRA-4138:
-------------------------------------------
If we're relying on overriding writeInt etc, does that mean we're giving up being able to use varints over the network? (Not a concern for the cache obviously but I'm thinking ahead to CASSANDRA-3024.)
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257089#comment-13257089 ]
Pavel Yaskevich commented on CASSANDRA-4138:
--------------------------------------------
Ok, let's give a Jonathan chance to make a final look.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253761#comment-13253761 ]
Jonathan Ellis commented on CASSANDRA-4138:
-------------------------------------------
Couldn't we just extend DataOutputStream instead of introducing ADOS?
(Pretty sure AbstractDataInput was a mistake as well.)
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4138) Add varint encoding to
Serializing Cache
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264108#comment-13264108 ]
Pavel Yaskevich commented on CASSANDRA-4138:
--------------------------------------------
bq. It's not immediately clear to me what the changes in ByteBufferUtil are doing – EDOS doesn't change writeByte so what is breaking? is this backwards compatible?
I can explain ByteBufferUtil changes - instead of doing short write manually (copy code that does it from DO) it just uses appropriate method from the DataOutput which would handle the short encode and write. EDOS don't really need to change the way we write bytes, this is only about encoding integer types compactly. All legacy tests are passing and this doesn't seem to touch code so dramatically to break it as soon as it's all about SerializingCache...
bq. Have you done any smoke tests to see what kind of savings you get on typical cached data? In other words: is our intuition correct that this is worth the extra complexity?
This is question to Vijay so I won't interfere, just want to note that he pointed out in the first comment that this saves ~10% of memory compared to normal DIS.
> Add varint encoding to Serializing Cache
> ----------------------------------------
>
> Key: CASSANDRA-4138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4138
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4138-Take1.patch, 0001-CASSANDRA-4138-V2.patch, 0001-CASSANDRA-4138-v4.patch, 0002-sizeof-changes-on-rest-of-the-code.patch, CASSANDRA-4138-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira