You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Vijay (Created) (JIRA)" <ji...@apache.org> on 2012/04/11 21:31:16 UTC
[jira] [Created] (CASSANDRA-4139) Add varint encoding to Messaging
service
Add varint encoding to Messaging service
----------------------------------------
Key: CASSANDRA-4139
URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
Project: Cassandra
Issue Type: Sub-task
Reporter: Vijay
Assignee: Vijay
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276241#comment-13276241 ]
Jonathan Ellis commented on CASSANDRA-4139:
-------------------------------------------
bq. the other option is to write constant 4 bytes just for the header, If it sounds reasonable i can rewrite it?
That would basically entail adding a writeFixedWidth(int) method to EDO, right? That sounds reasonable to me.
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276230#comment-13276230 ]
Vijay commented on CASSANDRA-4139:
----------------------------------
Hi Jonathan,
The problem is the backward compatibility part of it...
We do out.writeInt(header) which is used IncomingTcpConnection (header = input.readInt()) for the older versions the header will be corrupted...
the other option is to write constant 4 bytes just for the header, If it sounds reasonable i can rewrite it?
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4139) Add varint encoding to Messaging
service
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4139:
-----------------------------
Attachment: 0001-CASSANDRA-4139-v2.patch
Sorry for the delay, writing the patch was the easy part, fixing the test cases took a while :)
In Addition to the patch there is a fixes for the sizeof calculations and to TypeSize.sizeof(String) does constant 2 bytes instead of short.
Plz note we have to regenerate the SearalizationTest's files.
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280671#comment-13280671 ]
Jonathan Ellis commented on CASSANDRA-4139:
-------------------------------------------
Why not just reconnect on version change? (Only new-version nodes need to worry about this, since old-version nodes always send the same version.)
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280588#comment-13280588 ]
Jonathan Ellis commented on CASSANDRA-4139:
-------------------------------------------
bq. which will change over the period of a connection
how's that?
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4139) Add varint encoding to Messaging
service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4139:
--------------------------------------
Fix Version/s: (was: 1.2.0)
1.3
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.3
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v4.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf, ASF.LICENSE.NOT.GRANTED--0001-CASSANDRA-4139-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4139) Add varint encoding to Messaging
service
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4139:
-----------------------------
Attachment: 4139-Test.rtf
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280607#comment-13280607 ]
Vijay commented on CASSANDRA-4139:
----------------------------------
Trunk: we start with the latest version (OTCPC) and the lower versions in ITCPC (will start ignoring messages). Once we detect the right version we change the version of the message sent.
OutputTCPConnection
{code}
public void write(MessageOut<?> message, String id, DataOutputStream out) throws IOException
{
write(message, id, out, Gossiper.instance.getVersion(poolReference.endPoint()));
}
{code}
IncomingTcpConnection
{code}
MessagingService.validateMagic(input.readInt());
header = input.readInt();
assert isStream == (MessagingService.getBits(header, 3, 1) == 1) : "Connections cannot change type: " + isStream;
version = MessagingService.getBits(header, 15, 8);
logger.trace("Version is now {}", version);
receiveMessage(input, version);
{code}
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410003#comment-13410003 ]
Jonathan Ellis commented on CASSANDRA-4139:
-------------------------------------------
rebased and added some modifications at https://github.com/jbellis/cassandra/tree/4139-5. untested but hopefully shows a reasonable approach.
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0001-CASSANDRA-4139-v4.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453136#comment-13453136 ]
Jonathan Ellis commented on CASSANDRA-4139:
-------------------------------------------
(Vijay reports that he still has problems getting my branch to work over SSL.)
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.3
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v4.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf, ASF.LICENSE.NOT.GRANTED--0001-CASSANDRA-4139-v3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276224#comment-13276224 ]
Jonathan Ellis commented on CASSANDRA-4139:
-------------------------------------------
Sorry, didn't think of this the first time around...
What if we do the EncodedDataOutput wrapping in OutboundTcpConnection? That way we wouldn't need to re-wrap for every object we serialize. And the encoding does feel more like a connection-level thing than object-level.
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280580#comment-13280580 ]
Vijay commented on CASSANDRA-4139:
----------------------------------
* Tried the implementation, We cannot have it in the socket level because of the backward compatibility, we should wrap the connection only if the version is > 1.2 (which will change over the period of a connection).
* One thing which we can do is to cache the object which wrapped around but the wrapper is light weight so it should'nt be a lot of over head.
* In addition to this I noticed some wired problem (connection disconnects while reading, which is not the case when we are not caching) which some how breaks if i cache the objects and while communicating via SSL (i pushed the code to https://github.com/Vijay2win/cassandra/commit/ebf54b3df0419d6a4305aa4b8813e351d4ba7188#L1R176 to reproduce change FBUtilities.getDataOutput to getCachedDataOut)
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280677#comment-13280677 ]
Vijay commented on CASSANDRA-4139:
----------------------------------
Sure we can (Even better we can let the new version disconnect the old connection in that way we dont need to do ConcurrentMap.get for every message), but we still have that wired problem mentioned above :(
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4139) Add varint encoding to
Messaging service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270862#comment-13270862 ]
Jonathan Ellis commented on CASSANDRA-4139:
-------------------------------------------
Can you rebase? Finally got CASSANDRA-3617 committed, which conflicts.
(Probably useful: I added TypeSizes.sizeof(String) to supplement the raw encodedUTF8Length.)
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4139) Add varint encoding to Messaging
service
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4139:
-----------------------------
Attachment: 0001-CASSANDRA-4139-v4.patch
Rebased, Unit and functional tests passes and i have pushed the regenerated test binaries to https://github.com/Vijay2win/cassandra/tree/4139-v4
I have also created CASSANDRA-4357 to add commit log versioning to handle the unencoded data in the commit logs.
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4139-v3.patch, 0001-CASSANDRA-4139-v4.patch, 0002-add-bytes-written-metric.patch, 4139-Test.rtf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4139) Add varint encoding to Messaging
service
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4139:
-----------------------------
Attachment: 0002-add-bytes-written-metric.patch
0001-CASSANDRA-4139-v1.patch
0001 is the main patch, 0002 add's metrics for the measurement.
Attachement also has the smoke test result, results show 10% saving of the total network bandwidth after this patch.
Note: Commit log is not versioned hence i made commit log to serialize in 1.1 format, we can get this done in another ticket.
> Add varint encoding to Messaging service
> ----------------------------------------
>
> Key: CASSANDRA-4139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4139
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
> Reporter: Vijay
> Assignee: Vijay
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4139-v1.patch, 0002-add-bytes-written-metric.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira