You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/12/18 20:38:18 UTC
[jira] Created: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
replace RowMutation w/ serialized thrift structs
------------------------------------------------
Key: CASSANDRA-645
URL: https://issues.apache.org/jira/browse/CASSANDRA-645
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Jonathan Ellis
Priority: Minor
Fix For: 0.9
using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793667#action_12793667 ]
Jonathan Ellis commented on CASSANDRA-645:
------------------------------------------
This does use thrift more but (a) that is still better than hand-rolling serialization, and (b) using strictly the serialization properties means it's easy to replace with any other serialization lib that supports java; it's the combination of RPC support and lots of languages that makes it hard to switch on the client side.
> we need a way to accumulate ColumnFamily objects that contain changes before they are committed
that is what Memtable / CF does. the only difference is you need a CF.add(ThriftMutation) method. Not a big deal.
> and we need a single blob-like thing for the commit log to store
Whatever key-based unit you send to the data nodes, can be re-used here. I don't see the problem.
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798168#action_12798168 ]
Jonathan Ellis commented on CASSANDRA-645:
------------------------------------------
(I still think it was worth trying, though; thanks to Gary for putting the patch together.)
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Priority: Minor
> Attachments: 645-thrift.patch, 645-v1.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796346#action_12796346 ]
Jonathan Ellis commented on CASSANDRA-645:
------------------------------------------
for future reference, this would have been far easier to review if it were structured something like
01: restructure StorageProxy mutate -> mutateLocal etc.
02: move helper methods out of RowMutation into MutationHelper
03: actually replace RowMutation w/ TableMutation et al
i.e., keep distinct refactors distinct.
(git can help with this: when you realize you need a new piece of the puzzle, you can commit your current work, then git co -b 645-2 and separate it out better there, with the safety net of being able to go back to your original branch and try again if it goes poorly.)
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch, 645-v1.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793416#action_12793416 ]
Gary Dusbabek commented on CASSANDRA-645:
-----------------------------------------
I still don't get where the extra copy is happening. Are you referring to serialization/deserialization operations? Do you mean that the creation of RowMutation should be pushed down to be as near to the actual write as possible?
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793478#action_12793478 ]
Jonathan Ellis commented on CASSANDRA-645:
------------------------------------------
Currently it looks like this:
bytes on wire -> Thrift Mutation on coordinator node -> RowMutation on CN -> bytes on wire -> RM on data node
proposing
bytes on wire -> Thrift Mutation on coordinator node -> bytes on wire -> TM on data node
skipping copying/morphing the TM into RM
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-645:
------------------------------------
Attachment: 645-v1.patch
Replaced RowMutation with a thrift struct.
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch, 645-v1.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793661#action_12793661 ]
Gary Dusbabek commented on CASSANDRA-645:
-----------------------------------------
How deep should I maintain the thrift structures? Unless we're willing to carry this all the way into ColumnFamilyStore and Memtable (and I think we are not willing), we need a way to accumulate ColumnFamily objects that contain changes before they are committed (that and we need a single blob-like thing for the commit log to store).
RowMutation was handy for this.
The best idea I have a "TableMutation" that accepts batches of mutations, converts them into the corresponding ColumnFamily changes and then sends them off in a manner similar to Table.apply(), but this isn't far off from the "RowMutation on CN" step above.
BTW, it looks like we needlessly reserialze the RMs during commit log recovery.
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek reassigned CASSANDRA-645:
---------------------------------------
Assignee: Gary Dusbabek
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-645:
-------------------------------------
Attachment: 645-thrift.patch
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-645) replace RowMutation w/ serialized
thrift structs
Posted by "Eric Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793656#action_12793656 ]
Eric Evans commented on CASSANDRA-645:
--------------------------------------
Is this going to further entrench thrift? Please don't further entrench thrift.
> replace RowMutation w/ serialized thrift structs
> ------------------------------------------------
>
> Key: CASSANDRA-645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-645
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 645-thrift.patch
>
>
> using the client's representation internally will save basically a copy operation on the critical write path
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.