You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sylvain Lebresne (JIRA)" <ji...@apache.org> on 2011/01/05 16:42:46 UTC
[jira] Created: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Fit partitioned counter directly into CounterColumn.value
----------------------------------------------------------
Key: CASSANDRA-1936
URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Fix For: 0.8
The current implementation of CounterColumn keeps both the partitioned
counter and the total value of the counter (that is, the sum of the parts of
the partitioned counter).
This waste space and this requires the code to keep both representation in
sync. This ticket propose to remove the total value from the representation
and to only calculate it when returning the value to the client.
NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1936:
----------------------------------------
Attachment: (was: 0001-Put-partitioned-counter-directly-in-column-value.patch)
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977802#action_12977802 ]
Jonathan Ellis commented on CASSANDRA-1936:
-------------------------------------------
can you break the bytebuffer fixes out into a separate patch so we can apply to 0.7?
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch
>
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1936:
----------------------------------------
Attachment: (was: 0001-Put-partitioned-counter-directly-in-column-value.patch)
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1936:
----------------------------------------
Attachment: 0001-Put-partitioned-counter-directly-in-column-value.patch
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Kelvin Kakugawa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977959#action_12977959 ]
Kelvin Kakugawa commented on CASSANDRA-1936:
--------------------------------------------
The refactor to store the partitioned counter in the value, instead, is a good direction.
I noticed some material changes, though:
- client deltas are in the correct partitioned counter format, but targeted at the local node
- the RowMutation : updateCommutativeTypes was removed
The code works, now, as long as the coordinator node (the local node) is part of the replica set. However, if it's not, then all updates from those non-replica coordinators will be fixed at the highest delta. The reconciliation strategy (on a replica) is sum my node's updates, but take the highest update from all other nodes. (Just ran a distributed test to validate my hypothesis--the same test that's included on 1072.)
In the current code, value and partitioned counter are broken apart, because when the RowMutation is created we don't know which node we're going to write to, yet. So, we can't create the final partitioned counter. We could create a sentinel node that replicas need to look for, but that's dirty. The way I solved it was using value for the client delta and converting it to the partitioned counter (w/ the target node) via RM : updateCommutativeTypes.
I have an alternate proposal for this ticket, the second patch on 1072. I'll post it and we can take the best parts of each patch.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch
>
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1936:
----------------------------------------
Attachment: 0001-Put-partitioned-counter-directly-in-column-value.patch
Patch attached, a few comments:
* since this patch stuffs the context in the column value, who are
ByteBuffers, a good part of this patch deals with adapting the functions
of CounterContext to take and return ByteBuffers instead of plain byte[].
* the patch also corrects a few not completely related misuse of absolute
ByteBuffer's gets. Namely, there was a few occurrences where arrayOffset
was wrongly added to the provided index, like: bb.getLong(bb.position() +
bb.arrayOffset()).
* this patch breaks the on-disk file format. Since #1937 and #1938 will do
too, I'm fine with waiting that both are ready to commit all the 3 patches
together (I'm hoping to tackle these patches quickly).
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch
>
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977965#action_12977965 ]
Sylvain Lebresne commented on CASSANDRA-1936:
---------------------------------------------
You're right, don't know what I have smoked. Actually I was still thinking in the context of 1546, where it's always a replica that 'apply' the update (since the coordinator simply forward the update to a replica if its not one).
That being said, I do plan to introduce this part 1546, because it allowed to rehabilitate the consistency levels. So maybe I should do that before, in which case I think the method here would work.
Still, curious to see your patch. But I'll admit that I was actually happy to get rid of the updateCommutativeType logic.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch
>
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Kelvin Kakugawa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986702#action_12986702 ]
Kelvin Kakugawa commented on CASSANDRA-1936:
--------------------------------------------
I did basic testing, they all passed.
I'll try to fit in a look through of the code, this week.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1936:
----------------------------------------
Attachment: (was: 0001-Put-partitioned-counter-directly-in-column-value.patch)
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Kelvin Kakugawa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kelvin Kakugawa updated CASSANDRA-1936:
---------------------------------------
Attachment: 1936-ALT-0001-lazily-materialize-value.patch
Alternate strategy to lazily materialize value.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977807#action_12977807 ]
Sylvain Lebresne commented on CASSANDRA-1936:
---------------------------------------------
Oh right, forgot this was in 0.7 too. I'll do that.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch
>
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Kelvin Kakugawa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984506#action_12984506 ]
Kelvin Kakugawa commented on CASSANDRA-1936:
--------------------------------------------
Found a regression via distributed tests. No time to dig in, right now, but I'll get you more info when I do.
It's in the CounterContext : merge logic. There's a bad interaction w/ ByteBuffers, like it's trying to merge a badly formatted context.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Kelvin Kakugawa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977974#action_12977974 ]
Kelvin Kakugawa commented on CASSANDRA-1936:
--------------------------------------------
The partitionedCounter of my strategy could be refactored into value, like your strategy.
Yeah, unfortunately, the two-step RM creation (via updateCommutativeTypes) is still present.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Kelvin Kakugawa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987268#action_12987268 ]
Kelvin Kakugawa commented on CASSANDRA-1936:
--------------------------------------------
I looked through the code, it looks good.
I'm glad you removed that updateCommutativeTypes method and the, now irrelevant, update method.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987382#action_12987382 ]
Hudson commented on CASSANDRA-1936:
-----------------------------------
Integrated in Cassandra #694 (See [https://hudson.apache.org/hudson/job/Cassandra/694/])
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1936:
----------------------------------------
Attachment: 0001-Put-partitioned-counter-directly-in-column-value.patch
Attaching new patch rebased after #1944 (I've replaced my old patch).
It took a slightly different approach to the 'lazy materialization of value' in that it introduces a new type of (temporary) column (called CounterUpdateColumn - since it's a column holding a counter update - but if someone has a better idea for a name...) that holds the delta until it reaches a replica, at which time the CounterUpdateColumn is transformed to the equivalent CounterColumn.
I'm keen on this solution for two reasons:
# We don't have to keep a long delta in the CounterColumn that is useless as
soon as the column as been properly initialized.
# I have a patch for #1938 that is taking shape and I do have a use for this
CounterUpdateColumn. I'm also pretty sure it will be necessary when/if we
add the replay markers.
This is a reasonably big patch but this is in good part because of the move in CounterContext from byte[] to ByteBuffer.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984100#action_12984100 ]
Sylvain Lebresne commented on CASSANDRA-1936:
---------------------------------------------
Patch rebased after #2009
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1936:
----------------------------------------
Attachment: 0001-Put-partitioned-counter-directly-in-column-value.patch
Attaching new patch fixing 2 (stupid) problems related to direct ByteBuffers (thus it wasn't always showing): BBUtil.compareSubArrays() wasn't handling them at all and BBUtil.arrayCopy() used a gt instead of a lt when checking for out of bound index.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1936:
----------------------------------------
Comment: was deleted
(was: Was that with the test attached to #2005 ? I did not test the distributed tests yet,
but I ran a home-backed test (against a local 3 node cluster) doing a bunch of
adds and then checking the count is what expected and I wasn't able to reproduce
the problem yet (I've tried varying a few parameters: RF, with flush and compaction
between insert and read, consistency level).
So if you can share the test, I'd be more than happy to help the digging.)
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1936:
--------------------------------------
Remaining Estimate: 16h
Original Estimate: 16h
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 16h
> Remaining Estimate: 16h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1936:
--------------------------------------
Remaining Estimate: 8h (was: 16h)
Original Estimate: 8h (was: 16h)
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1936) Fit partitioned counter directly
into CounterColumn.value
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984749#action_12984749 ]
Sylvain Lebresne commented on CASSANDRA-1936:
---------------------------------------------
Was that with the test attached to #2005 ? I did not test the distributed tests yet,
but I ran a home-backed test (against a local 3 node cluster) doing a bunch of
adds and then checking the count is what expected and I wasn't able to reproduce
the problem yet (I've tried varying a few parameters: RF, with flush and compaction
between insert and read, consistency level).
So if you can share the test, I'd be more than happy to help the digging.
> Fit partitioned counter directly into CounterColumn.value
> ----------------------------------------------------------
>
> Key: CASSANDRA-1936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1936
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Fix For: 0.8
>
> Attachments: 0001-Put-partitioned-counter-directly-in-column-value.patch, 1936-ALT-0001-lazily-materialize-value.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The current implementation of CounterColumn keeps both the partitioned
> counter and the total value of the counter (that is, the sum of the parts of
> the partitioned counter).
> This waste space and this requires the code to keep both representation in
> sync. This ticket propose to remove the total value from the representation
> and to only calculate it when returning the value to the client.
> NOTE: this breaks the on-disk file format (for counters)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.