You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stu Hood (JIRA)" <ji...@apache.org> on 2011/04/12 00:15:06 UTC
[jira] [Created] (CASSANDRA-2455) Improve counter disk usage
Improve counter disk usage
--------------------------
Key: CASSANDRA-2455
URL: https://issues.apache.org/jira/browse/CASSANDRA-2455
Project: Cassandra
Issue Type: Improvement
Reporter: Stu Hood
Counter values currently use a huge amount of space on disk:
{{(header + length + RF * (nodeid + count + clock)) bytes}}
or
{{(2 + 2 + RF * (16 + 8 + 8)) bytes}}
Type specific compression (as on CASSANDRA-2398) is a long term solution to this problem, but we need a short term fix to make a large volume of counters possible.
The largest and most redundant part of the counter is the nodeid, which is now 16 bytes per replica. One proposed fix would be keep a per-sstable dictionary of all replica sets, and to assume the replicas are sorted by nodeid in the counter value. This would allow us to encode the replica as a single integer in the counter value, and to use it to look up the replica set in the dictionary. Assuming an integer replica set id, you could allow for 2^32 replica changes with 4 total bytes of overhead in each counter:
{{(header + length + replicasetid + RF (count + clock)) bytes}}
or
{{(2 + 2 + 4 + RF * (8 + 8)) bytes}}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2455) Improve counter disk usage
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood resolved CASSANDRA-2455.
---------------------------------
Resolution: Incomplete
I can't see any non-invasive solutions to this problem, so I'm focusing on CASSANDRA-674.
> Improve counter disk usage
> --------------------------
>
> Key: CASSANDRA-2455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2455
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Stu Hood
>
> Counter values currently use a huge amount of space on disk:
> {code}
> (header + length + RF * (nodeid + count + clock))
> ==
> (2 + 2 + RF * (16 + 8 + 8)) bytes
> {code}
> Type specific compression (as on CASSANDRA-2398) is a long term solution to this problem, but we need a short term fix to make a large volume of counters possible.
> The largest and most redundant part of the counter is the nodeid, which is now 16 bytes per replica. One proposed improvement would be keep a per-sstable dictionary of all replica sets, and to assume the replicas are sorted by nodeid in the counter value. This would allow us to encode the replica set as a single integer in the counter value, and to use it to look up the actual replicas in the dictionary. Assuming an integer replica set id, you could allow for 2^32 replica changes with 4 total bytes of overhead in each counter:
> {code}
> (header + length + replicasetid + RF (count + clock))
> ==
> (2 + 2 + 4 + RF * (8 + 8)) bytes
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2455) Improve counter disk usage
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-2455:
--------------------------------
Description:
Counter values currently use a huge amount of space on disk:
{code}
(header + length + RF * (nodeid + count + clock))
==
(2 + 2 + RF * (16 + 8 + 8)) bytes
{code}
Type specific compression (as on CASSANDRA-2398) is a long term solution to this problem, but we need a short term fix to make a large volume of counters possible.
The largest and most redundant part of the counter is the nodeid, which is now 16 bytes per replica. One proposed improvement would be keep a per-sstable dictionary of all replica sets, and to assume the replicas are sorted by nodeid in the counter value. This would allow us to encode the replica as a single integer in the counter value, and to use it to look up the replica set in the dictionary. Assuming an integer replica set id, you could allow for 2^32 replica changes with 4 total bytes of overhead in each counter:
{code}
(header + length + replicasetid + RF (count + clock))
==
(2 + 2 + 4 + RF * (8 + 8)) bytes
{code}
was:
Counter values currently use a huge amount of space on disk:
{{(header + length + RF * (nodeid + count + clock)) bytes}}
or
{{(2 + 2 + RF * (16 + 8 + 8)) bytes}}
Type specific compression (as on CASSANDRA-2398) is a long term solution to this problem, but we need a short term fix to make a large volume of counters possible.
The largest and most redundant part of the counter is the nodeid, which is now 16 bytes per replica. One proposed fix would be keep a per-sstable dictionary of all replica sets, and to assume the replicas are sorted by nodeid in the counter value. This would allow us to encode the replica as a single integer in the counter value, and to use it to look up the replica set in the dictionary. Assuming an integer replica set id, you could allow for 2^32 replica changes with 4 total bytes of overhead in each counter:
{{(header + length + replicasetid + RF (count + clock)) bytes}}
or
{{(2 + 2 + 4 + RF * (8 + 8)) bytes}}
> Improve counter disk usage
> --------------------------
>
> Key: CASSANDRA-2455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2455
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Stu Hood
>
> Counter values currently use a huge amount of space on disk:
> {code}
> (header + length + RF * (nodeid + count + clock))
> ==
> (2 + 2 + RF * (16 + 8 + 8)) bytes
> {code}
> Type specific compression (as on CASSANDRA-2398) is a long term solution to this problem, but we need a short term fix to make a large volume of counters possible.
> The largest and most redundant part of the counter is the nodeid, which is now 16 bytes per replica. One proposed improvement would be keep a per-sstable dictionary of all replica sets, and to assume the replicas are sorted by nodeid in the counter value. This would allow us to encode the replica as a single integer in the counter value, and to use it to look up the replica set in the dictionary. Assuming an integer replica set id, you could allow for 2^32 replica changes with 4 total bytes of overhead in each counter:
> {code}
> (header + length + replicasetid + RF (count + clock))
> ==
> (2 + 2 + 4 + RF * (8 + 8)) bytes
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2455) Improve counter disk usage
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-2455:
--------------------------------
Description:
Counter values currently use a huge amount of space on disk:
{code}
(header + length + RF * (nodeid + count + clock))
==
(2 + 2 + RF * (16 + 8 + 8)) bytes
{code}
Type specific compression (as on CASSANDRA-2398) is a long term solution to this problem, but we need a short term fix to make a large volume of counters possible.
The largest and most redundant part of the counter is the nodeid, which is now 16 bytes per replica. One proposed improvement would be keep a per-sstable dictionary of all replica sets, and to assume the replicas are sorted by nodeid in the counter value. This would allow us to encode the replica set as a single integer in the counter value, and to use it to look up the actual replicas in the dictionary. Assuming an integer replica set id, you could allow for 2^32 replica changes with 4 total bytes of overhead in each counter:
{code}
(header + length + replicasetid + RF (count + clock))
==
(2 + 2 + 4 + RF * (8 + 8)) bytes
{code}
was:
Counter values currently use a huge amount of space on disk:
{code}
(header + length + RF * (nodeid + count + clock))
==
(2 + 2 + RF * (16 + 8 + 8)) bytes
{code}
Type specific compression (as on CASSANDRA-2398) is a long term solution to this problem, but we need a short term fix to make a large volume of counters possible.
The largest and most redundant part of the counter is the nodeid, which is now 16 bytes per replica. One proposed improvement would be keep a per-sstable dictionary of all replica sets, and to assume the replicas are sorted by nodeid in the counter value. This would allow us to encode the replica as a single integer in the counter value, and to use it to look up the replica set in the dictionary. Assuming an integer replica set id, you could allow for 2^32 replica changes with 4 total bytes of overhead in each counter:
{code}
(header + length + replicasetid + RF (count + clock))
==
(2 + 2 + 4 + RF * (8 + 8)) bytes
{code}
> Improve counter disk usage
> --------------------------
>
> Key: CASSANDRA-2455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2455
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Stu Hood
>
> Counter values currently use a huge amount of space on disk:
> {code}
> (header + length + RF * (nodeid + count + clock))
> ==
> (2 + 2 + RF * (16 + 8 + 8)) bytes
> {code}
> Type specific compression (as on CASSANDRA-2398) is a long term solution to this problem, but we need a short term fix to make a large volume of counters possible.
> The largest and most redundant part of the counter is the nodeid, which is now 16 bytes per replica. One proposed improvement would be keep a per-sstable dictionary of all replica sets, and to assume the replicas are sorted by nodeid in the counter value. This would allow us to encode the replica set as a single integer in the counter value, and to use it to look up the actual replicas in the dictionary. Assuming an integer replica set id, you could allow for 2^32 replica changes with 4 total bytes of overhead in each counter:
> {code}
> (header + length + replicasetid + RF (count + clock))
> ==
> (2 + 2 + 4 + RF * (8 + 8)) bytes
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira