You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dominic Williams (Created) (JIRA)" <ji...@apache.org> on 2012/01/16 23:31:38 UTC
[jira] [Created] (CASSANDRA-3748) Range ghosts don't disappear as
expected and accumulate
Range ghosts don't disappear as expected and accumulate
-------------------------------------------------------
Key: CASSANDRA-3748
URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Environment: Cassandra on Debian
Reporter: Dominic Williams
Fix For: 1.0.8
I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
String startKey = RNG.nextUUID())
and then loading a page range of rows, specifying the key range as:
KeyRange keyRange = new KeyRange(pageSize);
keyRange.setStart_key(startKey);
keyRange.setEnd_key(maxKey);
The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3748) Range ghosts don't disappear as
expected and accumulate
Posted by "Dominic Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189291#comment-13189291 ]
Dominic Williams commented on CASSANDRA-3748:
---------------------------------------------
Hey, anyone got any ideas on this bug yet?
> Range ghosts don't disappear as expected and accumulate
> -------------------------------------------------------
>
> Key: CASSANDRA-3748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.3
> Environment: Cassandra on Debian
> Reporter: Dominic Williams
> Labels: compaction, ghost-row, range, remove
> Fix For: 1.0.8
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
> In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
> The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
> String startKey = RNG.nextUUID())
> and then loading a page range of rows, specifying the key range as:
> KeyRange keyRange = new KeyRange(pageSize);
> keyRange.setStart_key(startKey);
> keyRange.setEnd_key(maxKey);
> The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
> When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
> Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
> We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3748) Range ghosts don't disappear as
expected and accumulate
Posted by "Sylvain Lebresne (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne resolved CASSANDRA-3748.
-----------------------------------------
Resolution: Not A Problem
Fix Version/s: (was: 1.0.8)
Alright, closing. We can reopen anyway if this reproduce on 1.0.7.
> Range ghosts don't disappear as expected and accumulate
> -------------------------------------------------------
>
> Key: CASSANDRA-3748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.3
> Environment: Cassandra on Debian
> Reporter: Dominic Williams
> Labels: compaction, ghost-row, range, remove
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
> In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
> The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
> String startKey = RNG.nextUUID())
> and then loading a page range of rows, specifying the key range as:
> KeyRange keyRange = new KeyRange(pageSize);
> keyRange.setStart_key(startKey);
> keyRange.setEnd_key(maxKey);
> The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
> When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
> Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
> We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3748) Range ghosts don't disappear as
expected and accumulate
Posted by "Dominic Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205335#comment-13205335 ]
Dominic Williams commented on CASSANDRA-3748:
---------------------------------------------
Weird - ok just starting to dig into this with sstable2json, I ran repair and compact on the problem sstable again. This time it compaction actually deleted the ghosts. So I'm guessing non-reported issues were occurring with compact. Consider issue closed I'll upgrade to 1.07 and hopefully this will not happen again.
> Range ghosts don't disappear as expected and accumulate
> -------------------------------------------------------
>
> Key: CASSANDRA-3748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.3
> Environment: Cassandra on Debian
> Reporter: Dominic Williams
> Labels: compaction, ghost-row, range, remove
> Fix For: 1.0.8
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
> In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
> The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
> String startKey = RNG.nextUUID())
> and then loading a page range of rows, specifying the key range as:
> KeyRange keyRange = new KeyRange(pageSize);
> keyRange.setStart_key(startKey);
> keyRange.setEnd_key(maxKey);
> The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
> When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
> Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
> We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3748) Range ghosts don't disappear as
expected and accumulate
Posted by "Dominic Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204084#comment-13204084 ]
Dominic Williams commented on CASSANDRA-3748:
---------------------------------------------
Hmmm wondering if the main body of ghost rows aren't stuck there due to some earlier bug. Is there anyway I can get timestamps for these rows to see when they were deleted?
> Range ghosts don't disappear as expected and accumulate
> -------------------------------------------------------
>
> Key: CASSANDRA-3748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.3
> Environment: Cassandra on Debian
> Reporter: Dominic Williams
> Labels: compaction, ghost-row, range, remove
> Fix For: 1.0.8
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
> In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
> The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
> String startKey = RNG.nextUUID())
> and then loading a page range of rows, specifying the key range as:
> KeyRange keyRange = new KeyRange(pageSize);
> keyRange.setStart_key(startKey);
> keyRange.setEnd_key(maxKey);
> The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
> When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
> Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
> We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3748) Range ghosts don't disappear as
expected and accumulate
Posted by "Dominic Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194704#comment-13194704 ]
Dominic Williams commented on CASSANDRA-3748:
---------------------------------------------
Just an update - the range ghosts are still accumulating. It would appear these deleted rows are never being compacted away. Potentially a very serious bug (or system is only still running because of the caching layer)
> Range ghosts don't disappear as expected and accumulate
> -------------------------------------------------------
>
> Key: CASSANDRA-3748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.3
> Environment: Cassandra on Debian
> Reporter: Dominic Williams
> Labels: compaction, ghost-row, range, remove
> Fix For: 1.0.8
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
> In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
> The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
> String startKey = RNG.nextUUID())
> and then loading a page range of rows, specifying the key range as:
> KeyRange keyRange = new KeyRange(pageSize);
> keyRange.setStart_key(startKey);
> keyRange.setEnd_key(maxKey);
> The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
> When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
> Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
> We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3748) Range ghosts don't disappear as
expected and accumulate
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203749#comment-13203749 ]
Sylvain Lebresne commented on CASSANDRA-3748:
---------------------------------------------
I just added a test to https://github.com/riptano/cassandra-dtest to check range ghost are removed by compaction once gc_grace is elapsed and that test works fine (I checked current 1.0 branch and 1.0.3 specifically). So I'm not too sure what more to check. If you were able to reproduce in isolation that would obviously be very helpful.
> Range ghosts don't disappear as expected and accumulate
> -------------------------------------------------------
>
> Key: CASSANDRA-3748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.3
> Environment: Cassandra on Debian
> Reporter: Dominic Williams
> Labels: compaction, ghost-row, range, remove
> Fix For: 1.0.8
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
> In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
> The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
> String startKey = RNG.nextUUID())
> and then loading a page range of rows, specifying the key range as:
> KeyRange keyRange = new KeyRange(pageSize);
> keyRange.setStart_key(startKey);
> keyRange.setEnd_key(maxKey);
> The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
> When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
> Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
> We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3748) Range ghosts don't disappear as
expected and accumulate
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204330#comment-13204330 ]
Sylvain Lebresne commented on CASSANDRA-3748:
---------------------------------------------
One option would be to look at the sstables themselves, using say sstable2json, but that'll probably not be very user friendly. Another option would be to add some debug info and recompile.
> Range ghosts don't disappear as expected and accumulate
> -------------------------------------------------------
>
> Key: CASSANDRA-3748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.3
> Environment: Cassandra on Debian
> Reporter: Dominic Williams
> Labels: compaction, ghost-row, range, remove
> Fix For: 1.0.8
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
> In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
> The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
> String startKey = RNG.nextUUID())
> and then loading a page range of rows, specifying the key range as:
> KeyRange keyRange = new KeyRange(pageSize);
> keyRange.setStart_key(startKey);
> keyRange.setEnd_key(maxKey);
> The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
> When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
> Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
> We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3748) Range ghosts don't
disappear as expected and accumulate
Posted by "Dominic Williams (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194704#comment-13194704 ]
Dominic Williams edited comment on CASSANDRA-3748 at 1/27/12 1:39 PM:
----------------------------------------------------------------------
Just an update - the range ghosts are still accumulating. It would appear these deleted rows are never being compacted away. Potentially a very serious bug (our system is only still running because of the caching layer, but we are seeing some serious CPU cycles being generated by the generation of that info we cache now)
was (Author: dccwilliams):
Just an update - the range ghosts are still accumulating. It would appear these deleted rows are never being compacted away. Potentially a very serious bug (or system is only still running because of the caching layer)
> Range ghosts don't disappear as expected and accumulate
> -------------------------------------------------------
>
> Key: CASSANDRA-3748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3748
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.3
> Environment: Cassandra on Debian
> Reporter: Dominic Williams
> Labels: compaction, ghost-row, range, remove
> Fix For: 1.0.8
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> I have a problem where range ghosts are accumulating and cannot be removed by reducing GCSeconds and compacting.
> In our system, we have some cfs that represent "markets" where each row represents an item. Once an item is sold, it is removed from the market by passing its key to remove().
> The problem, which was hidden for some time by caching, is appearing on read. Every few seconds our system collates a random sample from each cf/market by choosing a random starting point:
> String startKey = RNG.nextUUID())
> and then loading a page range of rows, specifying the key range as:
> KeyRange keyRange = new KeyRange(pageSize);
> keyRange.setStart_key(startKey);
> keyRange.setEnd_key(maxKey);
> The returned rows are iterated over, and ghosts ignored. If insufficient rows are obtained, the process is repeated using the key of the last row as the starting key (or wrapping if necessary etc).
> When performance was lagging, we did a test and found that constructing a random sample of 40 items (rows) involved iterating over hundreds of thousands of ghost rows.
> Our first attempt to deal with this was to halve our GCGraceSeconds and then perform major compactions. However, this had no effect on the number of ghost rows being returned. Furthermore, on examination it seems clear that the number of ghost rows being created within GCSeconds window must be smaller than the number being returned. Thus looks like a bug.
> We are using Cassandra 1.0.3 with Sylain's patch from CASSANDRA-3510
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira