You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jason Harvey (JIRA)" <ji...@apache.org> on 2011/03/10 20:12:59 UTC
[jira] Created: (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Scrub resulting in "Keys must be written in ascending order" exception
----------------------------------------------------------------------
Key: CASSANDRA-2309
URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.7.3
Reporter: Jason Harvey
Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
{code}
DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
java.io.IOException: Keys must be written in ascending order.
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006050#comment-13006050 ]
Jonathan Ellis commented on CASSANDRA-2309:
-------------------------------------------
that means sstable2json can't deal with out-of-order rows either. we can probably change that.
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010324#comment-13010324 ]
Jonathan Ellis commented on CASSANDRA-2309:
-------------------------------------------
Actually, in your log paste it never starts any non-system CFs. So I return to my original guess, that it can't find the schema for some reason.
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006129#comment-13006129 ]
Jason Harvey commented on CASSANDRA-2309:
-----------------------------------------
That appears to work. Thanks!
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260621#comment-13260621 ]
Jonathan Ellis commented on CASSANDRA-2309:
-------------------------------------------
bq. Sstable2json doesn't work in the snapshot directories. Worthy of a new bug report?
I believe this is fixed now, but if not, yes.
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010322#comment-13010322 ]
Jason Harvey commented on CASSANDRA-2309:
-----------------------------------------
I should add that sstable2json does work on other sstables from my snapshot, just not the ones I need it to work on :(
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010499#comment-13010499 ]
Jason Harvey commented on CASSANDRA-2309:
-----------------------------------------
Gahhhh. Sstable2json doesn't work in the snapshot directories. Worthy of a new bug report? Would be nice to at least get a friendly error.
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010024#comment-13010024 ]
Jason Harvey commented on CASSANDRA-2309:
-----------------------------------------
Also, another very scary thing. The sstables with the unordered keys were written within the last couple days. What might cause sstables with unordered keys to be written in 0.7.4?
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reopened CASSANDRA-2309:
---------------------------------------
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010317#comment-13010317 ]
Jason Harvey commented on CASSANDRA-2309:
-----------------------------------------
bq. That looks a lot like "sstable2json couldn't find the Cassandra schema"
I grabbed some debug output. Looks like it can find the schema. It is dying just after it loads the index:
{code}
INFO 12:39:45,401 Loading settings from file:/etc/cassandra/cassandra.yaml
DEBUG 12:39:45,623 Syncing log with a period of 10000
INFO 12:39:45,623 DiskAccessMode is standard, indexAccessMode is mmap
DEBUG 12:39:45,709 setting auto_bootstrap to false
DEBUG 12:39:45,853 Initializing system.IndexInfo
DEBUG 12:39:45,880 Starting CFS IndexInfo
DEBUG 12:39:45,895 key cache capacity for IndexInfo is 1
DEBUG 12:39:45,896 row cache capacity for IndexInfo is 0
DEBUG 12:39:45,899 Initializing system.Schema
DEBUG 12:39:45,901 Starting CFS Schema
INFO 12:39:45,902 reading saved cache /var/lib/cassandra/saved_caches/system-Schema-KeyCache
DEBUG 12:39:45,903 completed reading (0 ms; 1 keys) saved cache /var/lib/cassandra/saved_caches/system-Schema-KeyCache
INFO 12:39:45,920 Opening /var/lib/cassandra/data/system/Schema-f-9
DEBUG 12:39:45,920 Load statistics for /var/lib/cassandra/data/system/Schema-f-9
INFO 12:39:45,935 JNA not found. Native methods will be disabled.
DEBUG 12:39:45,961 INDEX LOAD TIME for /var/lib/cassandra/data/system/Schema-f-9: 41 ms.
DEBUG 12:39:45,962 key cache contains 1/1 keys
DEBUG 12:39:45,962 adding /var/lib/cassandra/data/system/Schema-f-9 to list of files tracked for system.Schema
DEBUG 12:39:45,962 row cache capacity for Schema is 0
DEBUG 12:39:45,963 Initializing system.Migrations
DEBUG 12:39:45,965 Starting CFS Migrations
INFO 12:39:45,967 Opening /var/lib/cassandra/data/system/Migrations-f-9
DEBUG 12:39:45,967 Load statistics for /var/lib/cassandra/data/system/Migrations-f-9
DEBUG 12:39:45,969 INDEX LOAD TIME for /var/lib/cassandra/data/system/Migrations-f-9: 2 ms.
DEBUG 12:39:45,969 key cache contains 0/0 keys
DEBUG 12:39:45,970 adding /var/lib/cassandra/data/system/Migrations-f-9 to list of files tracked for system.Migrations
DEBUG 12:39:45,970 key cache capacity for Migrations is 1
DEBUG 12:39:45,970 row cache capacity for Migrations is 0
DEBUG 12:39:45,971 Initializing system.LocationInfo
DEBUG 12:39:45,973 Starting CFS LocationInfo
INFO 12:39:45,974 reading saved cache /var/lib/cassandra/saved_caches/system-LocationInfo-KeyCache
DEBUG 12:39:45,975 completed reading (1 ms; 1 keys) saved cache /var/lib/cassandra/saved_caches/system-LocationInfo-KeyCache
INFO 12:39:45,986 Opening /var/lib/cassandra/data/system/LocationInfo-f-290
DEBUG 12:39:45,986 Load statistics for /var/lib/cassandra/data/system/LocationInfo-f-290
DEBUG 12:39:45,989 INDEX LOAD TIME for /var/lib/cassandra/data/system/LocationInfo-f-290: 3 ms.
DEBUG 12:39:45,989 key cache contains 1/1 keys
INFO 12:39:45,989 Opening /var/lib/cassandra/data/system/LocationInfo-f-291
DEBUG 12:39:45,989 Load statistics for /var/lib/cassandra/data/system/LocationInfo-f-291
DEBUG 12:39:45,992 INDEX LOAD TIME for /var/lib/cassandra/data/system/LocationInfo-f-291: 3 ms.
DEBUG 12:39:45,992 key cache contains 1/2 keys
DEBUG 12:39:45,992 adding /var/lib/cassandra/data/system/LocationInfo-f-290 to list of files tracked for system.LocationInfo
DEBUG 12:39:45,993 adding /var/lib/cassandra/data/system/LocationInfo-f-291 to list of files tracked for system.LocationInfo
DEBUG 12:39:45,993 row cache capacity for LocationInfo is 0
DEBUG 12:39:45,994 Initializing system.HintsColumnFamily
DEBUG 12:39:45,996 Starting CFS HintsColumnFamily
INFO 12:39:45,997 reading saved cache /var/lib/cassandra/saved_caches/system-HintsColumnFamily-KeyCache
DEBUG 12:39:45,997 completed reading (1 ms; 1 keys) saved cache /var/lib/cassandra/saved_caches/system-HintsColumnFamily-KeyCache
DEBUG 12:39:45,998 not including compacted sstable HintsColumnFamily-71
DEBUG 12:39:45,998 not including compacted sstable HintsColumnFamily-71
DEBUG 12:39:45,999 not including compacted sstable HintsColumnFamily-71
DEBUG 12:39:45,999 not including compacted sstable HintsColumnFamily-71
DEBUG 12:39:45,999 not including compacted sstable HintsColumnFamily-71
INFO 12:39:46,000 Opening /var/lib/cassandra/data/system/HintsColumnFamily-f-72
DEBUG 12:39:46,000 Load statistics for /var/lib/cassandra/data/system/HintsColumnFamily-f-72
DEBUG 12:39:46,002 INDEX LOAD TIME for /var/lib/cassandra/data/system/HintsColumnFamily-f-72: 2 ms.
DEBUG 12:39:46,003 key cache contains 1/1 keys
INFO 12:39:46,003 Opening /var/lib/cassandra/data/system/HintsColumnFamily-f-73
DEBUG 12:39:46,003 Load statistics for /var/lib/cassandra/data/system/HintsColumnFamily-f-73
DEBUG 12:39:46,006 INDEX LOAD TIME for /var/lib/cassandra/data/system/HintsColumnFamily-f-73: 3 ms.
DEBUG 12:39:46,006 key cache contains 2/2 keys
DEBUG 12:39:46,006 adding /var/lib/cassandra/data/system/HintsColumnFamily-f-72 to list of files tracked for system.HintsColumnFamily
DEBUG 12:39:46,007 adding /var/lib/cassandra/data/system/HintsColumnFamily-f-73 to list of files tracked for system.HintsColumnFamily
DEBUG 12:39:46,007 row cache capacity for HintsColumnFamily is 0
INFO 12:39:46,048 Loading schema version 09021612-3bac-11e0-b472-fc83263a611f
DEBUG 12:39:46,059 collecting 0 of 2147483647: Avro/Schema:false:2042@1298067089267
DEBUG 12:39:46,060 collecting 1 of 2147483647: reddit:false:2502@1298067089267
WARN 12:39:46,396 Schema definitions were defined both locally and in cassandra.yaml. Definitions in cassandra.yaml were ignored.
INFO 12:39:46,415 Opening /var/lib/cassandra/data/reddit/snapshots/1300857807427-prescrub/permacache-f-161223
DEBUG 12:39:46,415 Load statistics for /var/lib/cassandra/data/reddit/snapshots/1300857807427-prescrub/permacache-f-161223
DEBUG 12:39:48,175 INDEX LOAD TIME for /var/lib/cassandra/data/reddit/snapshots/1300857807427-prescrub/permacache-f-161223: 1760 ms.
{
Exception in thread "main" java.lang.NullPointerException
at org.apache.cassandra.db.ColumnFamily.create(ColumnFamily.java:67)
at org.apache.cassandra.io.sstable.SSTableReader.createColumnFamily(SSTableReader.java:583)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:110)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:293)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:324)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:337)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:395)
{code}
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Harvey updated CASSANDRA-2309:
------------------------------------
Affects Version/s: 0.7.4
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2309.
---------------------------------------
Resolution: Not A Problem
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006054#comment-13006054 ]
Jonathan Ellis commented on CASSANDRA-2309:
-------------------------------------------
try the patch in CASSANDRA-2318
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006011#comment-13006011 ]
Jason Harvey commented on CASSANDRA-2309:
-----------------------------------------
Unfortunately sstable2json is also dying on this sstable :(
{code}
Exception in thread "main" java.lang.NullPointerException
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
at org.apache.cassandra.tools.SSTableExport.serializeColumns(SSTableExport.java:108)
at org.apache.cassandra.tools.SSTableExport.serializeRow(SSTableExport.java:257)
at org.apache.cassandra.tools.SSTableExport.serializeRow(SSTableExport.java:189)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:376)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:398)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:411)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:469)
{code}
Is this due to an issue with the sstable, or is it a separate bug?
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005264#comment-13005264 ]
Jonathan Ellis commented on CASSANDRA-2309:
-------------------------------------------
Well, it's right -- the "current key" sorts before the "last written key" so it's out of order.
If scrub is able to continue, then you should probably do an sstable2json, then let scrub go ahead and strip out those out-of-order keys, then use the log messages to pull the keys that were out of order out of your json dump.
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2309) Scrub resulting in "Keys must be
written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2309.
---------------------------------------
Resolution: Invalid
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010100#comment-13010100 ]
Jonathan Ellis commented on CASSANDRA-2309:
-------------------------------------------
bq. The sstable2json attempt is throwing the following
That looks a lot like "sstable2json couldn't find the Cassandra schema"
bq. What might cause sstables with unordered keys to be written in 0.7.4?
Possibly some kind of ByteBuffer race, although you'd think more people would hit that. Are the keys correct-but-unordered or unordered-and-garbage?
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010320#comment-13010320 ]
Jonathan Ellis commented on CASSANDRA-2309:
-------------------------------------------
the NPE is because it doesn't have metadata for the CF it's trying to instantiate. you probably need to add some debug statements along that stacktrace to figure out why.
Also: it looks like you're running with asserts off. That makes it trickier to debug.
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must
be written in ascending order" exception
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010023#comment-13010023 ]
Jason Harvey commented on CASSANDRA-2309:
-----------------------------------------
I have several tables with unordered keys, and they also can't be exported via sstable2json. The sstable2json attempt is throwing the following. I do have the patch for #2318. Running 0.7.4:
{code}
Exception in thread "main" java.lang.NullPointerException
at org.apache.cassandra.db.ColumnFamily.create(ColumnFamily.java:67)
at org.apache.cassandra.io.sstable.SSTableReader.createColumnFamily(SSTableReader.java:583)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:110)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:293)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:324)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:337)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:395)
{code}
> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-2309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.3, 0.7.4
> Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543) Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552) row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581) Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
> INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
> WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599) Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira