You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Tristan Seligmann (JIRA)" <ji...@apache.org> on 2012/10/18 07:16:05 UTC
[jira] [Created] (CASSANDRA-4832) AssertionError: keys must not be
empty
Tristan Seligmann created CASSANDRA-4832:
--------------------------------------------
Summary: AssertionError: keys must not be empty
Key: CASSANDRA-4832
URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.6
Environment: Debian 6.0.5
Reporter: Tristan Seligmann
I'm getting errors like this logged:
INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
java.lang.AssertionError: Keys must not be empty
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4832) AssertionError: keys must not
be empty
Posted by "Joe Fox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481209#comment-13481209 ]
Joe Fox commented on CASSANDRA-4832:
------------------------------------
I've run into the same symptoms as Chris, but while attempting to migrate data between a 1.1.1 cluster and 1.1.6.
Whilst using sstableloader to import the data, nodes would hit certain column indexes and stop processing further requests once the assertion was thrown - restarting the node would almost immediately throw the assertion again, and the node would just fail to rejoin the ring.
tpstats would show active flushwriter tasks but no further node activity.
We had to work around the issue by:
1. Removing all indexes from our target cluster schema
2. Importing the data via sstableloader
3. Scanning through relevant column families and inserting data into any empty indexed columns
4. Re-applying the indexes to the target cluster schema
Only then was the migration successful.
I'll also note that attempting to apply an index to a column which has null data will also throw the cluster out of sync as the nodes which throw the assertion fail to migrate their schemas properly
INFO [Creating index: Transactions.TransactionsCountryCode] 2012-10-21 07:45:25,978 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512 serialized/live bytes, 762 ops)
INFO [Creating index: Transactions.TransactionsStatus] 2012-10-21 07:45:25,980 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsStatus@673679943(38862/125966 serialized/live bytes, 762 ops)
INFO [FlushWriter:1] 2012-10-21 07:45:25,987 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,004 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsStatus@673679943(38862/125966 serialized/live bytes, 762 ops)
ERROR [FlushWriter:1] 2012-10-21 07:45:26,004 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[FlushWriter:1,5,main]
java.lang.AssertionError: Keys must not be empty
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:176)
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:295)
at org.apache.cassandra.db.Memtable.access$600(Memtable.java:48)
at org.apache.cassandra.db.Memtable$5.runMayThrow(Memtable.java:316)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [FlushWriter:2] 2012-10-21 07:45:26,049 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/*******/Transactions/*******-Transactions.TransactionsStatus-hf-2-Data.db (35259 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
INFO [Creating index: Transactions.TransactionsLastUpdateDate] 2012-10-21 07:45:26,313 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,314 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,743 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/*******/Transactions/*******-Transactions.TransactionsLastUpdateDate-hf-2-Data.db (37024 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
INFO [main] 2012-10-21 07:45:27,052 CommitLogReplayer.java (line 272) Finished reading /var/lib/cassandra/commitlog/CommitLog-1350768744805.log
INFO [main] 2012-10-21 07:45:27,054 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Versions@1851630436(83/103 serialized/live bytes, 3 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:27,054 Memtable.java (line 264) Writing Memtable-Versions@1851630436(83/103 serialized/live bytes, 3 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:27,061 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/Versions/system-Versions-hf-1-Data.db (247 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4832) AssertionError: keys must not be
empty
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4832:
--------------------------------------
Reviewer: jbellis
Component/s: Core
Priority: Minor (was: Major)
Fix Version/s: 1.1.7
Assignee: Tristan Seligmann
Labels: indexing (was: )
you're right. removed the assert in 72dcc298d335721c053444249c157e9a6431ebea.
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4832) AssertionError: keys must not
be empty
Posted by "Tristan Seligmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478741#comment-13478741 ]
Tristan Seligmann commented on CASSANDRA-4832:
----------------------------------------------
After some further investigation, it looks like this began after I upgraded from 1.1.2 to 1.1.6. The assertion seems to have been introduced as part of some debugging in CASSANDRA-4687. I'm not familiar with the code at all, but it seems to me that perhaps the assertion is bogus? If I have a secondary index on a particular column, and that column has an empty value in some row, would this not result in an empty key in the secondary index column family?
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4832) AssertionError: keys must
not be empty
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480477#comment-13480477 ]
Jonathan Ellis edited comment on CASSANDRA-4832 at 10/19/12 10:45 PM:
----------------------------------------------------------------------
you're right that the assert is bogus. removed it in 72dcc298d335721c053444249c157e9a6431ebea.
was (Author: jbellis):
you're right. removed the assert in 72dcc298d335721c053444249c157e9a6431ebea.
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4832) AssertionError: keys must not be
empty
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4832.
---------------------------------------
Resolution: Fixed
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4832) AssertionError: keys must
not be empty
Posted by "Joe Fox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481209#comment-13481209 ]
Joe Fox edited comment on CASSANDRA-4832 at 10/22/12 8:52 AM:
--------------------------------------------------------------
I've run into the same symptoms as Chris, but while attempting to migrate data between a 1.1.1 cluster and 1.1.6.
Whilst using sstableloader to import the data, nodes would hit certain column indexes and stop processing further requests once the assertion was thrown - restarting the node would almost immediately throw the assertion again, and the node would just fail to rejoin the ring.
tpstats would show active flushwriter tasks but no further node activity.
We had to work around the issue by:
1. Removing all indexes from our target cluster schema
2. Importing the data via sstableloader
3. Scanning through relevant column families and inserting data into any empty indexed columns
4. Re-applying the indexes to the target cluster schema
Only then was the migration successful.
I'll also note that attempting to apply an index to a column which has null data will also throw the cluster out of sync as the nodes which throw the assertion fail to migrate their schemas properly
INFO [Creating index: Transactions.TransactionsCountryCode] 2012-10-21 07:45:25,978 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512 serialized/live bytes, 762 ops)
INFO [Creating index: Transactions.TransactionsStatus] 2012-10-21 07:45:25,980 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsStatus@673679943(38862/125966 serialized/live bytes, 762 ops)
INFO [FlushWriter:1] 2012-10-21 07:45:25,987 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,004 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsStatus@673679943(38862/125966 serialized/live bytes, 762 ops)
ERROR [FlushWriter:1] 2012-10-21 07:45:26,004 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[FlushWriter:1,5,main]
java.lang.AssertionError: Keys must not be empty
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:176)
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:295)
at org.apache.cassandra.db.Memtable.access$600(Memtable.java:48)
at org.apache.cassandra.db.Memtable$5.runMayThrow(Memtable.java:316)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [FlushWriter:2] 2012-10-21 07:45:26,049 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/xxxxxx/Transactions/xxxxxx-Transactions.TransactionsStatus-hf-2-Data.db (35259 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
INFO [Creating index: Transactions.TransactionsLastUpdateDate] 2012-10-21 07:45:26,313 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,314 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,743 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/xxxxxx/Transactions/xxxxxx-Transactions.TransactionsLastUpdateDate-hf-2-Data.db (37024 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
INFO [main] 2012-10-21 07:45:27,052 CommitLogReplayer.java (line 272) Finished reading /var/lib/cassandra/commitlog/CommitLog-1350768744805.log
INFO [main] 2012-10-21 07:45:27,054 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Versions@1851630436(83/103 serialized/live bytes, 3 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:27,054 Memtable.java (line 264) Writing Memtable-Versions@1851630436(83/103 serialized/live bytes, 3 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:27,061 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/Versions/system-Versions-hf-1-Data.db (247 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
was (Author: e1zorro):
I've run into the same symptoms as Chris, but while attempting to migrate data between a 1.1.1 cluster and 1.1.6.
Whilst using sstableloader to import the data, nodes would hit certain column indexes and stop processing further requests once the assertion was thrown - restarting the node would almost immediately throw the assertion again, and the node would just fail to rejoin the ring.
tpstats would show active flushwriter tasks but no further node activity.
We had to work around the issue by:
1. Removing all indexes from our target cluster schema
2. Importing the data via sstableloader
3. Scanning through relevant column families and inserting data into any empty indexed columns
4. Re-applying the indexes to the target cluster schema
Only then was the migration successful.
I'll also note that attempting to apply an index to a column which has null data will also throw the cluster out of sync as the nodes which throw the assertion fail to migrate their schemas properly
INFO [Creating index: Transactions.TransactionsCountryCode] 2012-10-21 07:45:25,978 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512 serialized/live bytes, 762 ops)
INFO [Creating index: Transactions.TransactionsStatus] 2012-10-21 07:45:25,980 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsStatus@673679943(38862/125966 serialized/live bytes, 762 ops)
INFO [FlushWriter:1] 2012-10-21 07:45:25,987 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,004 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsStatus@673679943(38862/125966 serialized/live bytes, 762 ops)
ERROR [FlushWriter:1] 2012-10-21 07:45:26,004 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[FlushWriter:1,5,main]
java.lang.AssertionError: Keys must not be empty
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:176)
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:295)
at org.apache.cassandra.db.Memtable.access$600(Memtable.java:48)
at org.apache.cassandra.db.Memtable$5.runMayThrow(Memtable.java:316)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [FlushWriter:2] 2012-10-21 07:45:26,049 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/*******/Transactions/*******-Transactions.TransactionsStatus-hf-2-Data.db (35259 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
INFO [Creating index: Transactions.TransactionsLastUpdateDate] 2012-10-21 07:45:26,313 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,314 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277 serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,743 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/*******/Transactions/*******-Transactions.TransactionsLastUpdateDate-hf-2-Data.db (37024 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
INFO [main] 2012-10-21 07:45:27,052 CommitLogReplayer.java (line 272) Finished reading /var/lib/cassandra/commitlog/CommitLog-1350768744805.log
INFO [main] 2012-10-21 07:45:27,054 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Versions@1851630436(83/103 serialized/live bytes, 3 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:27,054 Memtable.java (line 264) Writing Memtable-Versions@1851630436(83/103 serialized/live bytes, 3 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:27,061 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/Versions/system-Versions-hf-1-Data.db (247 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4832) AssertionError: keys must not
be empty
Posted by "Tristan Seligmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478687#comment-13478687 ]
Tristan Seligmann commented on CASSANDRA-4832:
----------------------------------------------
Actually, it seems like this gets logged any time Cassandra tries to flush this column family now.
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4832) AssertionError: keys must not be
empty
Posted by "Chris Herron (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Herron updated CASSANDRA-4832:
------------------------------------
Attachment: FlushWriterKeyAssertionBlock.txt
Came across this investigating an apparent deadlock in Schema Migrations.
If this assertion fails on the flushWriter executor, it blocks indefinitely. Anything upstream locking-wise gets stuck also. This was on 1.1.6.
Log output below, thread dump attached.
ERROR [FlushWriter:3] 2012-10-19 22:27:56,948 org.apache.cassandra.service.AbstractCassandraDaemon Exception in thread Thread[FlushWriter:3,5,main]
java.lang.AssertionError: Keys must not be empty
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:176)
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:295)
at org.apache.cassandra.db.Memtable.access$600(Memtable.java:48)
at org.apache.cassandra.db.Memtable$5.runMayThrow(Memtable.java:316)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4832) AssertionError: keys must not
be empty
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499652#comment-13499652 ]
Jonathan Ellis commented on CASSANDRA-4832:
-------------------------------------------
The patch to fix this is git commit 72dcc298d335721c053444249c157e9a6431ebea.
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4832) AssertionError: keys must not
be empty
Posted by "Scott Fines (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499636#comment-13499636 ]
Scott Fines commented on CASSANDRA-4832:
----------------------------------------
Is there an actual patch for this, or a Workaround? We see this same AssertionError when attempting to start our nodes; restarting the node usually works correctly, however.
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4832) AssertionError: keys must not
be empty
Posted by "Tristan Seligmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479512#comment-13479512 ]
Tristan Seligmann commented on CASSANDRA-4832:
----------------------------------------------
I modified my application to avoid inserting the empty column in question (which was a bug anyway; if that particular column was empty, the application should not have been doing an insert at all) and the problem has gone away, so it seems like my hypothesis is at least partially correct. I'm going to attempt to reproduce this in a more controlled environment now that my immediate issue in production has been resolved.
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4832) AssertionError: keys must not be
empty
Posted by "Joe Fox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe Fox updated CASSANDRA-4832:
-------------------------------
Comment: was deleted
(was: I'm seeing the same issue as Chris after updating from 1.1.1 to 1.1.6.)
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4832) AssertionError: keys must not
be empty
Posted by "Joe Fox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481186#comment-13481186 ]
Joe Fox commented on CASSANDRA-4832:
------------------------------------
I'm seeing the same issue as Chris after updating from 1.1.1 to 1.1.6.
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira