You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Paul Ingalls <pa...@gmail.com> on 2013/07/19 19:04:31 UTC
NPE in CompactionExecutor
I'm seeing a number of NullPointerExceptions in the log of my cluster. You can see the log line below. I'm thinking this is probably bad. Any ideas?
ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
java.lang.NullPointerException
Not sure if this is related or not, but I'm also getting a bunch of AssertionErrors as well, even after running a scrub…
ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
java.lang.AssertionError: incorrect row data size 29502477 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-5262-Data.db; correct is 29725806
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Re: NPE in CompactionExecutor
Posted by aaron morton <aa...@thelastpickle.com>.
> There was no error stack, just that line in the log.
It's odd that the stack is not there.
This is an unhanded exception when running compaction. It may be related to the assertions.
If you can reproduce it please raise a ticket at https://issues.apache.org/jira/browse/CASSANDRA
Cheers
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 24/07/2013, at 3:50 AM, Paul Ingalls <pa...@gmail.com> wrote:
> I'm running the latest from the 1.2 branch as of a few days ago. I needed one of the patches that will be in 1.2.7
>
> There was no error stack, just that line in the log.
>
> I wiped the database (deleted all the files in the lib dir) and restarted my data load, and am consistently running into the incorrect row data size error, almost immediately… It seems to be specific to compacting large rows. I have been unsuccessful in getting a large row to compact…
>
> Paul
>
> On Jul 21, 2013, at 1:42 PM, aaron morton <aa...@thelastpickle.com> wrote:
>
>> What version are you running ?
>>
>>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
>>> java.lang.NullPointerException
>> What' the full error stack ?
>>
>>> Not sure if this is related or not, but I'm also getting a bunch of AssertionErrors as well, even after running a scrub…
>>>
>>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
>>> java.lang.AssertionError: incorrect row data size 29502477 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-5262-Data.db; correct is 29725806
>> Double check that the scrub was successful.
>>
>> If it's not detecting / fixing the problem look for previous log messages from that thread [CompactionExecutor:38] and see what sstables it was compacting. Try remove those. But I would give scrub another chance to get it sorted.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Cassandra Consultant
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 20/07/2013, at 5:04 AM, Paul Ingalls <pa...@gmail.com> wrote:
>>
>>> I'm seeing a number of NullPointerExceptions in the log of my cluster. You can see the log line below. I'm thinking this is probably bad. Any ideas?
>>>
>>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
>>> java.lang.NullPointerException
>>>
>>> Not sure if this is related or not, but I'm also getting a bunch of AssertionErrors as well, even after running a scrub…
>>>
>>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
>>> java.lang.AssertionError: incorrect row data size 29502477 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-5262-Data.db; correct is 29725806
>>> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
>>> at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
>>> at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
>>> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>>> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>>
>>
>
Re: NPE in CompactionExecutor
Posted by Paul Ingalls <pa...@gmail.com>.
I'm running the latest from the 1.2 branch as of a few days ago. I needed one of the patches that will be in 1.2.7
There was no error stack, just that line in the log.
I wiped the database (deleted all the files in the lib dir) and restarted my data load, and am consistently running into the incorrect row data size error, almost immediately… It seems to be specific to compacting large rows. I have been unsuccessful in getting a large row to compact…
Paul
On Jul 21, 2013, at 1:42 PM, aaron morton <aa...@thelastpickle.com> wrote:
> What version are you running ?
>
>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
>> java.lang.NullPointerException
> What' the full error stack ?
>
>> Not sure if this is related or not, but I'm also getting a bunch of AssertionErrors as well, even after running a scrub…
>>
>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
>> java.lang.AssertionError: incorrect row data size 29502477 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-5262-Data.db; correct is 29725806
> Double check that the scrub was successful.
>
> If it's not detecting / fixing the problem look for previous log messages from that thread [CompactionExecutor:38] and see what sstables it was compacting. Try remove those. But I would give scrub another chance to get it sorted.
>
> Cheers
>
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/07/2013, at 5:04 AM, Paul Ingalls <pa...@gmail.com> wrote:
>
>> I'm seeing a number of NullPointerExceptions in the log of my cluster. You can see the log line below. I'm thinking this is probably bad. Any ideas?
>>
>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
>> java.lang.NullPointerException
>>
>> Not sure if this is related or not, but I'm also getting a bunch of AssertionErrors as well, even after running a scrub…
>>
>> ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
>> java.lang.AssertionError: incorrect row data size 29502477 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-5262-Data.db; correct is 29725806
>> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
>> at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
>> at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
>> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>>
>>
>
Re: NPE in CompactionExecutor
Posted by aaron morton <aa...@thelastpickle.com>.
What version are you running ?
> ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
> java.lang.NullPointerException
What' the full error stack ?
> Not sure if this is related or not, but I'm also getting a bunch of AssertionErrors as well, even after running a scrub…
>
> ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
> java.lang.AssertionError: incorrect row data size 29502477 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-5262-Data.db; correct is 29725806
Double check that the scrub was successful.
If it's not detecting / fixing the problem look for previous log messages from that thread [CompactionExecutor:38] and see what sstables it was compacting. Try remove those. But I would give scrub another chance to get it sorted.
Cheers
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 20/07/2013, at 5:04 AM, Paul Ingalls <pa...@gmail.com> wrote:
> I'm seeing a number of NullPointerExceptions in the log of my cluster. You can see the log line below. I'm thinking this is probably bad. Any ideas?
>
> ERROR [CompactionExecutor:38] 2013-07-19 17:01:34,494 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
> java.lang.NullPointerException
>
> Not sure if this is related or not, but I'm also getting a bunch of AssertionErrors as well, even after running a scrub…
>
> ERROR [CompactionExecutor:38] 2013-07-19 17:01:06,192 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:38,1,main]
> java.lang.AssertionError: incorrect row data size 29502477 written to /mnt/datadrive/lib/cassandra/data/fanzo/tweets_by_team/fanzo-tweets_by_team-tmp-ic-5262-Data.db; correct is 29725806
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
> at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
> at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>
>