You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Daning <da...@netseer.com> on 2011/09/21 19:44:42 UTC

Unable to create compaction marker

I got this exception in system.log several times a day, do you have idea 
what caused this problem and what is the consequence?


ERROR [CompactionExecutor:12] 2011-09-15 14:29:19,578 
AbstractCassandraDaemon.java (line 139) Fatal exception in thread 
Thread[CompactionExecutor:12,1,main]
java.io.IOError: java.io.IOException: Unable to create compaction marker
         at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:642)
         at 
org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
         at 
org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
         at 
org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
         at 
org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1094)
         at 
org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:599)
         at 
org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507)
         at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142)
         at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108)
         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
         at java.util.concurrent.FutureTask.run(Unknown Source)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
Source)
         at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Unable to create compaction marker
         at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
         ... 13 more


Thank you in advance.

Daning

Re: Unable to create compaction marker

Posted by Radim Kolar <hs...@sendmail.cz>.
Dne 21.9.2011 20:01, Jonathan Ellis napsal(a):
> Means Cassandra couldn't create an empty file in the data directory
> designating a sstable as compacted.  I'd look for permissions
> problems.
>
> Short term there is no dire consequence, although it will keep
> re-compacting that sstable.  Longer term you'll run out of disk space
> since nothing gets deleted.
how it could run of out disk space? If cassandra is able to create new 
sstable, then it should be able to create empty files too.

Re: Unable to create compaction marker

Posted by Daning <da...@netseer.com>.
Thank you guys. I don't think have permission issue or out of space. 
might be ulimit set to low(1024), we will change ulimit -Hn and unlimit 
-Sn to see if that could solve the problem.

Daning

On 09/22/2011 01:12 AM, aaron morton wrote:
> It's in the yaml file…
>
> # directories where Cassandra should store data on disk.
> data_file_directories:
>      - /var/lib/cassandra/data
>
> The permissions are normally cassandra:cassandra
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/09/2011, at 12:06 PM, Daning wrote:
>
>> Thanks Jonathan. where is the data directory configured? since I did not any permission problem.
>>
>> Daning
>>
>> On 09/21/2011 11:01 AM, Jonathan Ellis wrote:
>>> Means Cassandra couldn't create an empty file in the data directory
>>> designating a sstable as compacted.  I'd look for permissions
>>> problems.
>>>
>>> Short term there is no dire consequence, although it will keep
>>> re-compacting that sstable.  Longer term you'll run out of disk space
>>> since nothing gets deleted.
>>>
>>> On Wed, Sep 21, 2011 at 1:44 PM, Daning<da...@netseer.com>   wrote:
>>>> I got this exception in system.log several times a day, do you have idea
>>>> what caused this problem and what is the consequence?
>>>>
>>>>
>>>> ERROR [CompactionExecutor:12] 2011-09-15 14:29:19,578
>>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>>> Thread[CompactionExecutor:12,1,main]
>>>> java.io.IOError: java.io.IOException: Unable to create compaction marker
>>>>         at
>>>> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:642)
>>>>         at
>>>> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
>>>>         at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
>>>>         at
>>>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
>>>>         at
>>>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1094)
>>>>         at
>>>> org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:599)
>>>>         at
>>>> org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507)
>>>>         at
>>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142)
>>>>         at
>>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108)
>>>>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>>>>         at java.util.concurrent.FutureTask.run(Unknown Source)
>>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>>>> Source)
>>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>>>         at java.lang.Thread.run(Unknown Source)
>>>> Caused by: java.io.IOException: Unable to create compaction marker
>>>>         at
>>>> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
>>>>         ... 13 more
>>>>
>>>>
>>>> Thank you in advance.
>>>>
>>>> Daning
>>>>
>>>


Re: Unable to create compaction marker

Posted by aaron morton <aa...@thelastpickle.com>.
It's in the yaml file…

# directories where Cassandra should store data on disk.
data_file_directories:
    - /var/lib/cassandra/data

The permissions are normally cassandra:cassandra

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 22/09/2011, at 12:06 PM, Daning wrote:

> Thanks Jonathan. where is the data directory configured? since I did not any permission problem.
> 
> Daning
> 
> On 09/21/2011 11:01 AM, Jonathan Ellis wrote:
>> Means Cassandra couldn't create an empty file in the data directory
>> designating a sstable as compacted.  I'd look for permissions
>> problems.
>> 
>> Short term there is no dire consequence, although it will keep
>> re-compacting that sstable.  Longer term you'll run out of disk space
>> since nothing gets deleted.
>> 
>> On Wed, Sep 21, 2011 at 1:44 PM, Daning<da...@netseer.com>  wrote:
>>> I got this exception in system.log several times a day, do you have idea
>>> what caused this problem and what is the consequence?
>>> 
>>> 
>>> ERROR [CompactionExecutor:12] 2011-09-15 14:29:19,578
>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>> Thread[CompactionExecutor:12,1,main]
>>> java.io.IOError: java.io.IOException: Unable to create compaction marker
>>>        at
>>> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:642)
>>>        at
>>> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
>>>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
>>>        at
>>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
>>>        at
>>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1094)
>>>        at
>>> org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:599)
>>>        at
>>> org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507)
>>>        at
>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142)
>>>        at
>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108)
>>>        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>>>        at java.util.concurrent.FutureTask.run(Unknown Source)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>>> Source)
>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>>        at java.lang.Thread.run(Unknown Source)
>>> Caused by: java.io.IOException: Unable to create compaction marker
>>>        at
>>> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
>>>        ... 13 more
>>> 
>>> 
>>> Thank you in advance.
>>> 
>>> Daning
>>> 
>> 
>> 
> 


Re: Unable to create compaction marker

Posted by Daning <da...@netseer.com>.
Thanks Jonathan. where is the data directory configured? since I did not 
any permission problem.

Daning

On 09/21/2011 11:01 AM, Jonathan Ellis wrote:
> Means Cassandra couldn't create an empty file in the data directory
> designating a sstable as compacted.  I'd look for permissions
> problems.
>
> Short term there is no dire consequence, although it will keep
> re-compacting that sstable.  Longer term you'll run out of disk space
> since nothing gets deleted.
>
> On Wed, Sep 21, 2011 at 1:44 PM, Daning<da...@netseer.com>  wrote:
>> I got this exception in system.log several times a day, do you have idea
>> what caused this problem and what is the consequence?
>>
>>
>> ERROR [CompactionExecutor:12] 2011-09-15 14:29:19,578
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[CompactionExecutor:12,1,main]
>> java.io.IOError: java.io.IOException: Unable to create compaction marker
>>         at
>> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:642)
>>         at
>> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
>>         at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
>>         at
>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
>>         at
>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1094)
>>         at
>> org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:599)
>>         at
>> org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507)
>>         at
>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142)
>>         at
>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108)
>>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>>         at java.util.concurrent.FutureTask.run(Unknown Source)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>> Source)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>         at java.lang.Thread.run(Unknown Source)
>> Caused by: java.io.IOException: Unable to create compaction marker
>>         at
>> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
>>         ... 13 more
>>
>>
>> Thank you in advance.
>>
>> Daning
>>
>
>


Re: Unable to create compaction marker

Posted by Jonathan Ellis <jb...@gmail.com>.
Means Cassandra couldn't create an empty file in the data directory
designating a sstable as compacted.  I'd look for permissions
problems.

Short term there is no dire consequence, although it will keep
re-compacting that sstable.  Longer term you'll run out of disk space
since nothing gets deleted.

On Wed, Sep 21, 2011 at 1:44 PM, Daning <da...@netseer.com> wrote:
> I got this exception in system.log several times a day, do you have idea
> what caused this problem and what is the consequence?
>
>
> ERROR [CompactionExecutor:12] 2011-09-15 14:29:19,578
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[CompactionExecutor:12,1,main]
> java.io.IOError: java.io.IOException: Unable to create compaction marker
>        at
> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:642)
>        at
> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
>        at
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
>        at
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1094)
>        at
> org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:599)
>        at
> org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507)
>        at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142)
>        at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108)
>        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>        at java.util.concurrent.FutureTask.run(Unknown Source)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>        at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Unable to create compaction marker
>        at
> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
>        ... 13 more
>
>
> Thank you in advance.
>
> Daning
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com