You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ching-Cheng Chen <cc...@evidentsoftware.com> on 2011/01/25 21:58:01 UTC
Files not deleted after compaction and GCed
Using cassandra 0.7.0
The class org.apache.cassandra.io.sstable.SSTableDeletingReference only
remove the xxxx-Data.db file, but leave the xxx-Compacted, xxx-Filter.db,
xxx-Index.db and xxx-Statistics.db intact.
And that's the behavior I saw. I ran manual compact then trigger a GC
from jconsole. The Data.db file got removed but not the others.
Is this the expected behavior?
Regards,
Chen
Re: Files not deleted after compaction and GCed
Posted by Ching-Cheng Chen <cc...@evidentsoftware.com>.
I think this might be what happening.
Since you are using ScheduledThreadPoolExecutor.schedule(), the exception
was swallowed by the FutureTask.
You will have to perform a get() method on the ScheduledFuture, and you will
get ExecutionException if there was any exception occured in run().
Regards,
Chen
On Wed, Jan 26, 2011 at 10:50 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Patch submitted.
>
> One thing I still don't understand is why
> RetryingScheduledThreadPoolExecutor isn't firing the
> DefaultUncaughtExceptionHandler, which should have logged that
> exception.
>
> On Wed, Jan 26, 2011 at 9:41 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> > Thanks for tracking that down! Created
> > https://issues.apache.org/jira/browse/CASSANDRA-2059 to fix.
> >
> > On Wed, Jan 26, 2011 at 8:17 AM, Ching-Cheng Chen
> > <cc...@evidentsoftware.com> wrote:
> >> It's a bug.
> >> In SSTableDeletingReference, it try this operation
> >> components.remove(Component.DATA);
> >> before
> >> STable.delete(desc, components);
> >> However, the components was reference to the components object which was
> >> created inside SSTable by
> >> this.components = Collections.unmodifiableSet(dataComponents);
> >> As you can see, you can't try the remove operation on that componets
> object.
> >> If I add a try block and output exception around the
> >> components.remove(Component.DATA), I got this.
> >> java.lang.UnsupportedOperationException
> >> at java.util.Collections$UnmodifiableCollection.remove(Unknown
> >> Source)
> >> at
> >>
> org.apache.cassandra.io.sstable.SSTableDeletingReference$CleanupTask.run(SSTableDeletingReference.java:103)
> >> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> >> Source)
> >> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> >> at java.util.concurrent.FutureTask.run(Unknown Source)
> >> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> >> Source)
> >> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> >> Source)
> >> at
> >>
> org.apache.cassandra.concurrent.RetryingScheduledThreadPoolExecutor$LoggingScheduledFuture.run(RetryingScheduledThreadPoolExecutor.java:81)
> >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> >> Source)
> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >> Source)
> >> at java.lang.Thread.run(Unknown Source)
> >> Regards,
> >> Chen
> >> On Tue, Jan 25, 2011 at 4:21 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>>
> >>> the other component types are deleted by this line:
> >>>
> >>> SSTable.delete(desc, components);
> >>>
> >>> On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen
> >>> <cc...@evidentsoftware.com> wrote:
> >>> > Nope, no exception at all.
> >>> > But if the same class
> >>> > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is
> >>> > responsible
> >>> > for delete other files, then that's not right.
> >>> > I checked the source code for SSTableDeletingReference, doesn't looks
> >>> > like
> >>> > it will delete other files type.
> >>> > Regards,
> >>> > Chen
> >>> >
> >>> > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis <jb...@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> No, that is not expected. All the sstable components are removed in
> >>> >> the same method; did you check the log for exceptions?
> >>> >>
> >>> >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen
> >>> >> <cc...@evidentsoftware.com> wrote:
> >>> >> > Using cassandra 0.7.0
> >>> >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference
> >>> >> > only
> >>> >> > remove the xxxx-Data.db file, but leave the xxx-Compacted,
> >>> >> > xxx-Filter.db,
> >>> >> > xxx-Index.db and xxx-Statistics.db intact.
> >>> >> > And that's the behavior I saw. I ran manual compact then
> trigger a
> >>> >> > GC
> >>> >> > from jconsole. The Data.db file got removed but not the others.
> >>> >> > Is this the expected behavior?
> >>> >> > Regards,
> >>> >> > Chen
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Jonathan Ellis
> >>> >> Project Chair, Apache Cassandra
> >>> >> co-founder of DataStax, the source for professional Cassandra
> support
> >>> >> http://www.datastax.com
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Jonathan Ellis
> >>> Project Chair, Apache Cassandra
> >>> co-founder of DataStax, the source for professional Cassandra support
> >>> http://www.datastax.com
> >>
> >>
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Files not deleted after compaction and GCed
Posted by Jonathan Ellis <jb...@gmail.com>.
Patch submitted.
One thing I still don't understand is why
RetryingScheduledThreadPoolExecutor isn't firing the
DefaultUncaughtExceptionHandler, which should have logged that
exception.
On Wed, Jan 26, 2011 at 9:41 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Thanks for tracking that down! Created
> https://issues.apache.org/jira/browse/CASSANDRA-2059 to fix.
>
> On Wed, Jan 26, 2011 at 8:17 AM, Ching-Cheng Chen
> <cc...@evidentsoftware.com> wrote:
>> It's a bug.
>> In SSTableDeletingReference, it try this operation
>> components.remove(Component.DATA);
>> before
>> STable.delete(desc, components);
>> However, the components was reference to the components object which was
>> created inside SSTable by
>> this.components = Collections.unmodifiableSet(dataComponents);
>> As you can see, you can't try the remove operation on that componets object.
>> If I add a try block and output exception around the
>> components.remove(Component.DATA), I got this.
>> java.lang.UnsupportedOperationException
>> at java.util.Collections$UnmodifiableCollection.remove(Unknown
>> Source)
>> at
>> org.apache.cassandra.io.sstable.SSTableDeletingReference$CleanupTask.run(SSTableDeletingReference.java:103)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source)
>> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>> at java.util.concurrent.FutureTask.run(Unknown Source)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>> Source)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> Source)
>> at
>> org.apache.cassandra.concurrent.RetryingScheduledThreadPoolExecutor$LoggingScheduledFuture.run(RetryingScheduledThreadPoolExecutor.java:81)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>> Source)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> Source)
>> at java.lang.Thread.run(Unknown Source)
>> Regards,
>> Chen
>> On Tue, Jan 25, 2011 at 4:21 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>
>>> the other component types are deleted by this line:
>>>
>>> SSTable.delete(desc, components);
>>>
>>> On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen
>>> <cc...@evidentsoftware.com> wrote:
>>> > Nope, no exception at all.
>>> > But if the same class
>>> > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is
>>> > responsible
>>> > for delete other files, then that's not right.
>>> > I checked the source code for SSTableDeletingReference, doesn't looks
>>> > like
>>> > it will delete other files type.
>>> > Regards,
>>> > Chen
>>> >
>>> > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis <jb...@gmail.com>
>>> > wrote:
>>> >>
>>> >> No, that is not expected. All the sstable components are removed in
>>> >> the same method; did you check the log for exceptions?
>>> >>
>>> >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen
>>> >> <cc...@evidentsoftware.com> wrote:
>>> >> > Using cassandra 0.7.0
>>> >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference
>>> >> > only
>>> >> > remove the xxxx-Data.db file, but leave the xxx-Compacted,
>>> >> > xxx-Filter.db,
>>> >> > xxx-Index.db and xxx-Statistics.db intact.
>>> >> > And that's the behavior I saw. I ran manual compact then trigger a
>>> >> > GC
>>> >> > from jconsole. The Data.db file got removed but not the others.
>>> >> > Is this the expected behavior?
>>> >> > Regards,
>>> >> > Chen
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Jonathan Ellis
>>> >> Project Chair, Apache Cassandra
>>> >> co-founder of DataStax, the source for professional Cassandra support
>>> >> http://www.datastax.com
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra support
>>> http://www.datastax.com
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Files not deleted after compaction and GCed
Posted by Jonathan Ellis <jb...@gmail.com>.
Thanks for tracking that down! Created
https://issues.apache.org/jira/browse/CASSANDRA-2059 to fix.
On Wed, Jan 26, 2011 at 8:17 AM, Ching-Cheng Chen
<cc...@evidentsoftware.com> wrote:
> It's a bug.
> In SSTableDeletingReference, it try this operation
> components.remove(Component.DATA);
> before
> STable.delete(desc, components);
> However, the components was reference to the components object which was
> created inside SSTable by
> this.components = Collections.unmodifiableSet(dataComponents);
> As you can see, you can't try the remove operation on that componets object.
> If I add a try block and output exception around the
> components.remove(Component.DATA), I got this.
> java.lang.UnsupportedOperationException
> at java.util.Collections$UnmodifiableCollection.remove(Unknown
> Source)
> at
> org.apache.cassandra.io.sstable.SSTableDeletingReference$CleanupTask.run(SSTableDeletingReference.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at
> org.apache.cassandra.concurrent.RetryingScheduledThreadPoolExecutor$LoggingScheduledFuture.run(RetryingScheduledThreadPoolExecutor.java:81)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> Regards,
> Chen
> On Tue, Jan 25, 2011 at 4:21 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> the other component types are deleted by this line:
>>
>> SSTable.delete(desc, components);
>>
>> On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen
>> <cc...@evidentsoftware.com> wrote:
>> > Nope, no exception at all.
>> > But if the same class
>> > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is
>> > responsible
>> > for delete other files, then that's not right.
>> > I checked the source code for SSTableDeletingReference, doesn't looks
>> > like
>> > it will delete other files type.
>> > Regards,
>> > Chen
>> >
>> > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis <jb...@gmail.com>
>> > wrote:
>> >>
>> >> No, that is not expected. All the sstable components are removed in
>> >> the same method; did you check the log for exceptions?
>> >>
>> >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen
>> >> <cc...@evidentsoftware.com> wrote:
>> >> > Using cassandra 0.7.0
>> >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference
>> >> > only
>> >> > remove the xxxx-Data.db file, but leave the xxx-Compacted,
>> >> > xxx-Filter.db,
>> >> > xxx-Index.db and xxx-Statistics.db intact.
>> >> > And that's the behavior I saw. I ran manual compact then trigger a
>> >> > GC
>> >> > from jconsole. The Data.db file got removed but not the others.
>> >> > Is this the expected behavior?
>> >> > Regards,
>> >> > Chen
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Files not deleted after compaction and GCed
Posted by Ching-Cheng Chen <cc...@evidentsoftware.com>.
It's a bug.
In SSTableDeletingReference, it try this operation
components.remove(Component.DATA);
before
STable.delete(desc, components);
However, the components was reference to the components object which was
created inside SSTable by
this.components = Collections.unmodifiableSet(dataComponents);
As you can see, you can't try the remove operation on that componets object.
If I add a try block and output exception around the
components.remove(Component.DATA), I got this.
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.remove(Unknown
Source)
at
org.apache.cassandra.io.sstable.SSTableDeletingReference$CleanupTask.run(SSTableDeletingReference.java:103)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at
org.apache.cassandra.concurrent.RetryingScheduledThreadPoolExecutor$LoggingScheduledFuture.run(RetryingScheduledThreadPoolExecutor.java:81)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Regards,
Chen
On Tue, Jan 25, 2011 at 4:21 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> the other component types are deleted by this line:
>
> SSTable.delete(desc, components);
>
> On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen
> <cc...@evidentsoftware.com> wrote:
> > Nope, no exception at all.
> > But if the same class
> > (org.apache.cassandra.io.sstable.SSTableDeletingReference) is responsible
> > for delete other files, then that's not right.
> > I checked the source code for SSTableDeletingReference, doesn't looks
> like
> > it will delete other files type.
> > Regards,
> > Chen
> >
> > On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>
> >> No, that is not expected. All the sstable components are removed in
> >> the same method; did you check the log for exceptions?
> >>
> >> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen
> >> <cc...@evidentsoftware.com> wrote:
> >> > Using cassandra 0.7.0
> >> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference
> only
> >> > remove the xxxx-Data.db file, but leave the xxx-Compacted,
> >> > xxx-Filter.db,
> >> > xxx-Index.db and xxx-Statistics.db intact.
> >> > And that's the behavior I saw. I ran manual compact then trigger a
> GC
> >> > from jconsole. The Data.db file got removed but not the others.
> >> > Is this the expected behavior?
> >> > Regards,
> >> > Chen
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Files not deleted after compaction and GCed
Posted by Jonathan Ellis <jb...@gmail.com>.
the other component types are deleted by this line:
SSTable.delete(desc, components);
On Tue, Jan 25, 2011 at 3:11 PM, Ching-Cheng Chen
<cc...@evidentsoftware.com> wrote:
> Nope, no exception at all.
> But if the same class
> (org.apache.cassandra.io.sstable.SSTableDeletingReference) is responsible
> for delete other files, then that's not right.
> I checked the source code for SSTableDeletingReference, doesn't looks like
> it will delete other files type.
> Regards,
> Chen
>
> On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> No, that is not expected. All the sstable components are removed in
>> the same method; did you check the log for exceptions?
>>
>> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen
>> <cc...@evidentsoftware.com> wrote:
>> > Using cassandra 0.7.0
>> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference only
>> > remove the xxxx-Data.db file, but leave the xxx-Compacted,
>> > xxx-Filter.db,
>> > xxx-Index.db and xxx-Statistics.db intact.
>> > And that's the behavior I saw. I ran manual compact then trigger a GC
>> > from jconsole. The Data.db file got removed but not the others.
>> > Is this the expected behavior?
>> > Regards,
>> > Chen
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Files not deleted after compaction and GCed
Posted by Ching-Cheng Chen <cc...@evidentsoftware.com>.
Nope, no exception at all.
But if the same class
(org.apache.cassandra.io.sstable.SSTableDeletingReference) is responsible
for delete other files, then that's not right.
I checked the source code for SSTableDeletingReference, doesn't looks like
it will delete other files type.
Regards,
Chen
On Tue, Jan 25, 2011 at 4:05 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> No, that is not expected. All the sstable components are removed in
> the same method; did you check the log for exceptions?
>
> On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen
> <cc...@evidentsoftware.com> wrote:
> > Using cassandra 0.7.0
> > The class org.apache.cassandra.io.sstable.SSTableDeletingReference only
> > remove the xxxx-Data.db file, but leave the xxx-Compacted, xxx-Filter.db,
> > xxx-Index.db and xxx-Statistics.db intact.
> > And that's the behavior I saw. I ran manual compact then trigger a GC
> > from jconsole. The Data.db file got removed but not the others.
> > Is this the expected behavior?
> > Regards,
> > Chen
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Files not deleted after compaction and GCed
Posted by Jonathan Ellis <jb...@gmail.com>.
No, that is not expected. All the sstable components are removed in
the same method; did you check the log for exceptions?
On Tue, Jan 25, 2011 at 2:58 PM, Ching-Cheng Chen
<cc...@evidentsoftware.com> wrote:
> Using cassandra 0.7.0
> The class org.apache.cassandra.io.sstable.SSTableDeletingReference only
> remove the xxxx-Data.db file, but leave the xxx-Compacted, xxx-Filter.db,
> xxx-Index.db and xxx-Statistics.db intact.
> And that's the behavior I saw. I ran manual compact then trigger a GC
> from jconsole. The Data.db file got removed but not the others.
> Is this the expected behavior?
> Regards,
> Chen
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com