You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by rock zhang <ro...@alohar.com> on 2015/07/25 00:55:47 UTC
Nodetool cleanup takes long time and no progress
Hi All,
After I added node, I run node tool cleanup on the old notes , but it takes forever, no error message, and I don't see space are freed.
What should I do ? Repair first ?
Thanks
Rock
Re: After clean up, the whole system is down
Posted by rock zhang <ro...@alohar.com>.
I tried to restart , but always get out of memory error.
Then I changed to use new seed node, then it works. I guess the possible reason is I run cleanup twice ?
I need to run repair to fix some inconsistency issue, right ?
Thanks
Rock
On Jul 25, 2015, at 8:39 AM, rock zhang <ro...@alohar.com> wrote:
> I got following exception after cleanup , and the system is down, any ideas? I start it again, now it nows, still have problem, some node can not be reached.
>
> INFO [ScheduledTasks:1] 2015-07-25 11:14:20,329 ColumnFamilyStore.java:877 - Enqueuing flush of sstable_activity: 9867808 (1%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:223] 2015-07-25 11:14:20,332 Memtable.java:339 - Writing Memtable-sstable_activity@244266028(1042875 serialized bytes, 459696 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:223] 2015-07-25 11:14:20,618 Memtable.java:378 - Completed flushing /data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-681-Data.db (556904 bytes) for commitlog position ReplayPosition(segmentId=1437184929989, position=14355612)
> INFO [CompactionExecutor:4] 2015-07-25 11:14:22,828 CompactionManager.java:800 - Cleaned up to /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-122851-Data.db. 5,304,248,132 to 4,967,144,610 (~93% of original) bytes for 30,962 keys. Time: 5,499,958ms.
> ERROR [CompactionExecutor:4] 2015-07-25 11:14:22,853 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:4,1,main]
> java.lang.AssertionError: /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-103842-Data.db
> at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:235) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:738) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:95) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:385) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:281) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
> ERROR [CompactionExecutor:4] 2015-07-25 11:14:22,854 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:4,1,main]
> java.lang.AssertionError: /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-94456-Data.db
> at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:235) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:738) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:95) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:385) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:281) ~[apache-cassandra-2.1.4.jar:2.1.4]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
>
>
>
> On Jul 24, 2015, at 5:10 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Fri, Jul 24, 2015 at 5:03 PM, rock zhang <ro...@alohar.com> wrote:
>> It already 2 hours, only progress is 6%, seems it is very slow. Is there any way to speedup ?
>>
>> Cleanup is a type of compaction; it obeys the compaction throttle.
>>
>> If I interrupted the process, what gonna happen ? Next time it just compact again, right ?
>>
>> Cleanup is never automatically triggered. Also, I doubt that it will be trivial for you to interrupt it.
>>
>> I think by default setting is the compaction occurs every day .
>>
>> This doesn't parse for me. Minor compaction is continuous if the node is receiving write traffic.
>>
>> =Rob
>>
>
After clean up, the whole system is down
Posted by rock zhang <ro...@alohar.com>.
I got following exception after cleanup , and the system is down, any ideas? I start it again, now it nows, still have problem, some node can not be reached.
INFO [ScheduledTasks:1] 2015-07-25 11:14:20,329 ColumnFamilyStore.java:877 - Enqueuing flush of sstable_activity: 9867808 (1%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:223] 2015-07-25 11:14:20,332 Memtable.java:339 - Writing Memtable-sstable_activity@244266028(1042875 serialized bytes, 459696 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:223] 2015-07-25 11:14:20,618 Memtable.java:378 - Completed flushing /data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-681-Data.db (556904 bytes) for commitlog position ReplayPosition(segmentId=1437184929989, position=14355612)
INFO [CompactionExecutor:4] 2015-07-25 11:14:22,828 CompactionManager.java:800 - Cleaned up to /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-122851-Data.db. 5,304,248,132 to 4,967,144,610 (~93% of original) bytes for 30,962 keys. Time: 5,499,958ms.
ERROR [CompactionExecutor:4] 2015-07-25 11:14:22,853 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:4,1,main]
java.lang.AssertionError: /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-103842-Data.db
at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:235) ~[apache-cassandra-2.1.4.jar:2.1.4]
at org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:738) ~[apache-cassandra-2.1.4.jar:2.1.4]
at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:95) ~[apache-cassandra-2.1.4.jar:2.1.4]
at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:385) ~[apache-cassandra-2.1.4.jar:2.1.4]
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:281) ~[apache-cassandra-2.1.4.jar:2.1.4]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
ERROR [CompactionExecutor:4] 2015-07-25 11:14:22,854 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:4,1,main]
java.lang.AssertionError: /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-94456-Data.db
at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:235) ~[apache-cassandra-2.1.4.jar:2.1.4]
at org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:738) ~[apache-cassandra-2.1.4.jar:2.1.4]
at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:95) ~[apache-cassandra-2.1.4.jar:2.1.4]
at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:385) ~[apache-cassandra-2.1.4.jar:2.1.4]
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:281) ~[apache-cassandra-2.1.4.jar:2.1.4]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
On Jul 24, 2015, at 5:10 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Fri, Jul 24, 2015 at 5:03 PM, rock zhang <ro...@alohar.com> wrote:
> It already 2 hours, only progress is 6%, seems it is very slow. Is there any way to speedup ?
>
> Cleanup is a type of compaction; it obeys the compaction throttle.
>
> If I interrupted the process, what gonna happen ? Next time it just compact again, right ?
>
> Cleanup is never automatically triggered. Also, I doubt that it will be trivial for you to interrupt it.
>
> I think by default setting is the compaction occurs every day .
>
> This doesn't parse for me. Minor compaction is continuous if the node is receiving write traffic.
>
> =Rob
>
Re: Nodetool cleanup takes long time and no progress
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Jul 24, 2015 at 5:03 PM, rock zhang <ro...@alohar.com> wrote:
> It already 2 hours, only progress is 6%, seems it is very slow. Is there
> any way to speedup ?
>
Cleanup is a type of compaction; it obeys the compaction throttle.
> If I interrupted the process, what gonna happen ? Next time it just
> compact again, right ?
>
Cleanup is never automatically triggered. Also, I doubt that it will be
trivial for you to interrupt it.
> I think by default setting is the compaction occurs every day .
>
This doesn't parse for me. Minor compaction is continuous if the node is
receiving write traffic.
=Rob
Re: Nodetool cleanup takes long time and no progress
Posted by rock zhang <ro...@alohar.com>.
Hi Jeff,
It already 2 hours, only progress is 6%, seems it is very slow. Is there any way to speedup ?
If I interrupted the process, what gonna happen ? Next time it just compact again, right ? I think by default setting is the compaction occurs every day .
Thanks
Rock
On Jul 24, 2015, at 4:58 PM, rock zhang <ro...@alohar.com> wrote:
> Thank you Jeff. I just added one more node, so i want to delete moved tokens.
>
> ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
> pending tasks: 1413
> compaction type keyspace table completed total unit progress
> Cleanup rawdata raw_data 25817918778 500823247199 bytes 5.16%
> Cleanup rawdata raw_data 27060645 45183439 bytes 59.89%
> Active compaction remaining time : 0h00m00s
> ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
> pending tasks: 1410
> compaction type keyspace table completed total unit progress
> Cleanup rawdata raw_data 25911965212 500823247199 bytes 5.17%
> Cleanup rawdata raw_data 11783629 43660120 bytes 26.99%
> Active compaction remaining time : 0h00m00s
> ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
> pending tasks: 1406
> compaction type keyspace table completed total unit progress
> Cleanup rawdata raw_data 5767168 45595353 bytes 12.65%
> Cleanup rawdata raw_data 26091690659 500823247199 bytes 5.21%
> Active compaction remaining time : 0h00m00s
> ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
> pending tasks: 1406
> compaction type keyspace table completed total unit progress
> Cleanup rawdata raw_data 35078225 45595353 bytes 76.93%
> Cleanup rawdata raw_data 26119307264 500823247199 bytes 5.22%
> Active compaction remaining time : 0h00m00s
>
>
>
>
> On Jul 24, 2015, at 4:04 PM, Jeff Jirsa <Je...@crowdstrike.com> wrote:
>
>> You can check for progress using `nodetool compactionstats` (which will show Cleanup tasks), or check for ‘Cleaned up’ messages in the log (/var/log/cassandra/system.log).
>>
>> However, `nodetool cleanup` has a very specific and limited task - it deletes data no longer owned by the node, typically after adding nodes or moving tokens. Have you done that recently? Or are you trying to clean up space after issuing a lot of deletes?
>>
>>
>>
>>
>>
>> On 7/24/15, 3:55 PM, "rock zhang" <ro...@alohar.com> wrote:
>>
>>> Hi All,
>>>
>>> After I added node, I run node tool cleanup on the old notes , but it takes forever, no error message, and I don't see space are freed.
>>>
>>> What should I do ? Repair first ?
>>>
>>> Thanks
>>> Rock
>
Re: Nodetool cleanup takes long time and no progress
Posted by rock zhang <ro...@alohar.com>.
Thank you Jeff. I just added one more node, so i want to delete moved tokens.
ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
pending tasks: 1413
compaction type keyspace table completed total unit progress
Cleanup rawdata raw_data 25817918778 500823247199 bytes 5.16%
Cleanup rawdata raw_data 27060645 45183439 bytes 59.89%
Active compaction remaining time : 0h00m00s
ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
pending tasks: 1410
compaction type keyspace table completed total unit progress
Cleanup rawdata raw_data 25911965212 500823247199 bytes 5.17%
Cleanup rawdata raw_data 11783629 43660120 bytes 26.99%
Active compaction remaining time : 0h00m00s
ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
pending tasks: 1406
compaction type keyspace table completed total unit progress
Cleanup rawdata raw_data 5767168 45595353 bytes 12.65%
Cleanup rawdata raw_data 26091690659 500823247199 bytes 5.21%
Active compaction remaining time : 0h00m00s
ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
pending tasks: 1406
compaction type keyspace table completed total unit progress
Cleanup rawdata raw_data 35078225 45595353 bytes 76.93%
Cleanup rawdata raw_data 26119307264 500823247199 bytes 5.22%
Active compaction remaining time : 0h00m00s
On Jul 24, 2015, at 4:04 PM, Jeff Jirsa <Je...@crowdstrike.com> wrote:
> You can check for progress using `nodetool compactionstats` (which will show Cleanup tasks), or check for ‘Cleaned up’ messages in the log (/var/log/cassandra/system.log).
>
> However, `nodetool cleanup` has a very specific and limited task - it deletes data no longer owned by the node, typically after adding nodes or moving tokens. Have you done that recently? Or are you trying to clean up space after issuing a lot of deletes?
>
>
>
>
>
> On 7/24/15, 3:55 PM, "rock zhang" <ro...@alohar.com> wrote:
>
>> Hi All,
>>
>> After I added node, I run node tool cleanup on the old notes , but it takes forever, no error message, and I don't see space are freed.
>>
>> What should I do ? Repair first ?
>>
>> Thanks
>> Rock
Re: Nodetool cleanup takes long time and no progress
Posted by Jeff Jirsa <Je...@crowdstrike.com>.
You can check for progress using `nodetool compactionstats` (which will show Cleanup tasks), or check for ‘Cleaned up’ messages in the log (/var/log/cassandra/system.log).
However, `nodetool cleanup` has a very specific and limited task - it deletes data no longer owned by the node, typically after adding nodes or moving tokens. Have you done that recently? Or are you trying to clean up space after issuing a lot of deletes?
On 7/24/15, 3:55 PM, "rock zhang" <ro...@alohar.com> wrote:
>Hi All,
>
>After I added node, I run node tool cleanup on the old notes , but it takes forever, no error message, and I don't see space are freed.
>
>What should I do ? Repair first ?
>
>Thanks
>Rock