You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Feng Qu <ma...@gmail.com> on 2012/02/22 22:49:49 UTC
nodetool ring runs very slow
We noticed that nodetool ring sometimes returns in 17-20 sec while it normally runs in less than a sec. There were some compaction running when it happened. Did compaction cause nodetool slowness? Anything else I should check?
>>> time nodetool -h hostname ring
....
real 0m17.595s
user 0m0.339s
sys 0m0.054s
Feng Qu
Re: nodetool ring runs very slow
Posted by Feng Qu <ma...@gmail.com>.
Haven't received any reply. Resend...
Feng Qu
>________________________________
> From: Feng Qu <ma...@gmail.com>
>To: Cassandra User Group <us...@cassandra.apache.org>
>Sent: Wednesday, February 22, 2012 1:49 PM
>Subject: nodetool ring runs very slow
>
>
>We noticed that nodetool ring sometimes returns in 17-20 sec while it normally runs in less than a sec. There were some compaction running when it happened. Did compaction cause nodetool slowness? Anything else I should check?
>>>> time nodetool -h hostname ring
>....
>real 0m17.595s
>user 0m0.339s
>sys 0m0.054s
>
>Feng Qu
>
>
>
Re: nodetool ring runs very slow
Posted by aaron morton <aa...@thelastpickle.com>.
You node is under significant memory pressure.
Look into:
* reducing caches
* reducing the JVM heap to 8GB
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 28/03/2012, at 6:03 AM, Feng Qu wrote:
> Hi Jonathan, similar problem happens again and there is only one GC running at that time per system.log.
>
> This is one node of a 6-node 0.8.6 ring. Heap size on this host is 16GB.
>
> [fengqu@slcdbx1035 cassandra]$ date; time cnt ring
> Tue Mar 27 09:24:20 GMT+7 2012
> Address DC Rack Status State Load Owns Token
> 141784319550391026443072753096570088106
> 10.89.74.60 slc rack0 Up Normal 343.97 GB 16.67% 0
> 10.2.128.55 phx rack0 Up Normal 485.48 GB 16.67% 28356863910078205288614550619314017621
> 10.89.74.67 slc rack0 Up Normal 252.82 GB 16.67% 56713727820156410577229101238628035242
> 10.2.128.56 phx rack0 Up Normal 258.49 GB 16.67% 85070591730234615865843651857942052864
> 10.89.74.62 slc rack0 Up Normal 251.02 GB 16.67% 113427455640312821154458202477256070485
> 10.2.128.57 phx rack0 Up Normal 451.97 GB 16.67% 141784319550391026443072753096570088106
>
> real 0m25.999s
> user 0m0.309s
> sys 0m0.048s
>
> WARN [ScheduledTasks:1] 2012-03-27 09:22:39,499 GCInspector.java (line 143) Heap is 0.8716646568744459 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> INFO [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1383 ms for 2 collections, 14756794512 used; max is 16928210944
> WARN [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 143) Heap is 0.8717279434204102 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> INFO [ScheduledTasks:1] 2012-03-27 09:24:04,844 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3090 ms for 2 collections, 14782314472 used; max is 16928210944
> WARN [ScheduledTasks:1] 2012-03-27 09:24:04,851 GCInspector.java (line 143) Heap is 0.8732354837083013 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> INFO [ScheduledTasks:1] 2012-03-27 09:24:46,610 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3057 ms for 2 collections, 14757982328 used; max is 16928210944
> WARN [ScheduledTasks:1] 2012-03-27 09:24:46,616 GCInspector.java (line 143) Heap is 0.871798111260587 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> INFO [ScheduledTasks:1] 2012-03-27 09:25:36,371 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 7430 ms for 2 collections, 14719911032 used; max is 16928210944
> WARN [ScheduledTasks:1] 2012-03-27 09:25:36,377 GCInspector.java (line 143) Heap is 0.8695491260532345 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>
> Feng Qu
> From: Jonathan Ellis <jb...@gmail.com>
> To: user <us...@cassandra.apache.org>
> Cc: Feng Qu <ma...@gmail.com>
> Sent: Friday, February 24, 2012 2:29 PM
> Subject: Re: nodetool ring runs very slow
>
> Read the server log and look for GCInspector output.
>
> On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu <ma...@gmail.com> wrote:
> > Hi Jonathan, how to check out whether it's in GC storming? This server
> > crashed few time due to Java heap out of memory. We use 8GB heap on a server
> > with 96GB ram. This is first node in a 6-node ring and it has opscenter
> > community running on it.
> >
> > Feng Qu
> >
> > ________________________________
> > From: Jonathan Ellis <jb...@gmail.com>
> > To: user@cassandra.apache.org; Feng Qu <ma...@gmail.com>
> > Sent: Thursday, February 23, 2012 1:19 PM
> > Subject: Re: nodetool ring runs very slow
> >
> > The only time I've seen nodetool be that slow is when it was talking
> > to a machine that was either swapping or deep into (JVM) GC storming.
> >
> > On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu <ma...@gmail.com> wrote:
> >> We noticed that nodetool ring sometimes returns in 17-20 sec while it
> >> normally runs in less than a sec. There were some compaction running when
> >> it
> >> happened. Did compaction cause nodetool slowness? Anything else I should
> >> check?
> >>>>> time nodetool -h hostname ring
> >> ....
> >> real 0m17.595s
> >> user 0m0.339s
> >> sys 0m0.054s
> >>
> >> Feng Qu
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
>
Re: nodetool ring runs very slow
Posted by Feng Qu <ma...@gmail.com>.
Hi Jonathan, similar problem happens again and there is only one GC running at that time per system.log.
This is one node of a 6-node 0.8.6 ring. Heap size on this host is 16GB.
[fengqu@slcdbx1035 cassandra]$ date; time cnt ring
Tue Mar 27 09:24:20 GMT+7 2012
Address
DC
Rack Status State
Load
Owns
Token
141784319550391026443072753096570088106
10.89.74.60
slc
rack0 Up
Normal 343.97 GB 16.67%
0
10.2.128.55
phx
rack0 Up
Normal 485.48 GB 16.67%
28356863910078205288614550619314017621
10.89.74.67
slc
rack0 Up
Normal 252.82 GB 16.67%
56713727820156410577229101238628035242
10.2.128.56
phx
rack0 Up
Normal 258.49 GB 16.67%
85070591730234615865843651857942052864
10.89.74.62 slc
rack0
Up Normal 251.02
GB 16.67% 113427455640312821154458202477256070485
10.2.128.57 phx
rack0 Up
Normal 451.97 GB 16.67%
141784319550391026443072753096570088106
real
0m25.999s
user 0m0.309s
sys 0m0.048s
WARN [ScheduledTasks:1] 2012-03-27 09:22:39,499 GCInspector.java (line 143) Heap is 0.8716646568744459 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
INFO [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1383 ms for 2 collections, 14756794512 used; max is 16928210944
WARN [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 143) Heap is 0.8717279434204102 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
INFO [ScheduledTasks:1] 2012-03-27 09:24:04,844 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3090 ms for 2 collections, 14782314472 used; max is 16928210944
WARN [ScheduledTasks:1] 2012-03-27 09:24:04,851 GCInspector.java (line 143) Heap is 0.8732354837083013 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
INFO [ScheduledTasks:1] 2012-03-27 09:24:46,610 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3057 ms for 2 collections, 14757982328 used; max is 16928210944
WARN [ScheduledTasks:1] 2012-03-27 09:24:46,616 GCInspector.java (line 143) Heap is 0.871798111260587 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
INFO [ScheduledTasks:1] 2012-03-27 09:25:36,371 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 7430 ms for 2 collections, 14719911032 used; max is 16928210944
WARN [ScheduledTasks:1] 2012-03-27 09:25:36,377 GCInspector.java (line 143) Heap is 0.8695491260532345 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Feng Qu
>________________________________
> From: Jonathan Ellis <jb...@gmail.com>
>To: user <us...@cassandra.apache.org>
>Cc: Feng Qu <ma...@gmail.com>
>Sent: Friday, February 24, 2012 2:29 PM
>Subject: Re: nodetool ring runs very slow
>
>Read the server log and look for GCInspector output.
>
>On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu <ma...@gmail.com> wrote:
>> Hi Jonathan, how to check out whether it's in GC storming? This server
>> crashed few time due to Java heap out of memory. We use 8GB heap on a server
>> with 96GB ram. This is first node in a 6-node ring and it has opscenter
>> community running on it.
>>
>> Feng Qu
>>
>> ________________________________
>> From: Jonathan Ellis <jb...@gmail.com>
>> To: user@cassandra.apache.org; Feng Qu <ma...@gmail.com>
>> Sent: Thursday, February 23, 2012 1:19 PM
>> Subject: Re: nodetool ring runs very slow
>>
>> The only time I've seen nodetool be that slow is when it was talking
>> to a machine that was either swapping or deep into (JVM) GC storming.
>>
>> On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu <ma...@gmail.com> wrote:
>>> We noticed that nodetool ring sometimes returns in 17-20 sec while it
>>> normally runs in less than a sec. There were some compaction running when
>>> it
>>> happened. Did compaction cause nodetool slowness? Anything else I should
>>> check?
>>>>>> time nodetool -h hostname ring
>>> ....
>>> real 0m17.595s
>>> user 0m0.339s
>>> sys 0m0.054s
>>>
>>> Feng Qu
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>>
>
>
>
>--
>Jonathan Ellis
>Project Chair, Apache Cassandra
>co-founder of DataStax, the source for professional Cassandra support
>http://www.datastax.com
>
>
>
Re: nodetool ring runs very slow
Posted by Jonathan Ellis <jb...@gmail.com>.
Read the server log and look for GCInspector output.
On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu <ma...@gmail.com> wrote:
> Hi Jonathan, how to check out whether it's in GC storming? This server
> crashed few time due to Java heap out of memory. We use 8GB heap on a server
> with 96GB ram. This is first node in a 6-node ring and it has opscenter
> community running on it.
>
> Feng Qu
>
> ________________________________
> From: Jonathan Ellis <jb...@gmail.com>
> To: user@cassandra.apache.org; Feng Qu <ma...@gmail.com>
> Sent: Thursday, February 23, 2012 1:19 PM
> Subject: Re: nodetool ring runs very slow
>
> The only time I've seen nodetool be that slow is when it was talking
> to a machine that was either swapping or deep into (JVM) GC storming.
>
> On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu <ma...@gmail.com> wrote:
>> We noticed that nodetool ring sometimes returns in 17-20 sec while it
>> normally runs in less than a sec. There were some compaction running when
>> it
>> happened. Did compaction cause nodetool slowness? Anything else I should
>> check?
>>>>> time nodetool -h hostname ring
>> ....
>> real 0m17.595s
>> user 0m0.339s
>> sys 0m0.054s
>>
>> Feng Qu
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: nodetool ring runs very slow
Posted by Jonathan Ellis <jb...@gmail.com>.
The only time I've seen nodetool be that slow is when it was talking
to a machine that was either swapping or deep into (JVM) GC storming.
On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu <ma...@gmail.com> wrote:
> We noticed that nodetool ring sometimes returns in 17-20 sec while it
> normally runs in less than a sec. There were some compaction running when it
> happened. Did compaction cause nodetool slowness? Anything else I should
> check?
>>>> time nodetool -h hostname ring
> ....
> real 0m17.595s
> user 0m0.339s
> sys 0m0.054s
>
> Feng Qu
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com