You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Sergio Bilello <la...@gmail.com> on 2019/10/17 02:46:01 UTC
Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine
Hello guys!
I performed a thread dump https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMTAvMTcvLS1kdW1wLnR4dC0tMC0zMC00MA==& while try to join the node with
-Dcassandra.join_ring=false
OR
-Dcassandra.join.ring=false
OR
-Djoin.ring=false
because the node spiked in load and latency was affecting the clients.
With or without that flag the node is high in latency and I see the load sky rocketing when the number of TCP established connections increases
Analyzing the /var/log/messages I am able to read
Oct 17 00:23:39 prod-personalization-live-data-cassandra-08 cassandra: INFO [Service Thread] 2019-10-17 00:23:39,030 GCInspector.java:284 - G1 Young Generation GC in 255ms. G1 Eden Space: 361758720 -> 0; G1 Old Gen: 1855455944 -> 1781007048; G1 Survivor Space: 39845888 -> 32505856;
Oct 17 00:23:40 prod-personalization-live-data-cassandra-08 cassandra: INFO [ScheduledTasks:1] 2019-10-17 00:23:40,352 NoSpamLogger.java:91 - Some operations were slow, details available at debug level (debug.log)
Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. Check SNMP counters.
I don't see anything on debug.log that looks to be relevant
The machine is on aws with 4 cpu with 32GB Ram and 1 TB SSD i3.xlarge
[sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 32 53 559304 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 1 107 118 0 0
MutationStage 0 0 2695 0 0
MemtableReclaimMemory 0 0 11 0 0
PendingRangeCalculator 0 0 33 0 0
GossipStage 0 0 4314 0 0
SecondaryIndexManagement 0 0 0 0 0
HintsDispatcher 0 0 0 0 0
RequestResponseStage 0 0 421865 0 0
Native-Transport-Requests 22 0 1903400 0 0
ReadRepairStage 0 0 59078 0 0
CounterMutationStage 0 0 0 0 0
MigrationStage 0 0 0 0 0
MemtablePostFlush 0 0 32 0 0
PerDiskMemtableFlushWriter_0 0 0 11 0 0
ValidationExecutor 0 0 0 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 11 0 0
InternalResponseStage 0 0 0 0 0
ViewMutationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
CacheCleanupExecutor 0 0 0 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
HINT 0
MUTATION 0
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
[sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$
top - 01:44:15 up 2 days, 1:45, 4 users, load average: 34.45, 27.71, 15.37
Tasks: 140 total, 1 running, 74 sleeping, 0 stopped, 0 zombie
%Cpu(s): 90.0 us, 4.5 sy, 3.0 ni, 1.1 id, 0.0 wa, 0.0 hi, 1.4 si, 0.0 st
KiB Mem : 31391772 total, 250504 free, 10880364 used, 20260904 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 19341960 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20712 cassand+ 20 0 194.1g 14.4g 4.6g S 392.0 48.2 74:50.48 java
20823 sergio.+ 20 0 124856 6304 3136 S 1.7 0.0 0:13.51 htop
7865 root 20 0 1062684 39880 11428 S 0.7 0.1 4:06.02 ir_agent
3557 consul 20 0 41568 30192 18832 S 0.3 0.1 13:16.37 consul
7600 root 20 0 2082700 46624 11880 S 0.3 0.1 4:14.60 ir_agent
1 root 20 0 193660 7740 5220 S 0.0 0.0 0:56.36 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kthreadd
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
7 root 20 0 0 0 0 S 0.0 0.0 0:06.04 ksoftirqd/0
[sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ free
total used free shared buff/cache available
Mem: 31391772 10880916 256732 426552 20254124 19341768
Swap: 0 0 0
[sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$
bash-4.2$ java -jar sjk.jar ttop -p 20712
Monitoring threads ...
2019-10-17T01:45:33.352+0000 Process summary
process cpu=363.58%
application cpu=261.91% (user=248.65% sys=13.26%)
other: cpu=101.67%
thread count: 474
heap allocation rate 583mb/s
[000039] user=13.56% sys=-0.59% alloc= 11mb/s - OptionalTasks:1
[000379] user= 8.57% sys=-0.27% alloc= 18mb/s - ReadStage-19
[000380] user= 7.85% sys= 0.22% alloc= 19mb/s - Native-Transport-Requests-21
[000295] user= 7.14% sys= 0.23% alloc= 14mb/s - Native-Transport-Requests-5
[000378] user= 7.14% sys=-0.03% alloc= 22mb/s - Native-Transport-Requests-17
[000514] user= 6.42% sys= 0.12% alloc= 20mb/s - Native-Transport-Requests-85
[000293] user= 6.66% sys=-0.32% alloc= 12mb/s - Native-Transport-Requests-2
[000392] user= 6.19% sys= 0.14% alloc= 9545kb/s - Native-Transport-Requests-12
[000492] user= 5.71% sys=-0.24% alloc= 15mb/s - Native-Transport-Requests-24
[000294] user= 5.23% sys=-0.25% alloc= 14mb/s - Native-Transport-Requests-3
[000381] user= 5.47% sys=-0.52% alloc= 7430kb/s - Native-Transport-Requests-23
[000672] user= 4.52% sys= 0.25% alloc= 14mb/s - Native-Transport-Requests-270
[000296] user= 5.23% sys=-0.47% alloc= 13mb/s - ReadStage-7
[000673] user= 4.52% sys= 0.05% alloc= 13mb/s - Native-Transport-Requests-269
[000118] user= 4.28% sys= 0.28% alloc= 13mb/s - Native-Transport-Requests-1
[000570] user= 4.28% sys= 0.27% alloc= 13mb/s - ReadStage-31
[000671] user= 4.28% sys= 0.25% alloc= 13mb/s - ReadStage-271
[000376] user= 4.52% sys=-0.03% alloc= 13mb/s - Native-Transport-Requests-13
[000302] user= 4.76% sys=-0.30% alloc= 13mb/s - ReadStage-8
[000377] user= 4.28% sys= 0.17% alloc= 13mb/s - ReadStage-15
^Cbash-4.2$ dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
33 2 65 0 0 0| 443k 7988k| 0 0 | 0 0 |3255 2324
92 6 1 0 0 2|4732k 0 |1437k 1039k| 0 0 | 20k 24k
94 4 1 0 0 1|2588k 104k|1371k 545k| 0 0 | 13k 14k
91 6 1 0 0 2|1548k 0 |2045k 949k| 0 0 | 17k 20k
94 4 1 0 0 2|1120k 0 |1364k 372k| 0 0 | 12k 11k
95 4 0 0 0 1|3616k 0 |2137k 333k| 0 0 | 12k 7875
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra 3.11.4 Node the load starts to increase after few
minutes to 40 on 4 CPU machine
Posted by Sergio <la...@gmail.com>.
Hi Reid,
Thank you for your extensive response. I don't think that we have such a
person and in any case, even if I am a Software Engineer I would be curious
to deep dive into the problem and understand the reason. The only
observation that I have right now is that I have in the same cluster 2
keyspaces and 3 datacenters.
Only the Cassandra Nodes that are serving a particular Datacenter and
Keyspace is having thousands TCP connections established and I see these
connections only from some clients.
We have 2 kinds of clients and those have been built with 2 different
approaches: Spring Cassandra Reactive and the other one with the Java
Cassandra driver without any wrapper.
I don't know a lot about it the latter one since I didn't write that code.
I want to share that just one note I asked to add LatencyAwarePolicy in the
JAVA Cassandra Driver and this decreased tremendously the CPU LOAD for any
new Cassandra node joining the cluster. I am thinking that there could be
some driver configuration that is not correct?!
I will verify my theory and I will share the results later on for the
interested reader or maybe to help someone that found the same bizarre
behavior.
However, even with thousands connection opened the load is below 3 in a 4
CPU machine and the latency is good.
Thanks and have a great weekend
Sergio
Il giorno ven 1 nov 2019 alle ore 07:56 Reid Pinchback <
rpinchback@tripadvisor.com> ha scritto:
> Hi Sergio,
>
>
>
> I’m definitely not enough of a network wonk to make definitive statements
> on network configuration, finding your in-company network expert is
> definitely going to be a lot more productive. I’ve forgotten if you are
> on-prem or in AWS, so if in AWS replace “your network wonk” with “your AWS
> support contact” if you’re paying for support. I will make two more
> concrete observations though, and you can run these notions down as
> appropriate.
>
>
>
> When C* starts up, see if the logs contain a warning about jemalloc not
> being detected. That’s something we missed in our 3.11.4 setup and is on
> my todo list to circle back around to evaluate later. JVMs have some
> rather complicated memory management that relates to efficient allocation
> of memory to threads (this isn’t strictly a JVM thing, but JVMs definitely
> care). If you have high connection counts, I can see that likely mattering
> to you. Also, as part of that, the memory arena setting of 4 that is
> Cassandra’s default may not be the right one for you. The more concurrency
> you have, the more that number may need to bump up to avoid contention on
> memory allocations. We haven’t played with it because our simultaneous
> connection counts are modest. Note that Cassandra can create a lot of
> threads but many of them have low activity so I think it’s more about how
> many area actually active. Large connection counts will move the needle up
> on you and may motivate tuning the arena count.
>
>
>
> When talking to your network person, I’d see what they think about C*’s
> defaults on TCP_NODELAY vs delayed ACKs. The Datastax docs say that the
> TCP_NODELAY default setting is false in C*, but I looked in the 3.11.4
> source and the default is coded as true. It’s only via the config file
> samples that bounce around that it typically gets set to false. There are
> times where Nagle and delayed ACKs don’t play well together and induce
> stalls. I’m not the person to help you investigate that because it gets a
> bit gnarly on the details (for example, a refinement to the Nagle algorithm
> was proposed in the 1990’s that exists in some OS’s and can make my
> comments here moot). Somebody who lives this stuff will be a more
> definitive source, but you are welcome to copy-paste my thoughts to them
> for context.
>
>
>
> R
>
>
>
> *From: *Sergio <la...@gmail.com>
> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Date: *Wednesday, October 30, 2019 at 5:56 PM
> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Subject: *Re: Cassandra 3.11.4 Node the load starts to increase after
> few minutes to 40 on 4 CPU machine
>
>
>
> *Message from External Sender*
>
> Hi Reid,
>
> I don't have anymore this loading problem.
> I solved by changing the Cassandra Driver Configuration.
> Now my cluster is pretty stable and I don't have machines with crazy CPU
> Load.
> The only thing not urgent but I need to investigate is the number of
> ESTABLISHED TCP connections. I see just one node having 7K TCP connections
> ESTABLISHED while the others are having around 4-6K connection opened. So
> the newest nodes added into the cluster have a higher number of ESTABLISHED
> TCP connections.
>
> default['cassandra']['sysctl'] = {
> 'net.ipv4.tcp_keepalive_time' => 60,
> 'net.ipv4.tcp_keepalive_probes' => 3,
> 'net.ipv4.tcp_keepalive_intvl' => 10,
> 'net.core.rmem_max' => 16777216,
> 'net.core.wmem_max' => 16777216,
> 'net.core.rmem_default' => 16777216,
> 'net.core.wmem_default' => 16777216,
> 'net.core.optmem_max' => 40960,
> 'net.ipv4.tcp_rmem' => '4096 87380 16777216',
> 'net.ipv4.tcp_wmem' => '4096 65536 16777216',
> 'net.ipv4.ip_local_port_range' => '10000 65535',
> 'net.ipv4.tcp_window_scaling' => 1,
> 'net.core.netdev_max_backlog' => 2500,
> 'net.core.somaxconn' => 65000,
> 'vm.max_map_count' => 1048575,
> 'vm.swappiness' => 0
> }
>
> These are my tweaked value and I used the values recommended from datastax.
>
> Do you have something different?
>
> Best,
> Sergio
>
>
>
> Il giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback <
> rpinchback@tripadvisor.com> ha scritto:
>
> Oh nvm, didn't see the later msg about just posting what your fix was.
>
> R
>
>
> On 10/30/19, 4:24 PM, "Reid Pinchback" <rp...@tripadvisor.com> wrote:
>
> Message from External Sender
>
> Hi Sergio,
>
> Assuming nobody is actually mounting a SYN flood attack, then this
> sounds like you're either being hammered with connection requests in very
> short periods of time, or your TCP backlog tuning is off. At least,
> that's where I'd start looking. If you take that log message and google it
> (Possible SYN flooding... Sending cookies") you'll find explanations. Or
> just googling "TCP backlog tuning".
>
> R
>
>
> On 10/30/19, 3:29 PM, "Sergio Bilello" <la...@gmail.com>
> wrote:
>
> >
> >Oct 17 00:23:03 prod-personalization-live-data-cassandra-08
> kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending
> cookies. Check SNMP counters.
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
Re: Cassandra 3.11.4 Node the load starts to increase after few
minutes to 40 on 4 CPU machine
Posted by Reid Pinchback <rp...@tripadvisor.com>.
Hi Sergio,
I’m definitely not enough of a network wonk to make definitive statements on network configuration, finding your in-company network expert is definitely going to be a lot more productive. I’ve forgotten if you are on-prem or in AWS, so if in AWS replace “your network wonk” with “your AWS support contact” if you’re paying for support. I will make two more concrete observations though, and you can run these notions down as appropriate.
When C* starts up, see if the logs contain a warning about jemalloc not being detected. That’s something we missed in our 3.11.4 setup and is on my todo list to circle back around to evaluate later. JVMs have some rather complicated memory management that relates to efficient allocation of memory to threads (this isn’t strictly a JVM thing, but JVMs definitely care). If you have high connection counts, I can see that likely mattering to you. Also, as part of that, the memory arena setting of 4 that is Cassandra’s default may not be the right one for you. The more concurrency you have, the more that number may need to bump up to avoid contention on memory allocations. We haven’t played with it because our simultaneous connection counts are modest. Note that Cassandra can create a lot of threads but many of them have low activity so I think it’s more about how many area actually active. Large connection counts will move the needle up on you and may motivate tuning the arena count.
When talking to your network person, I’d see what they think about C*’s defaults on TCP_NODELAY vs delayed ACKs. The Datastax docs say that the TCP_NODELAY default setting is false in C*, but I looked in the 3.11.4 source and the default is coded as true. It’s only via the config file samples that bounce around that it typically gets set to false. There are times where Nagle and delayed ACKs don’t play well together and induce stalls. I’m not the person to help you investigate that because it gets a bit gnarly on the details (for example, a refinement to the Nagle algorithm was proposed in the 1990’s that exists in some OS’s and can make my comments here moot). Somebody who lives this stuff will be a more definitive source, but you are welcome to copy-paste my thoughts to them for context.
R
From: Sergio <la...@gmail.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Wednesday, October 30, 2019 at 5:56 PM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Re: Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine
Message from External Sender
Hi Reid,
I don't have anymore this loading problem.
I solved by changing the Cassandra Driver Configuration.
Now my cluster is pretty stable and I don't have machines with crazy CPU Load.
The only thing not urgent but I need to investigate is the number of ESTABLISHED TCP connections. I see just one node having 7K TCP connections ESTABLISHED while the others are having around 4-6K connection opened. So the newest nodes added into the cluster have a higher number of ESTABLISHED TCP connections.
default['cassandra']['sysctl'] = {
'net.ipv4.tcp_keepalive_time' => 60,
'net.ipv4.tcp_keepalive_probes' => 3,
'net.ipv4.tcp_keepalive_intvl' => 10,
'net.core.rmem_max' => 16777216,
'net.core.wmem_max' => 16777216,
'net.core.rmem_default' => 16777216,
'net.core.wmem_default' => 16777216,
'net.core.optmem_max' => 40960,
'net.ipv4.tcp_rmem' => '4096 87380 16777216',
'net.ipv4.tcp_wmem' => '4096 65536 16777216',
'net.ipv4.ip_local_port_range' => '10000 65535',
'net.ipv4.tcp_window_scaling' => 1,
'net.core.netdev_max_backlog' => 2500,
'net.core.somaxconn' => 65000,
'vm.max_map_count' => 1048575,
'vm.swappiness' => 0
}
These are my tweaked value and I used the values recommended from datastax.
Do you have something different?
Best,
Sergio
Il giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback <rp...@tripadvisor.com>> ha scritto:
Oh nvm, didn't see the later msg about just posting what your fix was.
R
On 10/30/19, 4:24 PM, "Reid Pinchback" <rp...@tripadvisor.com>> wrote:
Message from External Sender
Hi Sergio,
Assuming nobody is actually mounting a SYN flood attack, then this sounds like you're either being hammered with connection requests in very short periods of time, or your TCP backlog tuning is off. At least, that's where I'd start looking. If you take that log message and google it (Possible SYN flooding... Sending cookies") you'll find explanations. Or just googling "TCP backlog tuning".
R
On 10/30/19, 3:29 PM, "Sergio Bilello" <la...@gmail.com>> wrote:
>
>Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. Check SNMP counters.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<ma...@cassandra.apache.org>
For additional commands, e-mail: user-help@cassandra.apache.org<ma...@cassandra.apache.org>
RE: [EXTERNAL] Re: Cassandra 3.11.4 Node the load starts to increase
after few minutes to 40 on 4 CPU machine
Posted by "Durity, Sean R" <SE...@homedepot.com>.
There is definitely a resource risk to having thousands of open connections to each node. Some of the drivers have (had?) less than optimal default settings, like acquiring 50 connections per Cassandra node. This is usually overkill. I think 5-10/node is much more reasonable. It depends on your app architecture and cluster node count. If there are lots of small micro-services, maybe they only need 2 connections per node.
Sean Durity – Staff Systems Engineer, Cassandra
From: Sergio <la...@gmail.com>
Sent: Wednesday, October 30, 2019 5:39 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine
Hi Reid,
I don't have anymore this loading problem.
I solved by changing the Cassandra Driver Configuration.
Now my cluster is pretty stable and I don't have machines with crazy CPU Load.
The only thing not urgent but I need to investigate is the number of ESTABLISHED TCP connections. I see just one node having 7K TCP connections ESTABLISHED while the others are having around 4-6K connection opened. So the newest nodes added into the cluster have a higher number of ESTABLISHED TCP connections.
default['cassandra']['sysctl'] = {
'net.ipv4.tcp_keepalive_time' => 60,
'net.ipv4.tcp_keepalive_probes' => 3,
'net.ipv4.tcp_keepalive_intvl' => 10,
'net.core.rmem_max' => 16777216,
'net.core.wmem_max' => 16777216,
'net.core.rmem_default' => 16777216,
'net.core.wmem_default' => 16777216,
'net.core.optmem_max' => 40960,
'net.ipv4.tcp_rmem' => '4096 87380 16777216',
'net.ipv4.tcp_wmem' => '4096 65536 16777216',
'net.ipv4.ip_local_port_range' => '10000 65535',
'net.ipv4.tcp_window_scaling' => 1,
'net.core.netdev_max_backlog' => 2500,
'net.core.somaxconn' => 65000,
'vm.max_map_count' => 1048575,
'vm.swappiness' => 0
}
These are my tweaked value and I used the values recommended from datastax.
Do you have something different?
Best,
Sergio
Il giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback <rp...@tripadvisor.com>> ha scritto:
Oh nvm, didn't see the later msg about just posting what your fix was.
R
On 10/30/19, 4:24 PM, "Reid Pinchback" <rp...@tripadvisor.com>> wrote:
Message from External Sender
Hi Sergio,
Assuming nobody is actually mounting a SYN flood attack, then this sounds like you're either being hammered with connection requests in very short periods of time, or your TCP backlog tuning is off. At least, that's where I'd start looking. If you take that log message and google it (Possible SYN flooding... Sending cookies") you'll find explanations. Or just googling "TCP backlog tuning".
R
On 10/30/19, 3:29 PM, "Sergio Bilello" <la...@gmail.com>> wrote:
>
>Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. Check SNMP counters.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<ma...@cassandra.apache.org>
For additional commands, e-mail: user-help@cassandra.apache.org<ma...@cassandra.apache.org>
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: Cassandra 3.11.4 Node the load starts to increase after few
minutes to 40 on 4 CPU machine
Posted by Sergio <la...@gmail.com>.
Hi Reid,
I don't have anymore this loading problem.
I solved by changing the Cassandra Driver Configuration.
Now my cluster is pretty stable and I don't have machines with crazy CPU
Load.
The only thing not urgent but I need to investigate is the number of
ESTABLISHED TCP connections. I see just one node having 7K TCP connections
ESTABLISHED while the others are having around 4-6K connection opened. So
the newest nodes added into the cluster have a higher number of ESTABLISHED
TCP connections.
default['cassandra']['sysctl'] = {
'net.ipv4.tcp_keepalive_time' => 60,
'net.ipv4.tcp_keepalive_probes' => 3,
'net.ipv4.tcp_keepalive_intvl' => 10,
'net.core.rmem_max' => 16777216,
'net.core.wmem_max' => 16777216,
'net.core.rmem_default' => 16777216,
'net.core.wmem_default' => 16777216,
'net.core.optmem_max' => 40960,
'net.ipv4.tcp_rmem' => '4096 87380 16777216',
'net.ipv4.tcp_wmem' => '4096 65536 16777216',
'net.ipv4.ip_local_port_range' => '10000 65535',
'net.ipv4.tcp_window_scaling' => 1,
'net.core.netdev_max_backlog' => 2500,
'net.core.somaxconn' => 65000,
'vm.max_map_count' => 1048575,
'vm.swappiness' => 0
}
These are my tweaked value and I used the values recommended from datastax.
Do you have something different?
Best,
Sergio
Il giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback <
rpinchback@tripadvisor.com> ha scritto:
> Oh nvm, didn't see the later msg about just posting what your fix was.
>
> R
>
>
> On 10/30/19, 4:24 PM, "Reid Pinchback" <rp...@tripadvisor.com>
> wrote:
>
> Message from External Sender
>
> Hi Sergio,
>
> Assuming nobody is actually mounting a SYN flood attack, then this
> sounds like you're either being hammered with connection requests in very
> short periods of time, or your TCP backlog tuning is off. At least,
> that's where I'd start looking. If you take that log message and google it
> (Possible SYN flooding... Sending cookies") you'll find explanations. Or
> just googling "TCP backlog tuning".
>
> R
>
>
> On 10/30/19, 3:29 PM, "Sergio Bilello" <la...@gmail.com>
> wrote:
>
> >
> >Oct 17 00:23:03 prod-personalization-live-data-cassandra-08
> kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending
> cookies. Check SNMP counters.
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
>
Re: Cassandra 3.11.4 Node the load starts to increase after few
minutes to 40 on 4 CPU machine
Posted by Reid Pinchback <rp...@tripadvisor.com>.
Oh nvm, didn't see the later msg about just posting what your fix was.
R
On 10/30/19, 4:24 PM, "Reid Pinchback" <rp...@tripadvisor.com> wrote:
Message from External Sender
Hi Sergio,
Assuming nobody is actually mounting a SYN flood attack, then this sounds like you're either being hammered with connection requests in very short periods of time, or your TCP backlog tuning is off. At least, that's where I'd start looking. If you take that log message and google it (Possible SYN flooding... Sending cookies") you'll find explanations. Or just googling "TCP backlog tuning".
R
On 10/30/19, 3:29 PM, "Sergio Bilello" <la...@gmail.com> wrote:
>
>Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. Check SNMP counters.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra 3.11.4 Node the load starts to increase after few
minutes to 40 on 4 CPU machine
Posted by Reid Pinchback <rp...@tripadvisor.com>.
Hi Sergio,
Assuming nobody is actually mounting a SYN flood attack, then this sounds like you're either being hammered with connection requests in very short periods of time, or your TCP backlog tuning is off. At least, that's where I'd start looking. If you take that log message and google it (Possible SYN flooding... Sending cookies") you'll find explanations. Or just googling "TCP backlog tuning".
R
On 10/30/19, 3:29 PM, "Sergio Bilello" <la...@gmail.com> wrote:
>
>Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. Check SNMP counters.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine
Posted by Sergio Bilello <la...@gmail.com>.
https://docs.datastax.com/en/drivers/java/2.2/com/datastax/driver/core/policies/LatencyAwarePolicy.html I had to change the Policy in the Cassandra Driver. I solved this problem few weeks ago. I am just posting the solution for anyone that could hit the same issue.
Best,
Sergio
On 2019/10/17 02:46:01, Sergio Bilello <la...@gmail.com> wrote:
> Hello guys!
>
> I performed a thread dump https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMTAvMTcvLS1kdW1wLnR4dC0tMC0zMC00MA==& while try to join the node with
>
> -Dcassandra.join_ring=false
>
> OR
> -Dcassandra.join.ring=false
>
> OR
>
> -Djoin.ring=false
>
> because the node spiked in load and latency was affecting the clients.
>
> With or without that flag the node is high in latency and I see the load sky rocketing when the number of TCP established connections increases
>
> Analyzing the /var/log/messages I am able to read
>
> Oct 17 00:23:39 prod-personalization-live-data-cassandra-08 cassandra: INFO [Service Thread] 2019-10-17 00:23:39,030 GCInspector.java:284 - G1 Young Generation GC in 255ms. G1 Eden Space: 361758720 -> 0; G1 Old Gen: 1855455944 -> 1781007048; G1 Survivor Space: 39845888 -> 32505856;
>
> Oct 17 00:23:40 prod-personalization-live-data-cassandra-08 cassandra: INFO [ScheduledTasks:1] 2019-10-17 00:23:40,352 NoSpamLogger.java:91 - Some operations were slow, details available at debug level (debug.log)
>
>
> Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. Check SNMP counters.
>
> I don't see anything on debug.log that looks to be relevant
>
> The machine is on aws with 4 cpu with 32GB Ram and 1 TB SSD i3.xlarge
>
>
>
>
>
> [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ nodetool tpstats
>
> Pool Name Active Pending Completed Blocked All time blocked
>
> ReadStage 32 53 559304 0 0
>
> MiscStage 0 0 0 0 0
>
> CompactionExecutor 1 107 118 0 0
>
> MutationStage 0 0 2695 0 0
>
> MemtableReclaimMemory 0 0 11 0 0
>
> PendingRangeCalculator 0 0 33 0 0
>
> GossipStage 0 0 4314 0 0
>
> SecondaryIndexManagement 0 0 0 0 0
>
> HintsDispatcher 0 0 0 0 0
>
> RequestResponseStage 0 0 421865 0 0
>
> Native-Transport-Requests 22 0 1903400 0 0
>
> ReadRepairStage 0 0 59078 0 0
>
> CounterMutationStage 0 0 0 0 0
>
> MigrationStage 0 0 0 0 0
>
> MemtablePostFlush 0 0 32 0 0
>
> PerDiskMemtableFlushWriter_0 0 0 11 0 0
>
> ValidationExecutor 0 0 0 0 0
>
> Sampler 0 0 0 0 0
>
> MemtableFlushWriter 0 0 11 0 0
>
> InternalResponseStage 0 0 0 0 0
>
> ViewMutationStage 0 0 0 0 0
>
> AntiEntropyStage 0 0 0 0 0
>
> CacheCleanupExecutor 0 0 0 0 0
>
>
>
> Message type Dropped
>
> READ 0
>
> RANGE_SLICE 0
>
> _TRACE 0
>
> HINT 0
>
> MUTATION 0
>
> COUNTER_MUTATION 0
>
> BATCH_STORE 0
>
> BATCH_REMOVE 0
>
> REQUEST_RESPONSE 0
>
> PAGED_RANGE 0
>
> READ_REPAIR 0
>
> [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$
>
>
>
>
>
> top - 01:44:15 up 2 days, 1:45, 4 users, load average: 34.45, 27.71, 15.37
>
> Tasks: 140 total, 1 running, 74 sleeping, 0 stopped, 0 zombie
>
> %Cpu(s): 90.0 us, 4.5 sy, 3.0 ni, 1.1 id, 0.0 wa, 0.0 hi, 1.4 si, 0.0 st
>
> KiB Mem : 31391772 total, 250504 free, 10880364 used, 20260904 buff/cache
>
> KiB Swap: 0 total, 0 free, 0 used. 19341960 avail Mem
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 20712 cassand+ 20 0 194.1g 14.4g 4.6g S 392.0 48.2 74:50.48 java
>
> 20823 sergio.+ 20 0 124856 6304 3136 S 1.7 0.0 0:13.51 htop
>
> 7865 root 20 0 1062684 39880 11428 S 0.7 0.1 4:06.02 ir_agent
>
> 3557 consul 20 0 41568 30192 18832 S 0.3 0.1 13:16.37 consul
>
> 7600 root 20 0 2082700 46624 11880 S 0.3 0.1 4:14.60 ir_agent
>
> 1 root 20 0 193660 7740 5220 S 0.0 0.0 0:56.36 systemd
>
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kthreadd
>
> 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H
>
> 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
>
> 7 root 20 0 0 0 0 S 0.0 0.0 0:06.04 ksoftirqd/0
>
>
>
>
>
> [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ free
>
> total used free shared buff/cache available
>
> Mem: 31391772 10880916 256732 426552 20254124 19341768
>
> Swap: 0 0 0
>
> [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$
>
>
>
>
>
>
>
> bash-4.2$ java -jar sjk.jar ttop -p 20712
>
> Monitoring threads ...
>
>
>
> 2019-10-17T01:45:33.352+0000 Process summary
>
> process cpu=363.58%
>
> application cpu=261.91% (user=248.65% sys=13.26%)
>
> other: cpu=101.67%
>
> thread count: 474
>
> heap allocation rate 583mb/s
>
> [000039] user=13.56% sys=-0.59% alloc= 11mb/s - OptionalTasks:1
>
> [000379] user= 8.57% sys=-0.27% alloc= 18mb/s - ReadStage-19
>
> [000380] user= 7.85% sys= 0.22% alloc= 19mb/s - Native-Transport-Requests-21
>
> [000295] user= 7.14% sys= 0.23% alloc= 14mb/s - Native-Transport-Requests-5
>
> [000378] user= 7.14% sys=-0.03% alloc= 22mb/s - Native-Transport-Requests-17
>
> [000514] user= 6.42% sys= 0.12% alloc= 20mb/s - Native-Transport-Requests-85
>
> [000293] user= 6.66% sys=-0.32% alloc= 12mb/s - Native-Transport-Requests-2
>
> [000392] user= 6.19% sys= 0.14% alloc= 9545kb/s - Native-Transport-Requests-12
>
> [000492] user= 5.71% sys=-0.24% alloc= 15mb/s - Native-Transport-Requests-24
>
> [000294] user= 5.23% sys=-0.25% alloc= 14mb/s - Native-Transport-Requests-3
>
> [000381] user= 5.47% sys=-0.52% alloc= 7430kb/s - Native-Transport-Requests-23
>
> [000672] user= 4.52% sys= 0.25% alloc= 14mb/s - Native-Transport-Requests-270
>
> [000296] user= 5.23% sys=-0.47% alloc= 13mb/s - ReadStage-7
>
> [000673] user= 4.52% sys= 0.05% alloc= 13mb/s - Native-Transport-Requests-269
>
> [000118] user= 4.28% sys= 0.28% alloc= 13mb/s - Native-Transport-Requests-1
>
> [000570] user= 4.28% sys= 0.27% alloc= 13mb/s - ReadStage-31
>
> [000671] user= 4.28% sys= 0.25% alloc= 13mb/s - ReadStage-271
>
> [000376] user= 4.52% sys=-0.03% alloc= 13mb/s - Native-Transport-Requests-13
>
> [000302] user= 4.76% sys=-0.30% alloc= 13mb/s - ReadStage-8
>
> [000377] user= 4.28% sys= 0.17% alloc= 13mb/s - ReadStage-15
>
>
>
>
>
> ^Cbash-4.2$ dstat
>
> You did not select any stats, using -cdngy by default.
>
> ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
>
> usr sys idl wai hiq siq| read writ| recv send| in out | int csw
>
> 33 2 65 0 0 0| 443k 7988k| 0 0 | 0 0 |3255 2324
>
> 92 6 1 0 0 2|4732k 0 |1437k 1039k| 0 0 | 20k 24k
>
> 94 4 1 0 0 1|2588k 104k|1371k 545k| 0 0 | 13k 14k
>
> 91 6 1 0 0 2|1548k 0 |2045k 949k| 0 0 | 17k 20k
>
> 94 4 1 0 0 2|1120k 0 |1364k 372k| 0 0 | 12k 11k
>
> 95 4 0 0 0 1|3616k 0 |2137k 333k| 0 0 | 12k 7875
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org