You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by manish khandelwal <ma...@gmail.com> on 2020/03/23 13:31:25 UTC
High value of "All Time Blocked" for "Native-Transport-Requests"
I am working with Cassandra 3.11.2. In nodetool tpstats output I am
observing high number in "All Time Blocked" column for
"Native-Transport-Requests"
pool. I have two questions
1. Does a high number of "ALL Time Blocked" count impact client latency?
2. Does it increase off head memory overtime?
Regards
Manish
Re: High value of "All Time Blocked" for "Native-Transport-Requests"
Posted by manish khandelwal <ma...@gmail.com>.
To add on to my previous email, I think we get this error when actual off
heap memory breaches "off heap memory limit" as mentioned in this JIRA
<https://issues.apache.org/jira/browse/CASSANDRA-10930>.
On Mon, Mar 30, 2020 at 3:10 PM manish khandelwal <
manishkhandelwal03@gmail.com> wrote:
> I have already updated that property to mmap_index_only. And in pmap I
> could see only index files are loaded into memory not the SSTables.
> I feel this is some to do with direct memory buffer. There is a
> opened JIRA ticket https://issues.apache.org/jira/browse/CASSANDRA-10930 as
> well regarding this.
>
> On Mon, Mar 30, 2020 at 2:05 PM Erick Ramirez <er...@datastax.com>
> wrote:
>
>> 1. Why Cassandra memory usage remains high even if no traffic is run?
>>> 2. In which case I will get above error ( *java.lang.OutOfMemoryError:
>>> Direct buffer memory*). How to avoid those cases?
>>>
>>
>> I don't have enough information to know if it applies to your case but
>> there's a good possibility that your SSTables are getting mapped to memory
>> and it's causing high memory usage.
>>
>> There's a hidden property called disk_access_mode that determines how
>> SSTables are accessed. In older versions of C*, the behaviour was to only
>> mmap() index files. But with the added support for direct buffer
>> compressing in C* 2.2, both data and index files are getting mapped to
>> memory when they get accessed.
>>
>> I've explained it in a bit more detail in this KB article --
>> https://support.datastax.com/hc/en-us/articles/360027838911. Note that
>> it specifically mentions DSE 5.0 and 5.1 but it equally applies to C* 3.0
>> through to 3.11. The solution is to add this line to cassandra.yaml and
>> restart C*:
>>
>> disk_access_mode: mmap_index_only
>>
>> If it's the cause of the symptoms you described, you'd be able to see
>> significantly less memory usage than before. Cheers!
>>
>> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
>> have answers! Share your expertise on https://community.datastax.com/.
>>
>>
Re: High value of "All Time Blocked" for "Native-Transport-Requests"
Posted by manish khandelwal <ma...@gmail.com>.
I have already updated that property to mmap_index_only. And in pmap I
could see only index files are loaded into memory not the SSTables.
I feel this is some to do with direct memory buffer. There is a opened JIRA
ticket https://issues.apache.org/jira/browse/CASSANDRA-10930 as well
regarding this.
On Mon, Mar 30, 2020 at 2:05 PM Erick Ramirez <er...@datastax.com>
wrote:
> 1. Why Cassandra memory usage remains high even if no traffic is run?
>> 2. In which case I will get above error ( *java.lang.OutOfMemoryError:
>> Direct buffer memory*). How to avoid those cases?
>>
>
> I don't have enough information to know if it applies to your case but
> there's a good possibility that your SSTables are getting mapped to memory
> and it's causing high memory usage.
>
> There's a hidden property called disk_access_mode that determines how
> SSTables are accessed. In older versions of C*, the behaviour was to only
> mmap() index files. But with the added support for direct buffer
> compressing in C* 2.2, both data and index files are getting mapped to
> memory when they get accessed.
>
> I've explained it in a bit more detail in this KB article --
> https://support.datastax.com/hc/en-us/articles/360027838911. Note that it
> specifically mentions DSE 5.0 and 5.1 but it equally applies to C* 3.0
> through to 3.11. The solution is to add this line to cassandra.yaml and
> restart C*:
>
> disk_access_mode: mmap_index_only
>
> If it's the cause of the symptoms you described, you'd be able to see
> significantly less memory usage than before. Cheers!
>
> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
> have answers! Share your expertise on https://community.datastax.com/.
>
>
Re: High value of "All Time Blocked" for "Native-Transport-Requests"
Posted by Erick Ramirez <er...@datastax.com>.
>
> 1. Why Cassandra memory usage remains high even if no traffic is run?
> 2. In which case I will get above error ( *java.lang.OutOfMemoryError:
> Direct buffer memory*). How to avoid those cases?
>
I don't have enough information to know if it applies to your case but
there's a good possibility that your SSTables are getting mapped to memory
and it's causing high memory usage.
There's a hidden property called disk_access_mode that determines how
SSTables are accessed. In older versions of C*, the behaviour was to only
mmap() index files. But with the added support for direct buffer
compressing in C* 2.2, both data and index files are getting mapped to
memory when they get accessed.
I've explained it in a bit more detail in this KB article --
https://support.datastax.com/hc/en-us/articles/360027838911. Note that it
specifically mentions DSE 5.0 and 5.1 but it equally applies to C* 3.0
through to 3.11. The solution is to add this line to cassandra.yaml and
restart C*:
disk_access_mode: mmap_index_only
If it's the cause of the symptoms you described, you'd be able to see
significantly less memory usage than before. Cheers!
GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
have answers! Share your expertise on https://community.datastax.com/.
Re: High value of "All Time Blocked" for "Native-Transport-Requests"
Posted by manish khandelwal <ma...@gmail.com>.
Thanks for the response. With continuous runs I am observing my off heap
memory increasing and *memory usage remains high even if no traffic is
running*. Sometimes a node is hung with following error:
ERROR [ReadStage-5] 2020-03-12 20:47:46,406 JVMStabilityInspector.java:74 -
OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_221]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
~[na:1.8.0_221]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_221]
at
org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:110)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:46)
As per my understanding, this error comes when memory reservation fails
for Direct Byte buffer.I want to understand following things
1. Why Cassandra memory usage remains high even if no traffic is run?
2. In which case I will get above error ( *java.lang.OutOfMemoryError:
Direct buffer memory*). How to avoid those cases?
Regards
Manish
On Tue, Mar 24, 2020 at 7:50 AM Erick Ramirez <er...@datastax.com>
wrote:
> It's more indicative of your nodes not able to service requests because
> they are busy or overloaded. In my experience when you see the number
> climbing because of peak load, the latency tends to show issues too. Other
> symptoms of the load issue are higher-than-normal pending reads and writes.
> The high memory utilisation is also a side-effect of the high traffic load.
> At its worst, you'd be seeing read/write timeouts or connection failures on
> the client side. Cheers!
>
>>
Re: High value of "All Time Blocked" for "Native-Transport-Requests"
Posted by Erick Ramirez <er...@datastax.com>.
It's more indicative of your nodes not able to service requests because
they are busy or overloaded. In my experience when you see the number
climbing because of peak load, the latency tends to show issues too. Other
symptoms of the load issue are higher-than-normal pending reads and writes.
The high memory utilisation is also a side-effect of the high traffic load.
At its worst, you'd be seeing read/write timeouts or connection failures on
the client side. Cheers!
>