You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Caleb Rackliffe (Jira)" <ji...@apache.org> on 2021/02/26 20:37:00 UTC
[jira] [Comment Edited] (CASSANDRA-16471)
org.apache.cassandra.io.util.DataOutputBuffer#scratchBuffer is around 50%
of all memory allocations
[ https://issues.apache.org/jira/browse/CASSANDRA-16471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291912#comment-17291912 ]
Caleb Rackliffe edited comment on CASSANDRA-16471 at 2/26/21, 8:36 PM:
-----------------------------------------------------------------------
bq. Now, since we have been writing to the buffer we need to flip it, so one possible optimization (which didn't look a big deal) would be to return the underline BB flipped as the call paths no longer need them.
This makes a lot of sense. If we've gotten to the point where we want to write, flipping the (thread-local) {{buffer}} should be quite safe.
bq. GC will free the memory but this can happen w/e, so felt its best to do as we know its dead memory.
+1
bq. We could up the default limit, but still have the issue we push the problem into GC, and all input is user generated so is unbound how much space is needed.
I guess what I'm saying is that we've already implicitly upped the default to be infinite. Is there some value that allows us to a.) avoid another configuration knob, b.) maintain essentially all of the improvement we've achieved here in terms of reducing GC pressure (again, I'm not sure how much of that was simply due to going off-heap), and c.) make it possible to reclaim a very large buffer that was the aberrant rather than normal case? (For example, it seems like the point of {{dob_max_recycle_bytes}} was to be able to discard from the heap a huge buffer created because of something like an abnormally large read response?)
was (Author: maedhroz):
bq. Now, since we have been writing to the buffer we need to flip it, so one possible optimization (which didn't look a big deal) would be to return the underline BB flipped as the call paths no longer need them.
This makes a lot of sense. If we've gotten to the point where we want to write, flipping the (thread-local) {{buffer}} should be quite safe.
bq. GC will free the memory but this can happen w/e, so felt its best to do as we know its dead memory.
+1
bq. We could up the default limit, but still have the issue we push the problem into GC, and all input is user generated so is unbound how much space is needed.
I guess what I'm saying is that we've already implicitly upped the default to be infinite. Is there some value that allows us to a.) avoid another configuration knob, b.) maintain essentially all of the improvement we've achieved here in terms of reducing GC pressure (again, I'm not sure how much of that was simply due to going off-heap), and c.) make it possible to reclaim a very large buffer that was the aberrant rather than normal case. (For example, it seems like the point of {{dob_max_recycle_bytes}} was to be able to discard from the heap a huge buffer created because of something like an abnormally large read response?)
> org.apache.cassandra.io.util.DataOutputBuffer#scratchBuffer is around 50% of all memory allocations
> ---------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16471
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16471
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Caching
> Reporter: David Capwell
> Assignee: David Capwell
> Priority: Normal
> Attachments: Screen Shot 2021-02-25 at 3.34.28 PM.png, Screen Shot 2021-02-25 at 4.14.19 PM.png
>
>
> While running workflows to compare 3.0 with trunk we found that allocations and GC are significantly higher for a write mostly workload (22% read, 3% delete, 75% write); below is what we saw for a 2h run
> Allocations
> 30: 1.64TB
> 40: 2.99TB
> GC Events
> 30: 7.39k events
> 40: 13.93k events
> When looking at the allocation output we saw the follow for memory allocations
> !https://issues.apache.org/jira/secure/attachment/13021238/Screen%20Shot%202021-02-25%20at%203.34.28%20PM.png!
> Here we see that org.apache.cassandra.io.util.DataOutputBuffer#expandToFit is around 52% of the memory allocations. When looking at this logic I see that allocations are on-heap and constantly throw away the buffer (as a means to allow GC to clean up).
> With the patch, allocations/gc are the following
> Allocations
> 30: 1.64TB
> 40 w/ patch: 1.77TB
> 40: 2.99TB
> GC Events
> 30: 7.39k events
> 40 w/ patch: 8k events
> 40: 13.93k events
> With the patch only 0.8% allocations
> !https://issues.apache.org/jira/secure/attachment/13021239/Screen%20Shot%202021-02-25%20at%204.14.19%20PM.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org