You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Pavel Yaskevich (JIRA)" <ji...@apache.org> on 2012/11/09 05:46:13 UTC
[jira] [Created] (CASSANDRA-4937) CRAR improvements (object cache +
CompressionMetadata chunk offset storage moved off-heap).
Pavel Yaskevich created CASSANDRA-4937:
------------------------------------------
Summary: CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
Key: CASSANDRA-4937
URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
Project: Cassandra
Issue Type: Bug
Reporter: Pavel Yaskevich
Assignee: Pavel Yaskevich
After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4937) CRAR improvements (object cache +
CompressionMetadata chunk offset storage moved off-heap).
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4937:
--------------------------------------
Priority: Major (was: Minor)
Affects Version/s: (was: 1.1.6)
Fix Version/s: (was: 1.1.7)
1.2.1
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Fix For: 1.2.1
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494553#comment-13494553 ]
Pavel Yaskevich commented on CASSANDRA-4937:
--------------------------------------------
If you are talking about CRAR object cache then it lasts until SSTableReader has no more references and SSTable itself is cleaned and prepared to be deleted, every caller upon using close method on CRAR would return instance to the queue to be reused by other threads (readers, compaction, streaming etc.).
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494543#comment-13494543 ]
Jonathan Ellis commented on CASSANDRA-4937:
-------------------------------------------
Back up a second ... how are you opening multiple CRAR for the same sstable in a single request? Secondary index reads?
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498603#comment-13498603 ]
Pavel Yaskevich commented on CASSANDRA-4937:
--------------------------------------------
bq. This feels like a good compromise to me: it's a lot better than what we have now for your use case: if an sstable is in the page cache we'd only dontneed it once gradually, instead of repeatedly with each trySkipCache. And it doesn't take us back to the Dark Ages of totally stomping the page cache with cold data during compaction.
I agree that it would be better than what we have, but we still would be degrading read performance if we do so, at least that wouldn't make impact as significant as right now. Practice shows that kernel is pretty efficient on reclaiming once used pages, when in need, and we still have writer that skips cache. The only 100% correct place to skip cache from SSTables is after they are compacted and before they put to the delete queue, which would give us good kernel cache margin for a new-coming SSTable.
I have actually experimented with preheating rows is we have compaction_preheat_key_cache enabled - for each of keys in preheat cache I WILLNEEDed first block of row in data file, that shows some good results and doesn't overcommit the cache because it uses memory freed after previous (compacted) sstables.
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Fix For: 1.2.1
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494468#comment-13494468 ]
Pavel Yaskevich commented on CASSANDRA-4937:
--------------------------------------------
bq. Still feels like a hack to me... Will think about alternatives.
You are welcome to think how to do so without involving more then one file, I have exploited lhf possibility :)
bq. But isn't that the point, that we do know that compaction only* does sequential i/o?
Even so, we will need to expose RAR method to skip cache to SSTable iterators so they can hint RAR to hit OS to remove cache after they are done with the row for example, which, for example, would generate too many system calls if you have a small rows (or try to somehow batch them together which would require coordination between sstable iterators) as well we would have to teach other components, which want to skip cache, how to do so (e.g. streaming). That individual approach creates nothing but complexing because reads would still be suffering even if we change granularity of hints.
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494441#comment-13494441 ]
Pavel Yaskevich commented on CASSANDRA-4937:
--------------------------------------------
bq. Still don't like the global objectCache code. SSTR queue (no Map) would be better. Or, what about SSTR Threadlocal?
The reason why it's done via queues associated with filenames is that we want to get rid of CRAR instances as soon as we done with individual SSTables and I don't want to involve SSTR there more then just cleaning queues upon termination.
bq. Isn't the right fix then to only drop the recently-read blocks during sequential i/o?
We can't because it would require (e.g. compaction) to know of how something is using files, it could remove cache from somebody's fit anyway and we would use too many system calls if we do that.
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4937) CRAR improvements (object cache +
CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4937:
---------------------------------------
Attachment: CASSANDRA-4937.patch
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4937) CRAR improvements (object cache +
CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4937:
---------------------------------------
Priority: Minor (was: Major)
Issue Type: Improvement (was: Bug)
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494212#comment-13494212 ]
Pavel Yaskevich commented on CASSANDRA-4937:
--------------------------------------------
bq. How much memory do you see CM using, prior to this change?
bq. I'm not sure how much can CRAR "objectCache" help if we trash the queue on reference release. This would be more useful if we (1) made it per-SSTR instead of global and (2) allowed CRAR to be used by more than one request. Either way, let's split this out to a separate patch – think this should be 1.2-only.
It depends on the data size per-SSTable, but on each 1GB overhead is about 20MB, which would definitely be promoted to old gen and kept there. That is not the biggest problem, the problem with compression allocation rate is that we are opening one CRAR per row read (CompressedSegmentedFile.getSegment) which allocates 128KB (64KB for compression/decompression buffers) plus has additional small memory overhead of checksum buffer and fields, that is what "objectCache" here to solve because it was figured that even when up to 12 SSTables are involved per read we wouldn't have more then "concurrent_reads" CRARs per-SSTable in the cache.
bq. RAR skip cache harms performance in situation when reads are done in parallel with compaction
The problem with the code we have right now is that it doesn't actually skip blocks that compaction read, it drops *whole* file page cache after fixed internal (128MB) so when you have long running compactions (trottled or big data files, for example) normal reads would hit the cold data very frequently. The only thing is working correctly right now in terms of skiping cache is SW. I have done some benchmarks with and without skiping cache and it shows that page replacement done by kernel is much better then our skip suggestions via recent read latency histograms.
bq. CM.close is added but I don't see it called anywhere
It should be called on CRAR.deallocate(), sorry, I must have missed that when I was merging, I will fix that and update the patch.
bq. If we're going to move CompressionMetadata off-heap, why not use a single Memory object instead of BLA? This should also be 1.2.
Yes, that could be done as a single object, I just didn't want to remove historical paging change especially actual overhead of paging is very-very low :)
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494545#comment-13494545 ]
Pavel Yaskevich commented on CASSANDRA-4937:
--------------------------------------------
bq. You are welcome to think how to do so without involving more then one file, I have exploited lhf possibility
That probably sounded confusing, I didn't mean to say that we are opening the same file multiple times per one request, I was just trying to say that I wanted to keep caching changes as local to CRAR as possible, without involving other "files" (or classes) in handling that cache, which allowed to preserve open/close calls without involving anything specific to CRAR into file handling.
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494549#comment-13494549 ]
Jonathan Ellis commented on CASSANDRA-4937:
-------------------------------------------
But doesn't your caching only last for a single request? What is the point then if not opening the same file multiple times?
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498067#comment-13498067 ]
Jonathan Ellis commented on CASSANDRA-4937:
-------------------------------------------
bq. We can't [only drop the recently-read blocks during sequential i/o] because it would require (e.g. compaction) to know of how something is using files, it could remove cache from somebody's fit anyway and we would use too many system calls if we do that.
You mean because RandomAccess reader doesn't know if we've been seeking back and forth? Technically that is true, but it looks like we only actually create RandomAccessReader with skipCache=True when we're doing sequential reads. So I'd be okay with just assuming it's sequential in reBuffer if skipCache=True.
If you wanted to make it more robust you could subclass RAR to MostlySequentialRAR (still need seek for when compaction needs two passes on wide rows) and only do skipCache in that subclass.
This feels like a good compromise to me: it's a lot better than what we have now for your use case: if an sstable *is* in the page cache we'd only dontneed it once gradually, instead of repeatedly with each trySkipCache. And it doesn't take us back to the Dark Ages of totally stomping the page cache with cold data during compaction.
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4937) CRAR improvements (object cache +
CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4937:
---------------------------------------
Attachment: CASSANDRA-4937.patch
updated the patch with missing CRAR metadata.close()
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4937) CRAR improvements (object cache +
CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4937:
---------------------------------------
Attachment: (was: CASSANDRA-4937.patch)
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494447#comment-13494447 ]
Jonathan Ellis commented on CASSANDRA-4937:
-------------------------------------------
bq. The reason why it's done via queues associated with filenames is that we want to get rid of CRAR instances as soon as we done with individual SSTables
Still feels like a hack to me... Will think about alternatives.
bq. We can't because it would require (e.g. compaction) to know of how something is using files
But isn't that the point, that we do know that compaction only* does sequential i/o?
* not quite, but close enough for this purpose
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494420#comment-13494420 ]
Jonathan Ellis commented on CASSANDRA-4937:
-------------------------------------------
Broke the CompressionMetadata chunk changes into CASSANDRA-4941.
Still don't like the global objectCache code. SSTR queue (no Map) would be better. Or, what about SSTR Threadlocal?
bq. The problem with the code we have right now is that it doesn't actually skip blocks that compaction read, it drops whole file page cache after fixed internal (128MB) so when you have long running compactions (trottled or big data files, for example) normal reads would hit the cold data very frequently.
Isn't the right fix then to only drop the recently-read blocks during sequential i/o?
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493736#comment-13493736 ]
Pavel Yaskevich commented on CASSANDRA-4937:
--------------------------------------------
As part of RAR changes instead of actively skipping cache while file is still in use, it would only release it when SSTable is no longer needed (in SSTableReader.releaseReference() before data/index cleanup).
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache
+ CompressionMetadata chunk offset storage moved off-heap).
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494166#comment-13494166 ]
Jonathan Ellis commented on CASSANDRA-4937:
-------------------------------------------
bq. we need to minimize allocation rate that compression involves
How much memory do you see CM using, prior to this change?
bq. RAR skip cache harms performance in situation when reads are done in parallel with compaction
But it's still the case that we absolutely want to keep compaction of cold data from thrashing the cache. What if we used the read metrics we collect to decide whether to skip cache instead of turning it off entirely?
Feedback on the code:
- I'm not sure how much can CRAR "objectCache" help if we trash the queue on reference release. This would be more useful if we (1) made it per-SSTR instead of global and (2) allowed CRAR to be used by more than one request. Either way, let's split this out to a separate patch -- think this should be 1.2-only.
- CM.close is added but I don't see it called anywhere
- If we're going to move CompressionMetadata off-heap, why not use a single Memory object instead of BLA? This should also be 1.2.
ISTM this needs to split into three patches -- objectCache, off-heap CM, and cache skipping.
> CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4937
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.1.6
> Reporter: Pavel Yaskevich
> Assignee: Pavel Yaskevich
> Priority: Minor
> Fix For: 1.1.7
>
> Attachments: CASSANDRA-4937.patch
>
>
> After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads.
> I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira